arXiv cs.AI· Mahmut S. Gokmen, Mitchell A. Klusty, Evan W. Damron, Caroline N. Leach, Samuel E. Armstrong, V. K. Cody Bumgardner·· 3 小时前AI 评分51
DuoJEPA 论文提出不对齐嵌入空间间的跨模态翻译方法
DuoJEPA: Cross-Modal Translation Between Unaligned Self-Supervised Embedding Spaces
AI 导读
arXiv 论文 DuoJEPA 提出，在两个模态各自的自监督 LeJEPA 嵌入空间保持不对齐的情况下，用单个 transformer 翻译层加对比目标实现跨模态检索与分类。
正文 · 原文
Abstract:Learning from paired modalities is commonly equated with aligning them: two encoders are trained until the embeddings of a pair coincide in a shared space. We study whether cross-modal retrieval and classification can instead be obtained while the two encoders' own embedding spaces remain directly unaligned. DuoJEPA trains one encoder per modality with the self-supervised LeJEPA objective and appends to each encoder a single transformer layer that translates its embedding into the other encoder's space, supervised by a contrastive objective computed entirely within that target space. No loss term compares the two own spaces directly, although the encoders are optimised jointly through the translation objective. In our experiments, the translator's residual connections determine where the cross-modal pressure is absorbed: with a residual translator the two encoders become directly aligned, whereas with a skipless (residual-free) translator a paired-cosine test detects no alignment in the final models, and at most a weak one in the ablations, at comparable cross-modal performance. The learned class-token translation is well approximated by a single scaled rotation of the source space. We evaluate DuoJEPA on two modality pairs that differ in nearly every respect. On chest radiographs and reports from CheXpert Plus, it attains a zero-shot AUC of 86.6 and a cross-modal retrieval score of 78.5, compared with 60.0 and 64.2 for our reimplementation of LeVLJEPA with the same encoders and data. On Sentinel-2 optical and Sentinel-1 radar imagery, the frozen encoders fall within the range of published Sentinel foundation models under our reproduction of the Copernicus-Bench linear-probe protocol, the translator retrieves the paired image of the other sensor among 4,000 candidates in 77-79% of cases, and a patch-level extension transfers the within-image similarity structure of the scene.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.15968 [cs.CV] |
| (or arXiv:2611.15968v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.15968 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mahmut Gokmen [view email]
[v1]
Wed, 11 Nov 2026 20:10:52 UTC (3,142 KB)
来源：arXiv cs.AI · arxiv.org