arXiv cs.AI· Yipeng Sun, Jiajun Wang, Siyuan Mei, Yan Xia, Siming Bayer, Vincent Christlein, Andreas Maier·· 2 小时前AI 评分43
跨模态对齐的关键是什么:arXiv 论文提出 LightAlign 少样本对齐方法
What Matters for Cross-Modal Alignment
AI 导读
一篇 arXiv 论文研究了零配对跨模态对齐的条件:在横跨自然图像、环境音频、胸片和 3D CT 四种模态的五个数据集上,该方法仅在 COCO 上成功。分析发现对齐需先定位模态共变的子空间,当共享子空间方差占比小时必须依赖图文对,且可用随机矩阵理论预测所需配对数。
正文 · 原文
Abstract:The Platonic Representation Hypothesis posits that models trained on different modalities converge toward a shared representation. Related work aligns frozen image and text encoders on natural images without paired data. We test zero-pair alignment on five datasets across four modalities: natural images, environmental audio, chest radiographs, and 3D computed tomography (CT). It succeeds only on COCO and fails on the other four. We analyze the failures and find that alignment must first locate the subspace where the modalities co-vary and then orient the mapping within it. Consistent with previous studies, medical imaging is the hardest case: its shared subspace carries the smallest fraction of the variance. When this fraction is small, alignment needs image-text pairs, and random-matrix theory predicts how many pairs are needed. These findings motivate LightAlign: a few pairs let a ridge regression between two frozen encoders locate and orient the mapping simultaneously, and a classifier fitted only on report text then predicts findings from the mapped images. On 3D CT, LightAlign surpasses models trained end to end on image-report pairs without finding-level labels: with only 200 pairs, LightAlign outperforms CT-CLIP on CT-RATE (0.754 vs. 0.733 AUROC), and with only 100 CT-RATE pairs, it does so on RAD-ChestCT, an out-of-distribution dataset from a different hospital. On CheXpert, it reaches 0.862 mean AUROC with only about 5% as many pairs as models trained on MIMIC-CXR.
| Comments: | Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.20000 [cs.CV] |
| (or arXiv:2611.20000v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.20000 arXiv-issued DOI via DataCite |
Submission history
From: Yipeng Sun [view email]
[v1]
Sun, 15 Nov 2026 13:18:52 UTC (5,356 KB)
[v2]
Tue, 17 Nov 2026 08:04:00 UTC (5,357 KB)
来源:arXiv cs.AI · arxiv.org