RoSA：缓解显著性坍缩的鲁棒显著性感知长文本图文对齐框架
Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
AI 导读
arXiv 论文提出 RoSA，一种面向长文本图文对齐的显著性感知微调框架，用于解决模型过度依赖显著文本、忽视非显著上下文导致的"显著性坍缩"问题。该方法将图文对分解为显著与上下文互补视图，结合物体级视觉显著性估计与基于 LLM 的文本分解，并用辅助目标显式对齐两类视图。RoSA 显著提升长文本检索表现，辅助目标即插即用，可为现有微调方法带来一致增益，代码与 checkpoint 已开源。
正文 · 原文
Abstract:Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.31005 [cs.AI] |
| (or arXiv:2610.31005v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31005 arXiv-issued DOI via DataCite |
Submission history
From: Qiuyu Kong [view email]
[v1]
Thu, 22 Oct 2026 09:44:57 UTC (32,220 KB)
[v2]
Fri, 23 Oct 2026 09:09:44 UTC (32,220 KB)
来源：arXiv cs.AI · arxiv.org