TeSCo 框架：利用文本语义增强跨视角目标对应鲁棒性
Exploiting Textual Semantics for Robust Cross-View Object Correspondence
AI 导读
研究者提出 TeSCo 框架，在视觉线索之外利用目标的文本语义进行跨视角目标对应（CVOC）。该框架先用视觉-语言模型从查询视角生成目标的文本表达，再通过文本条件跨模态融合（TCF）模块和迭代文本特征精炼（ITFR）模块将文本特征融入视觉特征。在 Ego-Exo4D 和 HANDAL-X 基准上，TeSCo 取得 SOTA 结果，代码已开源，论文被 NeurIPS 2026 接收。
正文 · 原文
Abstract:Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code is released at this https URL.
| Comments: | Accepted to NeurIPS 2026. 12 figures, 12 tables. Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| ACM classes: | I.4.6; I.4.8; I.2.10 |
| Cite as: | arXiv:2611.02356 [cs.CV] |
| (or arXiv:2611.02356v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02356 arXiv-issued DOI via DataCite |
Submission history
From: Bing Fan [view email]
[v1]
Mon, 2 Nov 2026 01:04:24 UTC (31,898 KB)
[v2]
Tue, 3 Nov 2026 22:42:56 UTC (31,898 KB)
来源：arXiv cs.AI · arxiv.org