arXiv cs.AI· Hyesoo Hong, Min Soo Kim, Wonje Jeung, Sangyeon Yoon, Dongjae Jeon, Albert No·· 3 小时前AI 评分51
VLMs Trace Without Tracking：诊断视觉语言模型路径跟随失败的原因
VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following
AI 导读
研究者系统性测试前沿视觉语言模型在受控设置下跟随连续视觉路径的能力，发现即使在无交叉干扰的简单场景中追踪也很脆弱，问题表现为邻近路径存在时路径保持不稳定。内部表征分析显示确定正确路径延续所需的信息在视觉编码阶段就已衰减；扩大模型规模、增加推理和显式追踪指令均无法有效解决，更真实场景中同样失败持续存在。
正文 · 原文
Abstract:Vision-language models (VLMs) perform strongly on multimodal benchmarks, yet their ability to reliably follow continuous visual structure remains unclear. We focus on path following, a basic operation that requires following a selected path through successive visual continuations. Before considering complex factors such as crossings or overlaps, we first test whether current VLMs can robustly follow a simple continuous line under controlled settings that remove such confounds. Across frontier models, we find that tracing is already fragile, with this vulnerability manifesting as unstable path preservation in the presence of other nearby paths. Analysis of internal representations shows that information needed to determine the correct path continuation becomes weaker, with this degradation already emerging in the visual encoding stage. Common remedies do not resolve this weakness: model scaling provides limited improvement, additional reasoning provides uneven gains at substantial cost, and explicit tracing instructions fail to stabilize performance. The same path-preservation failures and sensitivity to nearby paths persist in visually richer and more realistic settings, suggesting that current VLMs lack a robust mechanism for following a selected path.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.15672 [cs.CV] |
| (or arXiv:2605.15672v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.15672 arXiv-issued DOI via DataCite |
Submission history
From: Hyesoo Hong [view email]
[v1]
Fri, 15 May 2026 06:48:53 UTC (13,615 KB)
[v2]
Fri, 23 Oct 2026 04:27:49 UTC (16,236 KB)
来源：arXiv cs.AI · arxiv.org