SpecLook:通过推测视觉观察位置加速并提升 Agentic MLLM 可靠性的免训练框架
SpecLook: Speculating Where to Look for Fast and Reliable Agentic MLLMs
AI 导读
SpecLook 是一个免训练的推测-验证框架,用小模型起草图像局部观察并重答问题,答案不变且自信时直接结束,否则由大模型验证修正观察并融合答案,全程无需传统 zoom-in 工具调用循环。
正文 · 原文
Abstract:Agentic multimodal large language models (MLLMs) that think with images interleave reasoning with zoom-in tool calls. However, this sequential loop is slow, and its answers carry no reliability score. Prior speculation of tokens, reasoning steps, or answers keeps the loop, and answer speculation also accepts confident errors. We find that the loop mostly adds one zoom-in step, with which a small model is ahead of the agent on most splits. Our key insight is to speculate the agent's next observation, which, unlike a drafted answer, can be verified, corrected, and used to test the answer. We propose SpecLook, a training-free speculate-and-verify framework with three levels. A small model drafts a glimpse and re-answers from it, and the question ends if the answer is unchanged and confident. Otherwise, the agent's large model verifies and corrects the glimpse without tools, and their answers are fused. Only questions that still lack evidence reach the agent, in one forward pass that SpecLook-Fast skips, so no question runs the loop. The trace scores every answer's reliability without labels. Extensive experiments on V*, HR-Bench, and POPE show that both are more accurate than every baseline on every split. SpecLook reaches 87.2% and 86.2% mean accuracy with 7.7x and 4.7x speedups over the DeepEyes-7B and Thyme-RL agents, and SpecLook-Fast reaches 9.9x and 6.3x.
| Comments: | Technical report |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.23483 [cs.AI] |
| (or arXiv:2611.23483v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23483 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xi Ding [view email]
[v1]
Tue, 17 Nov 2026 20:34:56 UTC (8,163 KB)
来源:arXiv cs.AI · arxiv.org