跳到正文
arXiv cs.AI· Hanqi Mao, Jiajun Fan, Ruihan Guo, Shuaike Shen, Ziwen Wang, Hui Ren, Xianrui Zhong, Ge Liu·· 1 小时前AI 评分51

Vanguard:让视觉语言智能体在步骤级投机中判断何时信任小模型草稿

Vanguard: Knowing When to Trust the Draft in Speculative Vision-Language Agents

AI 导读

研究者提出训练无关的判断模型 Vanguard,用于视觉语言智能体的步骤级投机:由通用决策模型结合屏幕截图在一次前向传播中为候选步骤给出正确概率,超过阈值即采用小模型草稿,否则由大模型改写。

正文 · 原文

View PDF HTML (experimental)

Abstract:Vision-language agents that browse websites, operate phones and act in homes reason before every action, so every step waits for a large model, a latency and cost that stand in the way of deployment. Step-level speculation lets a small model draft each step and calls the large model only on rejected drafts, but its outcome rests on the judge, and a judge that reads only a step's text cannot tell whether the step fits the screen it acts on. We find that a few samples of a small model can contain a correct step about as often as the large model gets the step right, and not always on the same steps: checked against the screen, small drafts can be not only cheaper but better. We present Vanguard, a training-free judge that leaves the draft and target models unchanged: a general decision model, with the screenshot in view, gives each candidate step a probability of being right in one forward pass; the best candidate is kept if it clears a threshold, and otherwise the large model writes the step. On web, mobile and embodied agents, a 2B draft judged by Vanguard is at least as accurate as its 8B target and significantly more accurate on Mind2Web and EB-ALFRED, while the target writes only a minority of the steps; it completes far more closed-loop episodes, and with a 27B target from another family it is both faster and more accurate. Vanguard is the most accurate judge we compare on the agent tasks; it relies on the screenshot on web and mobile tasks, its calibrated threshold transfers to unseen websites, and across model families its gain over the target grows with the draft's pass@4 relative to the target's accuracy. Vanguard composes with token-level speculative decoding and matches trained process reward models on mathematics, turning speculation from a way to save the large model's work into one that can improve on it.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23917 [cs.AI]
  (or arXiv:2611.23917v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23917

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Hanqi Mao [view email]
[v1] Wed, 18 Nov 2026 06:09:08 UTC (11,821 KB)

来源:arXiv cs.AI · arxiv.org