跳到正文
arXiv cs.AI· Haolin Yang, Wenshan Wu, Jin Jiang, Yu Cheng, Yan Xia, Ting Song, Yangyu Huang, Zhiliang Peng, Wenhui Wang, Hangbo Bao, Yuzhe Zhang, Wen-Da Wei, Tengchao Lv, Li Dong, Lei Cui, Furu Wei·· 4 小时前AI 评分53

Vision-as-a-Tool 论文提出 VHarness 框架,将视觉推理模块化为可调用工具

Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding

AI 导读

arXiv:2611.17797 论文提出 Vision-as-a-Tool(VaaT)范式,把多模态大语言模型作为可调用的外部工具,让纯文本智能体在 agentic coding 中卸载视觉推理,视觉智能体也能按需增强原生感知。

正文 · 原文

Authors:Haolin Yang, Wenshan Wu, Jin Jiang, Yu Cheng, Yan Xia, Ting Song, Yangyu Huang, Zhiliang Peng, Wenhui Wang, Hangbo Bao, Yuzhe Zhang, Wen-Da Wei, Tengchao Lv, Li Dong, Lei Cui, Furu Wei

View PDF HTML (experimental)

Abstract:Multimodal agentic coding requires robust visual reasoning to analyze observations and evaluate generated artifacts. Current frameworks largely rely on a Thinking-with-Images paradigm, where agents depend on their intrinsic vision to execute visually grounded tasks. However, this architectural coupling introduces a fundamental bottleneck. Agent performance remains strictly bounded by internal perceptual limitations. Furthermore, text-only agents lack native perception. Without specialized tools, they cannot perform visual reasoning. To overcome these constraints, we introduce the Vision-as-a-Tool (VaaT) paradigm. Prior visual tool-use merely manipulates images for an agent's intrinsic vision to read. In contrast, VaaT reframes general-purpose multimodal large language models (MLLMs) as callable external utilities. As a result, text-only agents can completely offload visual reasoning during agentic execution. Meanwhile, vision-capable agents can dynamically augment their native perception on demand. We instantiate this paradigm in VHarness, a unified framework driven by five atomic actions. We comprehensively evaluate VHarness across Image-to-HTML, Image-to-SVG, and Image-to-3D generation tasks. Experimental results demonstrate that VHarness consistently improves visual fidelity and geometric accuracy across both text-only and multimodal coding agents. These findings highlight that modularizing visual reasoning into callable utilities provides a flexible and scalable foundation for future agentic coding systems.
Comments: 27 pages, including references and appendices. Project page: this https URL
Subjects: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2611.17797 [cs.AI]
  (or arXiv:2611.17797v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.17797

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Haolin Yang [view email]
[v1] Fri, 13 Nov 2026 06:23:19 UTC (5,696 KB)

来源:arXiv cs.AI · arxiv.org