跳到正文
arXiv cs.AI· Yuke Li, Xuehan Hou, Jie Chen·· 4 小时前AI 评分36

回归无关的布局感知匹配方法实现无幻觉 GUI Grounding

Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

AI 导读

一项 arXiv 研究提出回归无关(regression-free)的 GUI grounding 框架:冻结的 MLLM 先将抽象指令解析为含布局线索的结构化视觉描述,再由布局感知 grounding 模型通过布局先验候选匹配完成定位,不学习坐标回归,从而抑制模型幻觉并免去昂贵微调。

正文 · 原文

View PDF HTML (experimental)

Abstract:GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2608.09654 [cs.AI]
  (or arXiv:2608.09654v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2608.09654

arXiv-issued DOI via DataCite

Submission history

From: Xuehan Hou [view email]
[v1] Mon, 10 Aug 2026 14:29:07 UTC (3,044 KB)
[v2] Fri, 13 Nov 2026 08:38:37 UTC (3,044 KB)

来源:arXiv cs.AI · arxiv.org