IGG：面向可见性约束下交互式 GUI Grounding 的基准测试
IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
AI 导读
研究者发布 IGG 基准，评测智能体先通过交互让目标可见、再点击的 GUI grounding 能力，包含 601 个任务、16 个可控 GUI 环境和七类可见性约束。人类参与者准确率达 94.6%，GPT-5.5 为 79.4%，最强开源 VLM 仅 31.6%。对 Qwen3-VL 的干预实验显示，替模型执行显现操作可将其准确率提升至 81.9%，瓶颈在于执行交互而非识别目标。
正文 · 原文
Abstract:GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.
| Comments: | Accepted to EMNLP 2026 (Main Conference). 24 pages. Project page: this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32536 [cs.AI] |
| (or arXiv:2610.32536v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32536 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiyeon Son [view email]
[v1]
Fri, 23 Oct 2026 12:14:28 UTC (40,935 KB)
来源：arXiv cs.AI · arxiv.org