arXiv cs.AI· Xiaoyi Chen, Yifei Gao, Yang Xu, Zhenlong Wang, Jian Yan, Jiming An, Xingxing Song, Liqun Deng, Xiaojun Meng, Jiansheng Wei, Yi Zhang, Jitao Sang·· 5 小时前AI 评分37
GUITestScape：面向探索式 GUI 测试的开集评测
GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing
AI 导读
GUITestScape 是一个面向 MLLM 智能体探索式 GUI 测试的交互式基准，覆盖 61 个真实 Android 应用和 508 个预设缺陷，包含交互与显示两类。团队还提出开集评测器 GUIJudge，将测试轨迹分解为可独立诊断的能力，显著优于所有基线。实验显示缺陷检测仍是现有模型的关键瓶颈，将 GUIJudge 验证器集成到现有智能体可无重训提升检测性能。
正文 · 原文
Authors:Xiaoyi Chen, Yifei Gao, Yang Xu, Zhenlong Wang, Jian Yan, Jiming An, Xingxing Song, Liqun Deng, Xiaojun Meng, Jiansheng Wei, Yi Zhang, Jitao Sang
Abstract:Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.29532 [cs.SE] |
| (or arXiv:2605.29532v2 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29532 arXiv-issued DOI via DataCite |
Submission history
From: Xiaoyi Chen [view email]
[v1]
Thu, 28 May 2026 07:47:27 UTC (13,816 KB)
[v2]
Thu, 29 Oct 2026 03:44:30 UTC (15,884 KB)
来源：arXiv cs.AI · arxiv.org