arXiv cs.AI· Alice Gao, Weixi Tong, Rishab Vempati, Katharina Reinecke, R. Benjamin Shapiro, Tianyi Zhang, Jason Wu·· 3 小时前AI 评分27
训练 Computer Use Agent 评估图形界面可用性
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
AI 导读
一项研究提出新的训练目标，把可用性操作化为界面变体间的相对比较，用以微调 computer use agent（CUA）评估 GUI 可用性。该方法基于包含完全交互式 UI 与可用性标签的大规模数据集，在三个现成 CUA 模型上提升了相对可用性评估准确率，最佳模型超越了专有基线，并产出有依据的评估结果。
正文 · 原文
Abstract:Recent HCI research leverages AI agents to assess graphical user interfaces (GUIs) usability, but current agents still struggle. Computer use agents (CUAs) that are used in these assessments are predominantly developed using objective task trajectories with verifiable outcomes and lack an operationalized definition of usability and necessary training data. This work presents a novel training objective that operationalizes usability to fine-tune CUAs to assess GUI usability through relative comparisons between interface variants. We apply our method to CUA models using a large-scale dataset of fully interactive UIs paired with usability labels mixed with human this http URL empirical evaluations across real and synthetic GUIs show that our method i) improved relative usability assessment accuracy across three off-the-shelf CUA models, ii) enabled our best model to outperform proprietary baselines, and iii) produced grounded usability assessments. Our work lays a data-driven foundation for agent-based usability assessment, enabling exploration of new forms of scalable usability evaluation and interface design.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.26020 [cs.CL] |
| (or arXiv:2604.26020v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.26020 arXiv-issued DOI via DataCite |
Submission history
From: Alice Gao [view email]
[v1]
Tue, 28 Apr 2026 18:04:11 UTC (5,812 KB)
[v2]
Tue, 10 Nov 2026 20:28:07 UTC (6,457 KB)
来源：arXiv cs.AI · arxiv.org