计算机使用智能体训练语料中的无障碍债务：屏幕阅读器读不到的内容
Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use Agents
AI 导读
一篇 arXiv 论文测量了计算机使用智能体训练语料中的内容描述覆盖率缺口。在 Rico（2017 年采集）与 AndroidControl（2024 年采集）两个 Android 语料库共 17,373,614 个界面元素中，需要内容描述的元素分别只有 17.74% 和 57.27% 带描述，输入控件覆盖率最低（2.20% 与 27.21%）。
正文 · 原文
Abstract:Computer-use agents learn to operate mobile interfaces from corpora of captured screens and accessibility trees. The same trees are what a screen reader consumes, so an agent trained on them inherits whatever labelling gaps the underlying applications carry. This paper measures one of those gaps, content-description coverage, in two Android corpora: Rico, collected in 2017, and AndroidControl, collected in 2024. Across 17,373,614 interface elements, 17.74% of the Rico elements that need a content description under our coding rule carry one, against 57.27% in AndroidControl. The difference of 39.5 percentage points holds under five alternative coding rules and when whole apps are resampled. It describes two datasets that also differ in app selection, capture method and platform version; it does not measure change over time. Input controls are the least labelled role in both corpora (2.20% and 27.21%, or 30.99% in AndroidControl if hint text counts as a name). Descriptions in the older corpus are also five times more likely to be placeholders (10.25% against 1.95%), so the difference is not an artefact of developers typing filler. Resolving 26,205 task actions to the elements they touched shows that, among tappable graphics with no text, demonstrations operate elements described 9.0 points more often than the others on the screens they visit, about 3.5 points of it from the choice of element within a screen. That is a selection effect inside the benchmark. A two-coder reliability check (kappa = 0.91 and 0.96) and the full pipeline accompany the paper.
| Comments: | 8 pages, 2 figures, 6 tables. Accepted at the International Conference on Artificial Intelligence Centric Systems, Services, and Applications (AISSA 2026), San Antonio, TX, USA, December 8-10, 2026 |
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI) |
| ACM classes: | H.5.2; K.4.2 |
| Cite as: | arXiv:2611.16107 [cs.HC] |
| (or arXiv:2611.16107v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16107 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Obada Kraishan [view email]
[v1]
Wed, 11 Nov 2026 23:28:30 UTC (102 KB)
来源：arXiv cs.AI · arxiv.org