知识工作智能体基准设计研究：提出四字段表示并分析 GDPval、OfficeQA Pro、APEX-SWE
Designing Benchmarks for Knowledge Work
AI 导读
论文提出以工作为中心的基准表示法，从职业研究、情境行动、边界对象与效度理论推导出四个隐含设计选择，并给出 represented activity、tested setting、required work product、evaluated result 四个报告字段及一页模板。
正文 · 原文
Abstract:Agent benchmarks have attracted substantial attention; yet, benchmarks for AI agents performing knowledge work are still mostly described only by their tasks, environments, and metrics. Starting from the definition of knowledge work, this paper derives four other design choices that often remain implicit. Three research threads on how knowledge work is organized each yield one choice: studies of professions, situated action, and boundary objects and coordination, while validity theory provides the fourth. Following this reasoning, we propose a benchmark representation centered on the work being performed. Based on this representation, we propose reporting practices for four largely missing fields: represented activity, tested setting, required work product, and evaluated result. Benchmarks designed in disparate ways thereby share a common description and can be compared side by side. To support the reporting of work activities, we cluster O*NET task statements into 18 shared work activities through unsupervised learning paired with human review. We then use this representation to analyze cases from three popular knowledge work benchmarks: GDPval, OfficeQA Pro, and APEX-SWE. Across all three benchmarks, the scope covered by the evaluated result is narrower than that of the required work product. We audited the tests of all 25 public APEX-SWE Integration tasks, independently coded by two authors. We found from the test code that in 13 of these tasks, a deliverable can alter established content from the source data and still pass all tests. We synthesize these four fields into a one-page record, accompanied by a worked example and a fillable template.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.23262 [cs.AI] |
| (or arXiv:2605.23262v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23262 arXiv-issued DOI via DataCite |
Submission history
From: Yining Hua [view email]
[v1]
Fri, 22 May 2026 06:03:01 UTC (1,209 KB)
[v2]
Mon, 24 Aug 2026 13:42:48 UTC (2,081 KB)
[v3]
Tue, 3 Nov 2026 23:25:29 UTC (991 KB)
来源：arXiv cs.AI · arxiv.org