跳到正文
arXiv cs.AI· Yining Hua, Levi Lian·· 2 小时前AI 评分53

WorkWorlds 论文测量环境构建方式对智能体评测成绩的影响

WorkWorlds: Evaluating Agents in a Company Built Before the Task

AI 导读

Yining Hua 与 Levi Lian 发布 WorkWorlds 基础设施,用真实制药公司数据重建 PharmaCo,两位各十余年经验的专家各写 8 个任务并独立构建环境,共完成 192 次受控评测。

正文 · 原文

View PDF HTML (experimental)

Abstract:Many knowledge work benchmarks are built around single tasks, where the context and files for each task are selected concurrently or after task definition. Research has argued that this construction method does not reflect real workplace work, yet the impact has never been measured under controlled conditions. To compare the two constructions under controlled conditions, we introduce WorkWorlds, an infrastructure that modularizes company components into revision, date, and employee seat to be separately adjusted. Using historical data donated by a real company, we reconstructed a pharmaceutical company, PharmaCo. Two experts with over ten years of industry experience each wrote eight tasks based on real events that occurred at the company, and each independently constructed a task-specific environment for every task. Across 192 controlled evaluations, when the agent was switched from the task-specific environment to all the materials truly visible to the employee, evidence access dropped from 90.4 percent to 72.8 percent, and the criterion pass rate dropped from 76.7 percent to 68.0 percent. However, in cases where the agent found the required evidence, the pass rates under both conditions were almost identical. Furthermore, switching between the environments built by the two experts for the same task resulted in an average pass rate difference of 6.6 percentage points. The three agents also gained by different amounts, which changed the gaps between them. We argue that benchmarks should report how environment materials were selected and, if possible, they should test agent score changes after changing the environment builder.
Comments: 16 pages, 3 figures
Subjects: Artificial Intelligence (cs.AI)
Cite as: arXiv:2609.23806 [cs.AI]
  (or arXiv:2609.23806v3 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2609.23806

arXiv-issued DOI via DataCite

Submission history

From: Yining Hua [view email]
[v1] Sun, 20 Sep 2026 18:50:13 UTC (2,878 KB)
[v2] Wed, 23 Sep 2026 00:16:50 UTC (2,882 KB)
[v3] Wed, 18 Nov 2026 16:42:45 UTC (1,397 KB)

来源:arXiv cs.AI · arxiv.org