跳到正文
arXiv cs.AI· Hanfeng Liu, Zhenyu Luo, Dongchao Zhang, Zexi Liu, Zulong Chen, Jinxin Hu, Jia Xu, Zeyi Wen·· 2 小时前AI 评分33

OfficePrism:面向企业业务职能单域操作的智能体基准测试

OfficePrism: Benchmarking Agents on Single-Domain Operations Across Enterprise Business Functions

AI 导读

OfficePrism 基准通过 420 个覆盖七类业务职能的单域企业任务,在统一的 L1–L3 任务结构层级下评估智能体能力。每项任务在共享合成组织加特定职能数据的新环境中运行,由真实记录、业务规则和工具 schema 联合构建中文工单、隐藏可执行参照与预期响应,并经自动过滤与人工审核。

正文 · 原文

View PDF HTML (experimental)

Abstract:Broad workplace scores can mix competence within one business function with the extra cost of coordinating across functions and systems, making failures hard to localize. OfficePrism evaluates this prerequisite through 420 single-domain enterprise tasks spanning seven business functions under shared, benchmark-defined L1--L3 task-structure levels. Each task runs in a fresh environment combining a shared synthetic organization with one function-specific domain and yped state access. Our pipeline uses grounded records, business rules, and tool schemas to jointly construct a Chinese-language ticket, a hidden executable reference witness, and an expected response, followed by automated filtering and human curation. The same task set is evaluated through full-schema and search-and-dispatch packages. Across 124 model--harness--package configurations, the per-case joint natural-language-and-database pass rate ranges from 22.6% to 76.4%, with a median of 67.9%. L1 is highest in all 124 configurations, and L2 exceeds L3 in 122. Among 50 configurations sharing the same harness and model labels across both packages, search scores higher in 30, ties in one, and scores lower in 19.
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Software Engineering (cs.SE)
Cite as: arXiv:2611.22262 [cs.CL]
  (or arXiv:2611.22262v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22262

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zhenyu Luo [view email]
[v1] Tue, 17 Nov 2026 06:10:32 UTC (3,643 KB)

来源:arXiv cs.AI · arxiv.org