arXiv cs.AI· Delin Wang, Sinuo Wang, Rui Sun, Beidi Luan, Qiming Shi, Jing Li, Daxin Jiang, Zuo Bai·· 2 小时前AI 评分39
RubricAtlas：从专家研报中挖掘类型级评分标准评估深度研究智能体
RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents
AI 导读
arXiv 论文提出 RubricAtlas 框架，从专家报告中挖掘类型级分析要求来评估深度研究智能体的报告输出。该框架基于 3,079 份中文金融研报构建，覆盖 99 种研究类型，将二元检查点关联到报告段落并分别评估完整性与论证有效性。实验显示单一报告仅覆盖所属类型约一半的分析问题；十个智能体上两条裁判管线取得 κ=0.71 的检查点一致性与 Kendall τ=0.87 的排名一致性。
正文 · 原文
Abstract:Deep research agents conduct multi-step Web searches and synthesise retrieved information into analytical reports. Evaluating these outputs requires checking which analyses they deliver, beyond the presence of relevant facts. Query-specific binary rubrics make individual requirements explicit, but task-level checklists may leave the overall analytical structure underexamined. We introduce RubricAtlas, which derives type-level analytical requirements from expert reports on different objects within the same research type and organises detailed checks around recurring analytical questions. The requirements give related tasks a shared standard for what to analyse, without prescribing their answers. Our framework links binary checkpoints to report passages and evaluates completeness separately from argument validity. Built from 3,079 Chinese financial research reports, RubricAtlas covers 99 research types. Reference-report selection experiments show that individual reports cover only about half of their type's analytical questions, and that selecting requirements from different reports changes evaluation scores and agent rankings. Across ten agents, two judge pipelines achieve checkpoint agreement of $\kappa$ = 0.71 and agent-ranking agreement of Kendall $\tau$ = 0.87. Controlled deletions reduce scores on the affected analyses, while padding produces no detectable increase. These findings support type-level rubrics for comparing the analytical completeness of Web deep research outputs, with individual decisions traceable to report evidence.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.36878 [cs.AI] |
| (or arXiv:2610.36878v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36878 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zuo Bai [view email]
[v1]
Tue, 27 Oct 2026 09:18:00 UTC (188 KB)
来源：arXiv cs.AI · arxiv.org