arXiv cs.AI· Dongnuan Tian, Yan Yu, Yangfan Liu, Wenjun Tang, Junting Kan, Xiang Wang, Sicong Liu, Yangyang Zhang, Shuanghong Li·· 56 分钟前AI 评分45
投资研究 AI 智能体的可靠评估框架:面向性能与评估有效性的量化方案
Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment
AI 导读
一项 arXiv 论文提出面向投资研究任务的 AI 智能体评估框架,将结果质量与执行质量区分开,通过任务特定标准、确定性验证器和独立大语言模型评审来适配不同任务要求。
正文 · 原文
Abstract:We consider the problem of evaluating AI agents undertaking investment-research tasks, where agents interact with external information sources and computational tools to produce financial analyses and research reports. An important challenge is that the quality of an agent's final output does not necessarily reflect the quality of its execution process, while the reliability of the resulting assessment also depends on the validity of the evaluation procedure. Furthermore, investment-research tasks differ considerably in their requirements for numerical accuracy, information retrieval, financial analysis and supporting evidence, making it difficult to assess agent performance using a uniform set of criteria. We propose a domain-oriented evaluation framework that distinguishes outcome quality from execution quality and accommodates these differences through task-specific criteria, deterministic validators and independent large language model judges. We also introduce an evaluation-validity mechanism that examines the integrity of evaluation inputs, reference information, supporting evidence and grading outputs, with expert review used to resolve identified deficiencies and significant grading disagreements. Numerical results show that the proposed framework can distinguish differences in agent performance across evaluation dimensions, while identifying potential reliability issues in the evaluation process. The associated implementation and evaluation materials will be made available on GitHub.
| Comments: | 21 pages, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22338 [cs.AI] |
| (or arXiv:2611.22338v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22338 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongnuan Tian [view email]
[v1]
Tue, 17 Nov 2026 07:40:37 UTC (3,390 KB)
来源:arXiv cs.AI · arxiv.org