闭环智能体评估：分层测量与自校准方法（arXiv 论文）
Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration
AI 导读
一篇 arXiv 论文提出闭环评估（closed-loop evaluation）方法，用于评估已部署的 AI 智能体的实际工作效果，将评估建模为有偏评估者和删失结果下的性能估计。
正文 · 原文
Abstract:Once an agent is live, teams want to know whether it is doing its job, not just whether it throws errors. Yet most evaluation is one-off: a benchmark run, a look at traces, a single score. Its conclusions neither check themselves nor reach the agent. We propose closed-loop evaluation, an evaluation method for deployed agents. It models evaluation as performance estimation under biased raters and censored outcomes, and it requires every level's conclusion to trace back to atomic data and to be computed by versioned rules. The method covers four levels of analysis (span, trace, conversation and engagement), measures along a spine of Execution, Substance, Trajectory and Results, treats Conduct as a constraint throughout, and uses Provenance and Adjudication components to certify the evidence and the evaluators. Evaluator agents generated by an Agent Factory carry these components, each with a domain knowledge pack and an adapter harness. Two loops close the evaluation: an inner loop audits the evidence, rules and metrics themselves, and an outer loop uses per-component regression checks to drive agent improvement. We also give a pipeline that runs this process automatically. Using the business-analysis agent SharpInsight as a case and porting the method to public conversation data, we show how problems differ across levels and how evidence-and-rule judgment is more reliable than holistic scoring (ICC 0.86 vs. 0.81; kappa 0.81 vs. 0.63).
| Comments: | 7 pages |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.07051 [cs.AI] |
| (or arXiv:2611.07051v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07051 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gun Ahn [view email]
[v1]
Wed, 4 Nov 2026 19:11:47 UTC (136 KB)
来源：arXiv cs.AI · arxiv.org