arXiv cs.AI· Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao·· 2 小时前AI 评分48
企业级分析智能体评估框架对比研究：稳定性、风险检测与发现
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
AI 导读
一项发表于 KDD 2026 的三阶段对比研究，评估了三类面向企业级 LLM 分析智能体的评估框架：基于 trace 的 LLM 评审、纯文本 LLM 评审结合红队测试，以及确定性启发式规则。
正文 · 原文
Abstract:Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
| Comments: | Accepted to KDD 2026, Jeju, Korea. 9 pages, 3 figures |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.14493 [cs.AI] |
| (or arXiv:2611.14493v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14493 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Howard Zhang [view email]
[v1]
Tue, 10 Nov 2026 18:50:06 UTC (283 KB)
来源：arXiv cs.AI · arxiv.org