面向受监管企业的 Agentic AI 平台能力评估框架
A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises
AI 导读
作者提出一个评估受监管企业中 Agentic AI 平台的能力框架,基于某全球银行创新实验室的实测,对覆盖开源框架、专有产品和超大规模云生态的十个平台,在交易对手信用风险、检索增强生成和流程自动化三类金融工作负载上按九个能力维度评分。
推荐理由
论文基于某全球银行对十个智能体平台的实测,指出演示式评分存在光环效应,并给出先门控后评分的评估协议。
正文 · 原文
Abstract:Agentic AI platforms-systems in which LLM agents plan, invoke tools, retain memory and coordinate to change the state of enterprise systems-are being procured faster than they can be measured. Public agent benchmarks score models in fixed harnesses; enterprises buy integrated platforms whose orchestration, memory, governance and tooling layers dominate outcomes, and they buy them largely on the strength of vendor demonstrations. We report an industrial evaluation programme ("bakeoff") run by the innovation laboratory of a global bank in which ten platforms-spanning open-source frameworks, proprietary products and hyperscaler ecosystems-were assessed by a cross-functional panel on nine capability dimensions across three financial-services workloads: counterparty credit risk, retrieval-augmented generation and process automation. Scores separate the field into three tiers that are stable under alternative weightings, and surface workload-specific strengths that aggregates hide. A secondary analysis, however, finds Cronbach's alpha = 0.98 across nominally distinct dimensions and rank correlations >= 0.93 with the total for eight of nine: a halo effect indicating that demonstration-based scoring measures presentation as much as capability. From these lessons we propose a gate-then-score evaluation protocol with four layers (capability, execution reliability, governance gates, business value), trace-level reliability metrics estimated over repeated runs, an autonomy-readiness scale and a reproducibility protocol. We state precisely what the present evidence supports, and what a controlled harness must add to turn enterprise platform evaluation from structured judgement into measurement.
| Subjects: | Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Machine Learning (stat.ML) |
| Cite as: | arXiv:2611.24039 [cs.AI] |
| (or arXiv:2611.24039v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24039 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Abbas Raza Ali [view email]
[v1]
Wed, 18 Nov 2026 08:28:02 UTC (221 KB)
来源:arXiv cs.AI · arxiv.org