arXiv cs.AI· Amey Rambatla, Vatsal Goel, Sagar Gaur, Siddhant Loya, Anurag Mishra, Asif Mohammad·· 4 小时前AI 评分40
Signals：AI 智能体问题的持续诊断方法
Signals: Continuous Diagnostics of AI Agents
AI 导读
论文提出 Signals，一种从智能体执行轨迹中识别错误决策、资源浪费等问题的持续诊断方法。该方法基于 8 大类、45 种问题类型的分类体系，区分已发生故障、风险暴露和低效问题，将相关发现归类为可执行的工程问题并评估严重程度。在公开智能体执行数据上评估达到 91.43% F1，比使用相同模型和分类体系的单次 LLM 基线高约 18 个百分点。
正文 · 原文
Abstract:AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. We developed Signals to identify such problems from agentic traces and suggest concrete remediations. We built the method around a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. The method distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. We evaluated the method on public agent executions and achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy.
| Comments: | 11 pages, 1 figure, 6 tables. Author affiliation formatting corrected; scientific content unchanged |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.04507 [cs.AI] |
| (or arXiv:2611.04507v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.04507 arXiv-issued DOI via DataCite |
Submission history
From: Amey Rambatla [view email]
[v1]
Tue, 3 Nov 2026 08:32:43 UTC (51 KB)
[v2]
Wed, 4 Nov 2026 06:09:13 UTC (51 KB)
来源：arXiv cs.AI · arxiv.org