长程 LLM 智能体错误溯源实证研究：95 条失败轨迹分析错误何时进入及哪些控制可覆盖
Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them
AI 导读
arXiv 论文（arXiv:2611.13254）提出 episode 记录标注协议，完整阅读 Who&When 和 MAD 语料中 95 条失败轨迹，定位长任务 LLM 智能体故障的进入点、传播与检测缺口。
正文 · 原文
Abstract:When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried it to the symptom, whether anything in the trajectory contradicted it, or which control could have caught it. We define an episode record that ties a candidate fault to its origin, propagation, detection, response and consequence, located by operational phase and state domain, and that names the control families whose observation and reach cover it. We turned the record into a written annotation protocol and applied it to 95 failed runs of six multi-agent systems from the public Who&When and MAD corpora, reading each trajectory in full. Faults enter early (median at 0.18 of the run), mostly as wrong decisions or fabrications over task state, and in 37 of 88 Who&When runs before the step the corpus' human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error before the failure; in 88% of those no agent acted on it. Something flagged a deviation in 31% of runs, but no response removed the deviated state. Independent judges reading the unused evidence, task contracts fixed at admission, and re-checking beliefs against the environment reach most episodes; restoring the agent's own state would have sufficed in 35 of 95 runs. A text scan of the raw corpus also shows that 45% of the hand-crafted Who&When failures ended with a provider content-filter crash after which the harness emitted a final answer no agent produced, a failure class that neither symptom taxonomies nor responsible-agent labels represent. Protocol, records and scripts are included.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13254 [cs.AI] |
| (or arXiv:2611.13254v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13254 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shaohuang Wang [view email]
[v1]
Mon, 9 Nov 2026 21:25:08 UTC (235 KB)
来源：arXiv cs.AI · arxiv.org