AgenticRAG-FP：智能体 RAG 中因果失败归因的干预式基准

View PDF HTML (experimental)

Abstract:Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention site. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the dense Claude Haiku 4.5 structural sweep on three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories), but it predicts hop 1 in 96% of failures, an earliest-hop rule; at hops 2 and 3, an LLM judge and frozen-hop repair localize only 0.25 to 0.48 of failures. In larger content-corruption runs, depth-2 exact-hop accuracy is 0.00 for coverage, 0.49 for an LLM judge, 0.59 for frozen-hop repair, and 0.14 for suffix regeneration (n=51); the judge and frozen-hop repair do not significantly differ (exact McNemar p=0.33).

Comments:	12 pages. v2: updated author list and results to match the camera-ready version for the REALM Workshop at EMNLP 2026. Work done as part of the Anote AI Research Fellowship
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2608.20627 [cs.CL]
	(or arXiv:2608.20627v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2608.20627 arXiv-issued DOI via DataCite

Submission history

From: Lauren Pothuru [view email]
[v1] Thu, 20 Aug 2026 23:56:34 UTC (48 KB)
[v2] Tue, 10 Nov 2026 23:49:40 UTC (51 KB)