跳到正文
arXiv cs.AI· Moriya Dechtiar, Daniel Martin Katz, Michael J. Bommarito II, Hongming Wang, Sylvain Jaume·· 2 小时前AI 评分23

RagHarnessDebugger:Jev 引导的智能体 RAG 合同推理修复审计框架

RagHarnessDebugger: Jev-Guided Agentic RAG in Contract Inference Repair Auditing

AI 导读

研究者提出 RagHarnessDebugger,一个以 Jev 引导的智能体 RAG 审计框架,记录并调试合同推理修复过程中的每个决策。在 ContractNLI 保密协议数据上,审计层揭示了聚合指标下隐藏的失败模式,政策接受缺口在开发集与复制集分别为 77% 和 76%。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agentic systems using retrieval-augmented generation (RAG) must repair their own answers by gathering more evidence, testing alternative policies and deciding if, when, and what repairs are needed. Yet how do we know if a system's repairs are working when metrics like accuracy can be misleading? We present RagHarnessDebugger, a framework that records every repair decision in a Jev-guided harness for contract inference and audits the record to facilitate complete debugging of all decisions taken during a RAG repair journey. RagHarnessDebugger checks that each trace is intact, that each validation applies to the candidate it is counted for, and which answers the acceptance rule can reach. We tested our approach on ContractNLI non-disclosure agreements with two reader models and a replication set of separate contracts. Our debugger's audit layer exposed failure patterns that were hidden under aggregate metrics, policies acceptance cost of repairs vs regressions and policy acceptance gap of 77% and 76% of available but unrealised repairs on development and replication sets. Our audit, together with a labeled error analysis and replay mechanism, allowed following the execution path all the way to the terminal decision - uncovering where missing repairs became lost. Our revised harness improved accuracy over the initial BM25-only baseline by 3.6 points on development contracts and 3.7 points on the replication set, and even improved our full contract access reference score by 1.1 on development and 2.4 on replication set. While our work here demonstrates how the debugger can conduct decision root-cause analysis for missed repairs, the true value lies with providing this debugger for judgment visibility serving continuous improvement of RAG decisions and their explainability.
Keywords: agentic RAG, AI evidence repair, harness explainability, counterfactual replay, Jev, ContractNLI
Subjects: Artificial Intelligence (cs.AI); Multiagent Systems (cs.MA); Software Engineering (cs.SE)
Cite as: arXiv:2611.22991 [cs.AI]
  (or arXiv:2611.22991v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.22991

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Moriya Dechtiar [view email]
[v1] Tue, 17 Nov 2026 16:07:17 UTC (228 KB)

来源:arXiv cs.AI · arxiv.org