LLM 数据智能体的轨迹完整性：超越答案准确率的结构化计算审计
Trace Integrity for LLM Data Agents: Auditing Structured-Data Computation Beyond Answer Accuracy in Real-World Systems
AI 导读
一项 arXiv 论文提出 Trace Integrity（轨迹完整性）标准，用于评估 LLM 数据智能体在结构化数据问答中的计算过程是否显式、可执行、可回放且与答案一致，并定义衡量“答案正确但轨迹无效”比例的 CAIT Rate。
正文 · 原文
Abstract:Large language model (LLM) data agents answer structured-data questions by generating SQL, invoking tools, executing queries, and summarizing results. In operational settings, reliability depends on both reported answer and computation used to produce it, as errors may remain undetected when the resulting answer matches an expected value. We refer to the mismatch between natural-language intent and operator-level computation required to satisfy that intent as the Structure Gap, and define Trace Integrity as a reliability criterion for assessing whether the computation supporting an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We operationalize Trace Integrity through execution contracts that record the schema elements, operator plan, assumptions, executable query, verification status, and final-answer linkage associated with a request. We also define the Correct Answer / Invalid Trace (CAIT) Rate, which measures the proportion of correct answers whose recorded computations fail trace-integrity checks. These provide a way to evaluate whether answer-level success is supported by a valid and inspectable computation. We study this on 100 BIRD Mini-Dev text-to-SQL examples using three prompting conditions: Direct SQL, Operation Summary + SQL, and Contract-First SQL. The three conditions achieve answer accuracies 20.0%, 22.0%, and 24.0%, Trace Integrity Pass Rates 39.0%, 43.0%, and 40.0%, and CAIT Rates 55.0%, 59.1%, and 45.8%, respectively. The results show that answer accuracy, trace validity, and the frequency of correct answers supported by invalid traces capture different aspects of system behavior. Trace-level evaluation therefore complements conventional answer-based evaluation by providing evidence about the computational support, inspectability, and reproducibility of LLM data-agent outputs in real-world systems.
| Comments: | Accepted at AAAI FSS Trustworthy Agentic Systems (TAS) 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.26036 [cs.AI] |
| (or arXiv:2608.26036v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.26036 arXiv-issued DOI via DataCite |
Submission history
From: Srimonti Dutta [view email]
[v1]
Wed, 26 Aug 2026 17:15:24 UTC (139 KB)
[v2]
Fri, 23 Oct 2026 19:29:53 UTC (182 KB)
来源：arXiv cs.AI · arxiv.org