Annotrace：面向智能体轨迹步骤级标注的工作台
Annotrace: A Workbench for Step-Level Annotation of Agent Traces
AI 导读
arXiv 论文 Annotrace 是一个智能体轨迹步骤级标注工作台，支持 OpenTelemetry GenAI、LangSmith、Langfuse 等六种轨迹格式，将每次运行转为有序嵌套的步骤列表。
正文 · 原文
Abstract:Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $\kappa = 0.82$.
| Comments: | 4 pages, 4 figures, 1 table. Under review at the AAAI 2027 Demonstration Program |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Multiagent Systems (cs.MA) |
| ACM classes: | I.2.11; H.5.2 |
| Cite as: | arXiv:2610.23284 [cs.AI] |
| (or arXiv:2610.23284v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.23284 arXiv-issued DOI via DataCite |
Submission history
From: Arun Josephraj Arokiaraj [view email]
[v1]
Fri, 16 Oct 2026 10:55:26 UTC (370 KB)
[v2]
Mon, 19 Oct 2026 10:52:20 UTC (371 KB)
[v3]
Tue, 27 Oct 2026 17:51:39 UTC (371 KB)
来源：arXiv cs.AI · arxiv.org