ArbiGraph：可任意扩展的可验证任务图基准，用于评估上下文管理
ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management
AI 导读
研究者推出 ARBIGRAPH，一个评估工具辅助语言智能体在长推理流程中保留、更新、组合和丢弃上下文的基准生成器。它将任务表示为带可执行 Python 求解器的自然语言问题，通过类型化中间状态组合任务，支持控制图长度、依赖结构、干扰项数量和值类型，并实现精确自动验证。
正文 · 原文
Abstract:We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows. ARBIGRAPH represents each task as a natural-language problem with an executable Python solver, and composes tasks through typed intermediate states, instantiated here as scalar and list values. This design enables controllable task graphs whose length, dependency structure, distractor count, and value type can be varied while preserving exact automatic verification. We instantiate ARBIGRAPH with math, GSM-style word-problems, and Python-tracing task categories, and evaluate a Qwen3.5-27B tool-assisted agent across four topologies. The results show high accuracy on isolated tasks but substantial degradation on more complex dependent tasks: accuracy drops by up to 33.3% on branching chains of dependent math tasks. This shows that ARBIGRAPH exposes failures that are not visible from single-task evaluation alone. Our code, generated datasets, and evaluation results are available at this https URL
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.20764 [cs.AI] |
| (or arXiv:2607.20764v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.20764 arXiv-issued DOI via DataCite |
Submission history
From: Pavel Golikov [view email]
[v1]
Wed, 22 Jul 2026 22:30:12 UTC (265 KB)
[v2]
Sat, 24 Oct 2026 08:19:20 UTC (193 KB)
来源：arXiv cs.AI · arxiv.org