RetryBench：将重试权限与恢复结果分离的工具调用评测协议
RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use
AI 导读
RetryBench 是一个针对调用外部工具的 LLM 智能体的重试决策与故障恢复评测协议，按重试权限、已生效副作用和任务完成度三项指标评分，并在四个合成软件工程任务上进行了 1,728 次决策的提示词敏感性研究。
正文 · 原文
Abstract:LLM agents that call external tools must decide what to do when a tool call times out: retrying risks duplicate side effects, while not retrying can leave the task incomplete. We present RetryBench, an evaluation protocol for retry decisions and failure recovery in tool-using LLM agents. Each decision is scored on retry permission (is the retry justified by an idempotency-key guarantee or proof of non-execution?), settled effects (duplicates and omissions), and completion, by replaying it across all hidden execution worlds consistent with the observation. On four synthetic software-engineering tasks, we run two controlled prompt-sensitivity studies (1,728 decisions). Study 1 (idempotency-key descriptions): with the contract fixed, GPT-5.6 Luna reuses the valid idempotency key in 10/40 decisions when an extra key is described as a "Candidate recovery key" but in 39/40 when it is an "Alternative key string"; across three further phrasings per family the gap is 27/120 versus 113/120. GPT-5.6 Sol and Terra are at ceiling (40/40). Study 2 (action order): under underspecified lookup guarantees, retry rates are 17-27 points lower for GPT-5.6 Terra and GPT-5.4 when retry is listed first (option-order bias), mainly on two tasks; we make no claim about which behavior is better. Findings are limited to researcher-chosen phrasings, four tasks and unpinned model aliases. Code and all model responses are released.
| Comments: | LLM agents, tool use, idempotency, retry, failure recovery, evaluation, prompt sensitivity, position bias, reliability |
| Subjects: | Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.15884 [cs.AI] |
| (or arXiv:2610.15884v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15884 arXiv-issued DOI via DataCite |
Submission history
From: You Liu [view email]
[v1]
Sun, 11 Oct 2026 20:00:44 UTC (80 KB)
[v2]
Sun, 18 Oct 2026 00:55:36 UTC (311 KB)
[v3]
Thu, 22 Oct 2026 06:44:33 UTC (312 KB)
[v4]
Wed, 4 Nov 2026 07:00:04 UTC (310 KB)
来源：arXiv cs.AI · arxiv.org