MiRL：面向终端智能体错误行为的定向强化学习方法
MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors
AI 导读
arXiv 论文提出 MiRL（Misbehavior-driven RL），一种面向终端智能体的错误定向强化学习方法：先对策略进行错误画像找出可泛化的错误行为，再从这些错误发生的步骤启动 rollout，仅在通过测试且避免错误时给予奖励。
正文 · 原文
Abstract:Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.
| Comments: | 20 pages, 14 figures, 5 tables. Project page: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.35697 [cs.LG] |
| (or arXiv:2610.35697v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35697 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Runchu Tian [view email]
[v1]
Mon, 26 Oct 2026 16:25:05 UTC (1,079 KB)
来源：arXiv cs.AI · arxiv.org