arXiv cs.AI· Siyuan Du, Yuxin Liu, Meng Tang, Haoran Li, Yechao Bai, Feiyang Pan, Jia He·· 1 小时前AI 评分33
先深后广:逆向到正向课程与优势分解训练 LLM Agent
Depth Before Breadth: Reverse-to-Forward Curriculum with Advantage Decomposition for LLM Agent
AI 导读
一篇 arXiv 论文提出用于训练大语言模型 Agent 的策略诱导能力区域方法,将长程任务学习分解为到达更可解状态与从该状态完成任务两个子问题,并用配对优势和逆向到正向渐进课程分别提供信号。
正文 · 原文
Abstract:Reinforcement learning (RL) has become a common approach for training large language model (LLM) agents on long-horizon tasks. However, learning from sparse terminal rewards makes credit assignment and exploration difficult. We approach these challenges through a policy-induced capability region, the set of states from which the current policy can reliably complete a task. Near its boundary, an agent may need only a few actions to reach a more solvable state, even when the goal remains distant. This reduces long-horizon learning to two coupled subproblems of reaching that state and completing the task from there. We address them with paired advantages and a progressive reverse-to-forward curriculum. The advantages provide separate signals for reaching more solvable states and completing the task from them, while the curriculum expands the capability region backward along successful reference trajectories and then into off-reference states through on-policy exploration. On ALFWorld and WebShop, our method outperforms competitive agentic RL methods at both model scales, achieving success rates of 90.1%/96.1% and 80.2%/82.6% with Qwen2.5-1.5B/7B-Instruct, respectively.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24697 [cs.AI] |
| (or arXiv:2611.24697v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24697 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Meng Tang [view email]
[v1]
Wed, 18 Nov 2026 16:25:14 UTC (1,226 KB)
来源:arXiv cs.AI · arxiv.org