arXiv cs.AI· Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, Guanghao Li, Huatong Song, Yining Li, Yifan Luo, Kunbao Ding, Feng Chang, Chuan Hao, Ran Tao, Bryan Dai, Jiafeng Liang, Jinyang Jiang·· 2 小时前AI 评分34
多尺度策略优化（MPO）提升长程 LLM 智能体训练效果
Multiscale Policy Optimization for Long-Horizon LLM Agents
AI 导读
一项 arXiv 研究提出多尺度策略优化（MPO），针对长程 LLM 智能体在回合（turn）尺度上进行优势估计（MAE）与回合级重要性采样，以改进 token 间的信用分配。在 4B 和 30B-A3B 模型上，MPO 于搜索、工具使用和自动化任务中均一致超越强 RL 基线。
正文 · 原文
Authors:Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, Guanghao Li, Huatong Song, Yining Li, Yifan Luo, Kunbao Ding, Feng Chang, Chuan Hao, Ran Tao, Bryan Dai, Jiafeng Liang, Jinyang Jiang
Abstract:Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35013 [cs.AI] |
| (or arXiv:2610.35013v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35013 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tao Ren [view email]
[v1]
Mon, 26 Oct 2026 08:22:11 UTC (1,482 KB)
来源：arXiv cs.AI · arxiv.org