面向 LLM 智能体的扰动鲁棒策略学习：SPrPO 实现稳定优化
Learning Perturbation Robust Policies for LLM Agents with Stable Optimization
AI 导读
arXiv 论文提出稳定扰动鲁棒策略优化方法 SPrPO，用于提升强化学习训练的 LLM 智能体策略对隐藏状态噪声、剪枝和量化等扰动的鲁棒性。该方法在策略优化过程中施加自适应、敏感性感知的扰动，并分析了扰动策略更新保持稳定单调改进的条件。在 ALFWorld 和 WebShop 基准上的多类型、多尺度实验显示，SPrPO 在提升扰动鲁棒性的同时保持了稳定的策略优化。
正文 · 原文
Abstract:Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.34064 [cs.LG] |
| (or arXiv:2609.34064v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.34064 arXiv-issued DOI via DataCite |
Submission history
From: Pengxin Wang [view email]
[v1]
Mon, 28 Sep 2026 00:43:16 UTC (5,674 KB)
[v2]
Tue, 10 Nov 2026 05:09:29 UTC (5,674 KB)
来源：arXiv cs.AI · arxiv.org