arXiv cs.AI· Mengsong Wu, Tianchen Zhou, Zhaochen Su, Zhenhua Liu, Zhe Cui, Aimin Zhou·· 4 小时前AI 评分41
AttnSR：基于注意力步级奖励的长程智能体强化学习方法
AttnSR: Attention Step Reward for Long-Horizon Agentic RL
AI 导读
研究者提出 AttnSR，一种步级奖励增强方法，从策略模型训练回放中提取注意力来分配步级奖励，无需单独训练奖励模型或额外 rollout。
正文 · 原文
Abstract:Long-horizon LLM agents make multi-step decisions, yet outcome-reward-only reinforcement learning provides little direct supervision for individual steps. We introduce AttnSR, a step-level reward augmentation method that derives reward allocations from policy attention extracted during training replay, without a separately trained reward model or additional rollouts. AttnSR traces backward through the policy model attention graph to estimate the relevance of earlier steps to the final response. The terminal step, measured relative to a running baseline, determines whether these rewards reinforce or penalize the selected steps. In a controlled Qwen2.5-7B-Instruct study, AttnSR consistently outperforms the PPO baseline across all evaluated checkpoints on ALFWorld, raising final held-out success from 91.2% to 97.1%. The ablation experiment shows that the benefit depends on how intermediate rewards are assigned, rather than simple accumulation of step rewards over time. AttnSR also improves ScienceWorld at every checkpoint while cutting environment interactions by 29.9%. These results support policy attention as a practical prior for step-level reward allocation in long-horizon agentic RL. Code is available at this https URL .
| Comments: | 20 pages, 4 figures, including appendices. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07932 [cs.LG] |
| (or arXiv:2611.07932v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07932 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mengsong Wu [view email]
[v1]
Thu, 5 Nov 2026 12:37:19 UTC (371 KB)
来源：arXiv cs.AI · arxiv.org