arXiv cs.AI· Jiamin He, Mark Rowland, Zhaohan Daniel Guo, Hado van Hasselt, Diana Borsa·· 3 小时前AI 评分38
基于 excursion 策略的离策略强化学习方法研究
Off-policy Learning with Excursion Policies
AI 导读
一篇 NeurIPS 2026 论文提出 excursion 策略：智能体先按目标策略执行多步"excursion"，再回到行为策略，作为策略线性插值之外的结构化替代方案。作者推导了此类策略的动态规划结果，并给出可证明可靠的离策略 TD 算法用于 excursion 价值估计。在连续控制基准上，excursion 策略达到与混合策略相当的效率-稳定性权衡。
正文 · 原文
Abstract:Off-policy reinforcement learning (RL) tries to learn the value of a target policy while data comes from a different behavior policy. This often requires trading off bias for variance or stability. One way to trade these off is to linearly interpolate between the behavior and target policies, but this represents only one point in a broader space of interpolation mechanisms. We introduce excursion policies--a structured alternative where the agent follows the target policy for a multi-step "excursion" before reverting to the behavior policy. We derive dynamic programming results for these policies, establishing the theoretical basis for their use in RL. We further develop provably sound off-policy temporal-difference (TD) algorithms for excursion value estimation. We find that excursion policies provide an efficiency-stability trade-off comparable to mixture policies in continuous control benchmarks, and our analysis illustrates conceptual differences in their behavior.
| Comments: | Accepted to NeurIPS 2026 (camera-ready) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13275 [cs.LG] |
| (or arXiv:2611.13275v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13275 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiamin He [view email]
[v1]
Mon, 9 Nov 2026 21:41:23 UTC (1,742 KB)
来源：arXiv cs.AI · arxiv.org