MINT:面向多步 off-policy RL 截断的会面时间指示器
MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL
AI 导读
强化学习研究提出 MINT(Meeting-time INdicators for Truncation),用二元停止指示器替代多步 off-policy 时序差分方法中的逐步 trace 系数乘积,以缓解方差控制与长时序信号之间的权衡。
正文 · 原文
Abstract:Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
| Comments: | Accepted at NeurIPS 2026. 81 pages, 40 figures, including appendices. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.24413 [cs.LG] |
| (or arXiv:2611.24413v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24413 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Seungyub Han [view email]
[v1]
Wed, 18 Nov 2026 13:40:04 UTC (2,788 KB)
来源:arXiv cs.AI · arxiv.org