跳到正文
arXiv cs.AI· Seungyub Han, Taehyun Cho, Dohyeong Kim, Kyungjae Lee, Jungwoo Lee·· 2 小时前AI 评分25

MINT:面向多步 off-policy RL 截断的会面时间指示器

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

AI 导读

强化学习研究提出 MINT(Meeting-time INdicators for Truncation),用二元停止指示器替代多步 off-policy 时序差分方法中的逐步 trace 系数乘积,以缓解方差控制与长时序信号之间的权衡。

正文 · 原文

View PDF HTML (experimental)

Abstract:Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.
Comments: Accepted at NeurIPS 2026. 81 pages, 40 figures, including appendices. Code: this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24413 [cs.LG]
  (or arXiv:2611.24413v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24413

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Seungyub Han [view email]
[v1] Wed, 18 Nov 2026 13:40:04 UTC (2,788 KB)

来源:arXiv cs.AI · arxiv.org