延迟同态强化学习（DHRL）：面向延迟反馈环境的统一状态抽象框架

View PDF HTML (experimental)

Abstract:Reinforcement learning in real-world systems often involves delayed feedback, which violates the Markov assumption and impedes both learning and control. Canonical augmentation-based approaches address this issue by augmenting the state with action histories, but suffer from state-space explosion and the resulting sample-complexity burden. Despite recent progress, state-of-the-art augmentation-based methods either primarily alleviate this burden for the critic or employ separate mechanisms for the actor and critic. To address this limitation, we propose delayed homomorphic reinforcement learning (DHRL), a framework grounded in MDP homomorphisms that introduces a belief-equivalence relation over the augmented state space to collapse control-redundant augmented states into a compact abstract state space. This framework yields exact abstraction under deterministic dynamics and approximate abstraction under stochastic dynamics, providing a unified abstraction mechanism that benefits both the actor and critic. In finite domains, exact abstraction preserves optimality and recovers the delay-free sample-complexity order, while approximate abstraction admits a bound on the value loss of the resulting policy. For continuous domains, we introduce deep delayed homomorphic policy gradient (D$^2$HPG), a deep actor-critic instantiation of the DHRL framework. Experiments on MuJoCo continuous-control benchmarks demonstrate that D$^2$HPG outperforms strong augmentation-based baselines, especially in long-delay regimes. Code is available at this https URL.

Submission history

From: Jongsoo Lee [view email]
[v1] Sat, 4 Apr 2026 08:38:52 UTC (1,549 KB)
[v2] Sat, 2 May 2026 05:53:33 UTC (2,406 KB)
[v3] Wed, 11 Nov 2026 12:51:31 UTC (1,014 KB)