Deep Double Q-learning（DDQL）：显式双 Q 网络缓解深度 RL 过估计
Deep Double Q-learning
AI 导读
研究者提出深度强化学习算法 Deep Double Q-learning（DDQL），通过 Double Q-learning 目标显式训练两个 Q 函数，克服 Double DQN 单一 Q 函数、估计器相关导致的过估计问题。
正文 · 原文
Abstract:Double Q-learning is a classical control algorithm that mitigates the maximization bias of Q-learning. To do so, it explicitly trains two independent action-value functions and uses them to decouple action-selection from action-evaluation when computing bootstrap targets. Double DQN adapts this decoupling to deep reinforcement learning (RL) but directly trains only a single action-value function and does not fully decouple its estimators. Consequently, the two estimators remain correlated, and overestimation persists. In this paper, we introduce Deep Double Q-learning (DDQL), a deep RL algorithm that explicitly trains two Q-functions through the Double Q-learning objective. DDQL stabilizes training through a combination of techniques, including lower replay ratios, longer target network update intervals, and shared layers. Across 57 Atari 2600 games, DDQL improves aggregate performance over Double DQN, outperforming it on 47 games while further reducing overestimation. We study key design choices of DDQL, including its network architecture, replay ratio, and minibatch sampling strategies.
| Comments: | Neural Information Processing Systems (NeurIPS), 2026; |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2507.00275 [cs.LG] |
| (or arXiv:2507.00275v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2507.00275 arXiv-issued DOI via DataCite |
Submission history
From: Prabhat Nagarajan [view email]
[v1]
Mon, 30 Jun 2025 21:32:46 UTC (5,968 KB)
[v2]
Thu, 14 May 2026 20:49:30 UTC (6,356 KB)
[v3]
Mon, 9 Nov 2026 21:12:50 UTC (7,574 KB)
来源：arXiv cs.AI · arxiv.org