迈向面向连续控制的批处理到流式深度强化学习
Towards Batch-to-Streaming Deep Reinforcement Learning for Continuous Control
AI 导读
论文提出两种流式深度强化学习算法 S2AC 与 SDAC，兼容最先进的批处理 RL 方法，适用于 Sim2Real 迁移等设备端微调场景。两者在标准基准上性能与最先进流式基线相当，且无需逐环境繁琐调参。研究还表明朴素的批处理到流式转换不能保证保留预训练策略性能，并提出一种原则性解决方法。
正文 · 原文
Abstract:State-of-the-art deep reinforcement learning (RL) methods have achieved remarkable performance in continuous control tasks, yet their computational complexity is often incompatible with the constraints of resource-limited hardware, due to their reliance on replay buffers, batch updates, and target networks. The emerging paradigm of streaming deep RL addresses this limitation through purely online updates, achieving strong empirical performance on standard benchmarks. In this work, we propose two novel streaming deep RL algorithms, Streaming Soft Actor-Critic (S2AC) and Streaming Deterministic Actor-Critic (SDAC), explicitly designed to be compatible with state-of-the-art batch RL methods, making them particularly suitable for on-device finetuning applications such as Sim2Real transfer. Both algorithms achieve performance comparable to state-of-the-art streaming baselines on standard benchmarks without requiring tedious per-environment hyperparameter tuning. We further investigate the batch-to-streaming transition, showing that a naive transition does not guarantee preservation of pre-trained policy performance, and propose a principled approach to address this challenge.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.08588 [cs.LG] |
| (or arXiv:2603.08588v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.08588 arXiv-issued DOI via DataCite |
Submission history
From: Matteo Cederle [view email]
[v1]
Mon, 9 Mar 2026 16:40:06 UTC (7,773 KB)
[v2]
Mon, 11 May 2026 08:11:49 UTC (8,660 KB)
[v3]
Thu, 29 Oct 2026 17:11:53 UTC (8,668 KB)
来源：arXiv cs.AI · arxiv.org