用大规模并行加速安全强化学习
Accelerating Safe Reinforcement Learning with Massive Parallelism
AI 导读
一篇 NeurIPS 2026 论文研究如何把安全强化学习扩展到大规模并行训练场景，即大量环境以短 rollout 视界并行执行。作者将交错环境重置建模为短轨迹段的相位混合分布，提出相位聚合代价估计器从短 rollout 批次重建 episode 级代价估计，并结合有限样本不确定性与分布不匹配设计约束收紧框架。
正文 · 原文
Abstract:We investigate the problem of scaling safe reinforcement learning (safe RL) to massively parallel training regimes, where a large number of environments are exe- cuted with short rollout horizons. Although this setting improves data throughput and wall-clock efficiency, it creates a mismatch with constrained Markov decision processes (CMDPs), whose safety constraints are defined over full-episode cost returns. We address this mismatch by analyzing staggered environment resets as a phase-mixture distribution over short trajectory segments. This analysis charac- terizes the bias induced by phase coverage and motivates a phase-aggregated cost estimator that reconstructs episode-level cost estimates from short rollout batches. We further incorporate finite-sample uncertainty and distribution mismatch into a constraint-tightening framework, yielding a scalable Safe RL method compatible with on-policy optimization. Using an MJX-based implementation of Safety Gym, our experiments show that the proposed method achieves performance comparable to non-massively parallel implementations while substantially reducing wall-clock training time.
| Comments: | Accepted at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.08273 [cs.LG] |
| (or arXiv:2611.08273v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.08273 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joonyoung Lim [view email]
[v1]
Thu, 5 Nov 2026 16:43:57 UTC (1,450 KB)
来源：arXiv cs.AI · arxiv.org