潜空间 Q-Barrier 屏蔽方法提升安全 In-Context 强化学习部署表现
Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning
AI 导读
研究者提出一种 latent Q-Barrier shield 方法,针对安全 in-context 强化学习在 OOD 部署偏移下奖励-安全权衡不佳的问题,在部署前学习上下文表示、隐空间动力学和集成 cost critic,无需参数更新即可依据剩余安全预算和预测未来成本过滤或软重加权候选动作。
正文 · 原文
Abstract:Safe in-context reinforcement learning (ICRL) adapts online from interaction history without test-time parameter updates while controlling episode cost under a safety budget. Under out-of-distribution (OOD) deployment shifts, pretraining-only safe ICRL can give poor reward-safety tradeoffs because the remaining budget affects behavior only through frozen policy conditioning, not an explicit action-level check against predicted future cost. We propose a latent Q-Barrier shield that learns a context representation, latent dynamics, and an ensemble cost critic before deployment. Without parameter updates, the shield infers context from history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. We prove a conditional one-step barrier-margin bound over realized candidate sets, with explicit Bellman and latent-prediction error terms. Across five safe ICRL benchmarks, the shield improves deployment-time reward-safety tradeoffs over a strong safe-ICRL baseline: after a short context window, it achieves higher return in four of five benchmarks while matching or lowering average episode cost in all five. Code is available at this https URL.
| Comments: | Accepted at NeurIPS 2026. Camera-ready version |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.25267 [cs.LG] |
| (or arXiv:2605.25267v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25267 arXiv-issued DOI via DataCite |
Submission history
From: Minjae Kwon [view email]
[v1]
Sun, 24 May 2026 21:45:28 UTC (355 KB)
[v2]
Mon, 16 Nov 2026 21:56:38 UTC (337 KB)
来源:arXiv cs.AI · arxiv.org