基于近似下层响应的约束双层强化学习研究
Constrained Bilevel Reinforcement Learning via Approximate Lower-Level Responses
AI 导读
一篇 arXiv 论文研究约束双层强化学习，其中下层问题为带一般参数化策略的约束马尔可夫决策过程，下层解映射可为集值且不满足 Lipschitz 连续。作者定义基于固定随机下层算法返回策略的“近似响应”上层目标，通过随机化下层训练中的奖励与约束值（不改变均值）得到无偏梯度估计，无需对下层更新求导。
正文 · 原文
Abstract:We study constrained bilevel reinforcement learning in which the lower-level problem is a constrained Markov decision process with a generally parameterized policy. The lower-level solution map can be set-valued and need not be Lipschitz continuous in the upper-level variable, so standard hypergradient analyses do not apply. We therefore define the upper-level objective as the expected upper-level loss of the policy returned by a fixed stochastic lower-level algorithm, which we call the approximate response. This objective is well defined without uniqueness of the lower-level solution. To compute its gradient, we randomize the reward and constraint values used during lower-level training without changing their means, so that the upper-level variable affects training only through known probabilities. This yields an unbiased estimator of the upper-level objective's gradient that does not differentiate through the lower-level updates. We establish lower-level reward and feasibility guarantees, upper-level stationarity, and a total sample complexity of $\widetilde{\mathcal{O}}(\delta_\ell^{-6}\epsilon_o^{-2})$ over both levels, where $\delta_\ell$ is the lower-level accuracy and $\epsilon_o$ is the upper-level stationarity error. Setting $\delta_\ell=\epsilon_o=\epsilon$ gives $\widetilde{\mathcal{O}}(\epsilon^{-8})$.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.00282 [cs.LG] |
| (or arXiv:2602.00282v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.00282 arXiv-issued DOI via DataCite |
Submission history
From: Naman Saxena [view email]
[v1]
Fri, 30 Jan 2026 20:10:21 UTC (51 KB)
[v2]
Thu, 12 Nov 2026 02:54:10 UTC (83 KB)
来源：arXiv cs.AI · arxiv.org