增广拉格朗日方法求解提升贝尔曼线性规划（AL-LBLP）
An Augmented Lagrangian Approach to Lifted Bellman Linear Programming
AI 导读
论文提出 AL-LBLP，用乘子法求解离线 RL 的 Lifted Bellman Linear Programming：约束系数改为由对偶上升学习的乘子，仅一个罚参数 ρ。
正文 · 原文
Abstract:Temporal-difference critics regress onto a bootstrapped target and usually need a target network. Lifted Bellman Linear Programming (LBLP) instead expresses the in-sample optimal values of offline RL as the solution to a linear program over the dataset. Its neural solver ALBUM relaxes the constraints into hinge penalties with fixed coefficients, and its guarantees are conditions on coefficients chosen in advance. We solve the same program with the method of multipliers: the coefficients become multipliers learned by dual ascent, with one penalty parameter $\rho$. In the convex case the fixed points are the KKT pairs of LBLP, and at every stationary point the applied constraint weights balance the objective exactly. The critic can serve a Gaussian actor, a flow actor, and best-of-N sampling, and for the Gaussian actor we propose an advantage-normalized DDPG+BC objective. With a single $Q$ and $V$ and no target network, our AL-LBLP with best-of-N reaches 60.1\% average success on the 50 state-based OGBench tasks. On long-horizon tasks, it reaches 81.5% on humanoidmaze-giant and 79.8% on antmaze-giant. It does not collapse up to batch size 8192.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14183 [cs.AI] |
| (or arXiv:2611.14183v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14183 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jongchan Park [view email]
[v1]
Tue, 10 Nov 2026 15:05:22 UTC (132 KB)
来源：arXiv cs.AI · arxiv.org