arXiv cs.AI· Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen·· 4 小时前AI 评分38
失败风险定价：带凸正则化的截断屏障策略梯度
The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization
AI 导读
arXiv 论文提出一种预算约束下的策略梯度方法：用屏障惩罚随预算减少增长、并在 -m 处从下方截断奖励，截断水平 m 即“失败风险的价格”。理论证明最优值可精确分解为效用、边际成本与价格乘风险三项，且在凸正则化下训练流指数收敛，将策略梯度的均值场收敛从相对熵扩展到 weight decay。实验在维护基准、油箱控制器等任务上度量了边界的松弛程度。
正文 · 原文
Abstract:Many operations run on a budget that must rarely run out, such as a wear allowance. A wrapper keeps the remaining budget in the state, subtracts a barrier that grows as the budget shrinks and clips the combined reward from below at -m, and a standard policy-gradient learner trains unchanged. The clip level m is a price of failure risk, since the trained value splits exactly into utility before failure, the cost of a margin and the price times the risk, so whatever policy training returns earns the most such utility net of that cost among policies with no greater risk, up to its optimization error. Its risk exceeds a comparison policy's by at most order 1/m when charged costs stay bounded, and value or validation data certify it. The main theoretical result is exponential convergence of the population training flow under any parameter regularizer convex along the motions of the network's units and strong enough against the price, biased toward the prior policy. This extends mean-field convergence of policy gradients from relative entropy to weight decay, which covers continuous-time training of finite networks. Relative entropy adds noise that lets learning leave flat regions, weight decay pulls the parameters toward a reference, and action regularization bounds the risk change by the distance to a trusted rule. Experiments on an exact maintenance benchmark, a fuel-tank controller, a solvable flow and a benchmark-suite task measure the bounds' slack. We state when another method is better, without claiming this one outperforms them.
| Comments: | 38 pages, 7 figures, 2 tables; Online Resource 1 (44 pages) as an ancillary file. Revised and corrected version, replaces v2 (title changed). Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Optimization and Control (math.OC); Probability (math.PR); Machine Learning (stat.ML) |
| MSC classes: | 90C40, 93E20, 68T05, 49Q22, 90C15 |
| ACM classes: | I.2.6; G.1.6 |
| Cite as: | arXiv:2411.19193 [cs.LG] |
| (or arXiv:2411.19193v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2411.19193 arXiv-issued DOI via DataCite |
Submission history
From: Pekka Malo [view email]
[v1]
Thu, 28 Nov 2024 15:04:43 UTC (104 KB)
[v2]
Tue, 16 Sep 2025 14:10:15 UTC (53 KB)
[v3]
Tue, 27 Oct 2026 14:54:27 UTC (1,317 KB)
来源：arXiv cs.AI · arxiv.org