Dual-Adaptive Z-Score PPO：基于动作置信度与归一化优势幅度的动态裁剪
Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude
AI 导读
论文提出 PPO-DAZ3，用动作置信度（所选动作在当前策略下的概率）和更新置信度（归一化优势幅度）为每个样本动态设定 PPO 裁剪阈值，其余训练流程不变。
正文 · 原文
Abstract:Proximal Policy Optimization (PPO) remains a popular on-policy reinforcement learning algorithm owing to its stability and ease of implementation. Its clipping threshold, however, is fixed, so every training sample receives the same trust-region width regardless of how reliable the corresponding policy update is. This work examines adaptive-clipping variants of PPO in which the constant clipping parameter is replaced by sample-dependent thresholds computed from policy and return statistics. The central method, Dual-Adaptive Z-Score PPO (PPO-DAZ3), sets the clipping interval for each sample from two signals: action confidence, given by the probability of the selected action under the current policy, and update confidence, given by the normalized advantage magnitude. The rest of the standard PPO training pipeline is left unchanged. Evaluation on three classic control benchmarks (LunarLander-v3, CartPole-v1, and Acrobot-v1) indicates that adaptive clipping improves learning in some settings but not all. PPO-DAZ3 performs best on LunarLander-v3, the smooth advantage-based variant PPO-AZ3S leads on Acrobot-v1, and standard PPO stays competitive on CartPole-v1, suggesting that adaptive clipping is most beneficial on harder control tasks.
| Comments: | 6 pages, 4 figures. This is the author's accepted manuscript of the paper published in the 2026 6th International Conference on Electrical, Computer and Energy Technologies (ICECET 2026), Rome, Italy. The final published version is available via IEEE Xplore at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32194 [cs.LG] |
| (or arXiv:2610.32194v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32194 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | 2026 6th International Conference on Electrical, Computer and Energy Technologies (ICECET) |
| Related DOI: | https://doi.org/10.1109/ICECET65726.2026.11633063
DOI(s) linking to related resources |
Submission history
From: Taha Mahmoud [view email]
[v1]
Fri, 23 Oct 2026 05:40:26 UTC (1,017 KB)
来源：arXiv cs.AI · arxiv.org