灾难性过拟合、熵差与 Participation Ratio：一种无噪声 l^p 范数快速对抗训练方案
Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless $l^p$ Norm Solution for Fast Adversarial Training
AI 导读
一项研究提出仅通过控制 l^p 训练范数来缓解快速对抗训练（FGSM）中的灾难性过拟合（CO），无需噪声注入、正则化或梯度裁剪。研究发现 CO 在 l^∞ 范数下比 l^2 更普遍，其根源是高度集中的梯度与激进范数约束的相互作用。基于 Participation Ratio 和熵度量量化梯度集中度，研究构建了自适应 l^p-FGSM，可依据梯度信息自动调节训练范数，实验显示其鲁棒性具有竞争力。
正文 · 原文
Abstract:Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants. While existing solutions rely on noise injection, regularization, or gradient clipping, we propose a novel solution that purely controls the $l^p$ training norm to mitigate CO. Our study is motivated by the empirical observation that CO is more prevalent under the $l^{\infty}$ norm than the $l^2$ norm. Leveraging this insight, we develop a framework for generalized $l^p$ attack as a fixed point problem and craft $l^p$-FGSM attacks to understand the transition mechanics from $l^2$ to $l^{\infty}$. This leads to our core insight: CO emerges when highly concentrated gradients, where information localizes in few dimensions, interact with aggressive norm constraints. By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive $l^p$-FGSM that automatically tunes the training norm based on gradient information. Extensive experiments demonstrate that this approach achieves competitive robustness without requiring additional regularization or noise injection, providing a novel and theoretically-principled pathway to mitigate the CO problem.
| Comments: | 25 pages, 16 figures, 7 tables. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.02360 [cs.LG] |
| (or arXiv:2505.02360v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.02360 arXiv-issued DOI via DataCite |
Submission history
From: Fares B. Mehouachi [view email]
[v1]
Mon, 5 May 2025 04:41:21 UTC (4,666 KB)
[v2]
Sat, 16 May 2026 12:36:31 UTC (4,667 KB)
[v3]
Fri, 23 Oct 2026 11:28:52 UTC (3,413 KB)
来源：arXiv cs.AI · arxiv.org