UTA：通用训练自适应学习率衰减方法
UTA: Universal Training Adaptive Learning Rate Decay
AI 导读
UTA 学习率衰减方法提出，将学习率衰减建模为闭环的形状与尺度问题，以连续时间形式实现训练状态自适应，克服了 UBA 方法依赖 SGD 动态的局限。该方法在 LLaMA 60M 至 1B 语言模型及 ResNet50、ViT-B 视觉模型（ImageNet）上均带来一致提升，即插即用且开销可忽略。消融实验验证了其对峰值学习率、最终衰减比及微调场景的鲁棒性。
正文 · 原文
Abstract:Learning rate decay (LRD) is critical for training high-performance models, yet most designs remain empirically driven. Although the Universal Budget Aware (UBA) LRD offers a theoretical framework, its reliance on SGD dynamics limits its applicability to modern Adam-type optimizers and leaves its key `optimization difficulty parameter' opaque. To bridge this gap, we propose the Universal Training Adaptive (UTA) LRD. Unlike UBA's fixed offline trajectory anchored solely to iterations, UTA treats LRD as a closed-loop shape-and-scale problem. By continuous-time formulation, UTA is training-state adaptive with segment-wise optimal shape. This adaptivity links external iterations with internal training states, enabling a near-optimal path in both progression shape and learning rate (LR) scale, a property unattainable by other LRDs, including UBA. Empirically, we demonstrate consistent improvements across language models (LLaMA 60M to 1B) and vision models (ResNet50, ViT-B) on ImageNet with plug-and-play ease and negligible overhead. Extensive ablation studies further confirm UTA's robustness across dimensions, including peak LR, final decay ratio, its parameter $\varphi_0$, component efficacy and generalization to fine-tuning scenarios, underscoring its practical implementability.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.35113 [cs.LG] |
| (or arXiv:2610.35113v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35113 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anda Tang [view email]
[v1]
Mon, 26 Oct 2026 09:53:01 UTC (338 KB)
来源：arXiv cs.AI · arxiv.org