跳到正文
arXiv cs.AI· Shreyas Chaudhari, Jose M. F. Moura·· 2 小时前AI 评分20

ReLU 网络中 Bias-Only 训练动力学研究

Bias-Only Training Dynamics in ReLU Networks

AI 导读

一项 arXiv 研究通过激活码刻画了单隐层 ReLU 网络中仅训练偏置时的动力学:梯度只依赖输入的激活码,固定激活码下训练等价于以码为特征的线性最小二乘问题。随着网络宽度增大,切换神经元的比例趋近于零,无限宽极限下训练动力学趋近于初始激活码上的线性最小二乘,实验支持了这些理论结论。

正文 · 原文

View PDF HTML (experimental)

Abstract:Neural networks with randomly initialized, fixed weights and trainable biases are universal approximators, yet the dynamics of bias-only training remain largely unexplored. We study these dynamics in one-hidden-layer ReLU networks through activation codes, the binary patterns of active neurons that partition the input space into convex polyhedral regions. We show that the gradients with respect to the biases depend on each input only through its activation code, and hence the training dynamics can be characterized with these codes. In particular, as long as the activation codes remain fixed, training reduces to a linear least-squares problem with the codes as features. Training can then be understood as a sequence of least-squares problems, with transitions occurring when neurons change their activation states. We show that the fraction of switching neurons vanishes as the network widens, and the training dynamics approach those of linear least squares on the initial activation codes in the infinite-width limit. Our experiments support these theoretical findings, showing that while activation switches can noticeably affect training in narrower networks, their relative frequency and the associated changes in the dynamics diminish as the width increases.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24797 [cs.LG]
  (or arXiv:2611.24797v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.24797

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shreyas Chaudhari [view email]
[v1] Wed, 18 Nov 2026 17:41:18 UTC (113 KB)

来源:arXiv cs.AI · arxiv.org