神经网络的标度与重整化：Grokking 现象的有效重整化群描述
Scaling and Renormalization in Neural Networks: An Effective Renormalization-Group Description of Grokking
AI 导读
一项 arXiv 研究用标度与重整化群（RG）理论定量描述深度神经网络中从记忆到泛化的延迟转变——Grokking。研究在两层神经网络上对模加法任务训练的模型集合定义序参量（权重范数平方），用损失的二阶展开构造耦合矩阵并以 Fiedler 向量确定有效几何，再通过块自旋聚合执行实空间 RG。线性化得到的五个临界指数经验估计与 RG 预测偏差在 1-10% 以内。
正文 · 原文
Abstract:Grokking is a delayed transition from memorization to generalization in deep neural networks (NNs). We examine whether scaling and renormalization group (RG) theories can describe this dynamics, establishing a quantitative physical framework for neural generalization transitions. For modular addition on a two-layer NN, an ensemble of trained models defines an order parameter (the squared weight norm) and fluctuation proxies for the heat capacity and the susceptibility, with training epoch and weight decay serving as thermal and field control coordinates. To bridge neural architectures to real-space RG, we construct an exact coupling matrix via a second-order expansion of the loss with respect to per-neuron norms and use its Fiedler vector to define an effective geometry. This spectral ordering yields a monotonic spatial decay of coupling strength that is absent under random permutations. We then execute real-space RG by block-spin aggregating adjacent neurons and parameterizing the coupling matrix into off-diagonal band strengths. Linearizing this local RG map yields relevant scaling exponents and five derived critical exponents, with direct empirical estimates matching RG predictions within 1-10%. Overall, our framework provides a consistent effective description of grokking, suggesting the scaling and renormalization group formalism as a tool to investigate NN dynamics.
| Comments: | 22 pages, 10 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); High Energy Physics - Theory (hep-th); Quantum Physics (quant-ph) |
| Cite as: | arXiv:2610.36113 [cs.LG] |
| (or arXiv:2610.36113v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36113 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Min Chen [view email]
[v1]
Mon, 26 Oct 2026 18:25:08 UTC (3,423 KB)
来源：arXiv cs.AI · arxiv.org