深度神经网络为何涌现几何连续性：残差连接与旋转对称性破缺
Why Geometric Continuity Emerges in Deep Neural Networks: Residual Connections and Rotational Symmetry Breaking
AI 导读
一项 NeurIPS 2026 研究通过 toy MLP 和小型 transformer 实验，解释了深层网络相邻层主奇异向量方向相似的几何连续性来源。研究发现两个机制：残差连接产生跨层梯度一致性从而对齐各层权重更新，破坏对称性的非线性则把所有层约束到共享坐标系，防止旋转漂移；一个非线性但旋转等变的激活只能达到 ReLU 连续性的一半以下，说明起作用的是对称性破缺而非非线性本身。
正文 · 原文
Abstract:Weight matrices in deep networks exhibit geometric continuity---principal singular vectors of adjacent layers point in similar directions. While this property has been widely observed, its origin remains unexplained. Through experiments on toy MLPs and small transformers, we identify two mechanisms: residual connections create cross-layer gradient coherence that aligns weight updates across layers, and symmetry-breaking nonlinearities constrain all layers to a shared coordinate frame, preventing the rotation drift that would otherwise destabilize weight structure. Crucially, a nonlinear but rotation-equivariant activation reaches less than half of ReLU's continuity, isolating symmetry breaking---not nonlinearity alone---as the active ingredient. Activation and normalization play distinct roles: activation concentrates continuity in the leading singular direction, while normalization distributes it across multiple directions. In transformers, continuity is projection-specific: Q, K, Gate, and Up (which read from the residual stream) develop input-space ($\mathbf{v}_1$) continuity; O and Down (which write to it) develop output-space ($\mathbf{u}_1$) continuity; V alone, lacking an adjacent nonlinearity, develops only low continuity.
| Comments: | NeurIPS 2026 (camera-ready). 10 pages main text, 33 pages total including appendices and checklist. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| MSC classes: | 68T07 |
| ACM classes: | I.2.6; I.2.7 |
| Cite as: | arXiv:2605.04971 [cs.LG] |
| (or arXiv:2605.04971v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.04971 arXiv-issued DOI via DataCite |
Submission history
From: Kyungwon Jeong [view email]
[v1]
Wed, 6 May 2026 14:27:18 UTC (4,596 KB)
[v2]
Mon, 26 Oct 2026 05:03:38 UTC (4,204 KB)
来源：arXiv cs.AI · arxiv.org