神经网络学习轨迹中的表示复杂度：以宽度换精度与时间
Representation Complexity Along Learning Trajectories: Trading Width for Precision and Time
AI 导读
arXiv 论文《Representation Complexity Along Learning Trajectories》研究过参数化在学习过程中的计算作用，提出神经网络学习的 oracle 模型，用 Kolmogorov n-宽度量化有效表示复杂度，并将前向表示的复杂度界转移到梯度查询特征上。
正文 · 原文
Abstract:What computational role does overparameterisation play during learning? Information-plane analyses suggest that neural networks can traverse representations substantially more complex than those required by their final predictors. We explore when this transient complexity serves a computational purpose by comparing the representation complexity needed for prediction with that needed to make an unknown target statistically visible through gradient feedback. We introduce an oracle model of neural network learning and use Kolmogorov \(n\)-widths to quantify effective representation complexity. Our main technical result transfers complexity bounds from the forward representation to the corresponding gradient-query features. Combined with statistical-query lower bounds, this gives a task-agnostic lower bound on the initial effective width required for target-dependent gradient information to become detectable. For \(k\)-sparse parity in \(D\) dimensions, a known target admits an \(O(k)\)-width representation, while the general bound already implies a polynomial separation between representation and discovery. We then exploit parity's invariance under coordinatewise sign reflections to identify a sharper unsigned effective width, which removes sign distinctions invisible to the parity information channel. At a transfer-compatible measurement resolution, its critical scale is \(\widetilde{\Theta}_k\left( D^{k-1}\tau^2 \right)\), up to logarithmic factors. Thus, the representation resources required to discover a predictor can substantially exceed those required to represent it once learned. Interpreting precision through accumulation over fresh samples yields a corresponding width-time frontier. Experiments support the predicted dimension dependence and examine how optimisation and generalisation depart from this idealised frontier.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.39027 [cs.LG] |
| (or arXiv:2610.39027v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39027 arXiv-issued DOI via DataCite |
Submission history
From: Ryan Singh [view email]
[v1]
Wed, 28 Oct 2026 23:01:16 UTC (422 KB)
来源：arXiv cs.AI · arxiv.org