arXiv cs.AI· Patrick Krauss, Achim Schilling, Andreas Maier, Thomas Kinfe, Claus Metzner·· 2 小时前AI 评分41
研究发现神经网络的最终权重会留下可解码的学习轨迹痕迹
Learning Trajectories Leave Decodable Traces in Neural Network Weight Space and Internal Representations
AI 导读
一项 arXiv 研究测试了神经网络最终权重是否保留学习历史信息。在 30 个网络上,全部正确匹配到 10 个候选中的初始权重;线性探针在 80 个留出网络上以 100% 准确率解码出先前两个任务的学习顺序。2,160 个对齐端点中有 2,157 个在采样插值点无正损失屏障,表明任务训练约束网络形成相关但可区分的解,最终权重选择性保留历史信息。
正文 · 原文
Abstract:Neural networks trained on the same task develop related weight-space structures, yet remain distinguishable after learning. Do these differences merely reflect alternative solutions, or do they preserve information about individual learning histories? We tested whether final weights reveal initialization, task order, and earlier trajectory states. Training reorganized the ensemble: initial pairwise distances did not consistently predict final distances. Nevertheless, all 30 trained networks were correctly matched to their initial weights among ten candidates per dataset. Networks sharing an initialization also developed recognizable final hidden representations across different batch schedules, although direct comparisons with initial activity did not recover identity. Learning order remained readable as well. A linear probe decoded the order of two earlier tasks with 100% accuracy on 80 held-out networks after ten epochs of a common final task. Earlier intermediate weight states were partly predictable, but individual initial weights could not be reliably reconstructed. Final weights thus retain selective historical information rather than a complete trajectory record. Perturbed networks recovered classification performance without returning to their unperturbed training paths. This separation persisted across datasets, intervention times, and network layers. Moreover, 2,157 of 2,160 aligned endpoint pairs showed no positive loss barrier at sampled interpolation points, indicating low-loss connections between distinct solutions rather than convergence to one weight vector. Together, these findings show that a shared task constrains networks to related successful solutions without erasing their individual histories. Final weights reflect not only what a network has learned, but also where learning began and how it unfolded.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Neurons and Cognition (q-bio.NC) |
| Cite as: | arXiv:2611.24782 [cs.LG] |
| (or arXiv:2611.24782v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24782 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Patrick Krauss [view email]
[v1]
Wed, 18 Nov 2026 17:27:55 UTC (5,221 KB)
来源:arXiv cs.AI · arxiv.org