响应相关信息不足以判定神经网络训练状态等价性
Response-Relevant Information Is Not Sufficient for Neural Training-State Equivalence
AI 导读
一项 arXiv 研究提出操作性定义:两个检查点若在训练扰动后继续训练时响应相似,才算训练状态等价。
正文 · 原文
Abstract:Response-relevant information is not sufficient evidence that two neural networks are in the same training state. We define training-state equivalence operationally: two checkpoints are equivalent, relative to an intervention family and horizon, if they respond alike when training is perturbed and continues. On fresh runs of a one-layer transformer learning modular addition, response to a weight-decay pulse is highly reproducible relative to between-checkpoint differences (rho_state = 0.934 at 500 steps), establishing the target as measurable. Fifteen descriptors of weight spectra, representation geometry and optimiser geometry carry within-run response-relevant information beyond weight norm and loss on a development dataset, predicting held-out response deviation across seed blocks. But their apparent 16.1% advantage at matching checkpoints by future response does not separate from a conditional placebo preserving each descriptor's fitted dependence on the conventional observables, and a calibrated positive control makes simple insensitivity an unlikely explanation. Information a decoder can extract need not be organised in the pairwise geometry that similarity judgments require.
| Comments: | 18 pages, 5 figures. Code and provenance verified artifacts: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23245 [cs.LG] |
| (or arXiv:2611.23245v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23245 arXiv-issued DOI via DataCite |
Submission history
From: Xuan Khanh Truong [view email]
[v1]
Fri, 4 Sep 2026 05:49:16 UTC (111 KB)
来源:arXiv cs.AI · arxiv.org