低损失区域错位导致 Grokking 现象:arXiv 论文提出新分析框架
Misalignment of Low-Loss Regions Causes Grokking
AI 导读
一篇 arXiv 论文提出基于 mode connectivity 与低损失区域几何的分析框架,认为 Grokking(验证集泛化的延迟出现)源于训练与验证划分诱导的低损失区域错位。
正文 · 原文
Abstract:Grokking refers to the delayed emergence of validation-set generalization after a model has already overfit the training set. Although first observed in small algorithmic tasks trained with transformers, its underlying mechanism remains unsettled. In this work, we develop an analysis framework based on mode connectivity and the geometry of low-loss regions. The framework predicts that the standard modular-arithmetic setting does not always produce grokking: under a symmetry-preserving train/validation split, we observe a stable anti-grokking case in which validation performance does not recover. This counterexample challenges several existing correlational explanations of grokking. More broadly, our analysis framework and results further suggest that grokking arises when the low-loss regions induced by the training and validation partitions are misaligned. Once these regions become well aligned, training hyperparameters alone cannot produce grokking and the observed dynamics collapse to either trainable or non-trainable behavior.
| Comments: | 28 pages, 27 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.00620 [cs.LG] |
| (or arXiv:2610.00620v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00620 arXiv-issued DOI via DataCite |
Submission history
From: Yongding Tian [view email]
[v1]
Wed, 30 Sep 2026 19:22:20 UTC (2,371 KB)
[v2]
Tue, 17 Nov 2026 16:09:05 UTC (3,126 KB)
来源:arXiv cs.AI · arxiv.org