做上下文学习不需要特征学习：核机器视角下的 ICL 记忆与泛化
You Don't Need Feature Learning to Do In-Context Learning
AI 导读
一项 arXiv 研究（cs.AI/cs.LG）用旋转不变核机器研究 Raventos et al. (2023) 提出的 ICL 线性回归任务，通过核特征基分解得到学习的谱图景。研究发现随样本量增长，核机器先泛化、后退化为记忆，泛化解近似为记忆解的低阶截断。这表明特征学习对 ICL 并非必需，且 transformer 实验上确实通过训练任务的高阶统计来记忆任务。
正文 · 原文
Abstract:How should we understand memorization vs. generalization in in-context learning (ICL) tasks, and when does one happen vs. the other? To investigate, we study how rotation-invariant kernel machines learn the ICL linear regression task proposed by Raventos et al. (2023). By decomposing the task in the kernel eigenbasis, we obtain a spectral picture of learning in this ICL setting. This picture reveals that as sample size grows, kernel machines learn successively-higher-degree components of the memorizing solution, which depend upon successively-higher-degree statistics of the training task set. The generalizing solution is approximately a low-degree truncation of the memorizing solution, and so the model first generalizes, then degrades into memorization as training proceeds.
The takeaways are threefold. First, if this case is indicative, learning a "generalizing" solution on an ICL task is really just underfitting the memorizing solution! Second, the fact that kernel machines can generalize on this task means that feature learning is inessential to ICL. Third, transformers empirically trained to do ICL are shown to memorize tasks through the higher-order statistics of the training tasks that the kernel-based theory predicts, suggesting it applies even to models capable of feature learning.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.13431 [cs.LG] |
| (or arXiv:2611.13431v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13431 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joseph Turnbull [view email]
[v1]
Tue, 10 Nov 2026 01:16:06 UTC (307 KB)
来源：arXiv cs.AI · arxiv.org