Kernel Ridge Attention:带选择性遗忘的非线性记忆序列模型
Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting
AI 导读
arXiv 论文提出 Kernel Ridge Attention(KRA),一种结合学习非线性特征与协方差引导记忆更新的循环序列混合器。它通过递归岭回归存储关联,用内容相关的 Ornstein–Uhlenbeck 转移控制记忆保留与遗忘,推理状态大小与上下文长度无关。在受控语言建模实验中,KRA 预测质量优于 attention 和循环基线,联想召回强于所评估的循环模型,并支持分布式并行训练。
正文 · 原文
Abstract:Kernel Ridge Attention (KRA) is a recurrent sequence mixer that combines learned nonlinear features with covariance-guided memory updates. It stores associations through recursive ridge regression, using correlations among past keys to determine how new observations modify memory. A content-dependent Ornstein--Uhlenbeck transition controls retention and restores uncertainty for future writes. This construction admits an exact temporal-kernel interpretation and maintains an inference state whose size is independent of context length. Associative block maps and explicit adjoints support parallel training while preserving the recursive update. In controlled language-modeling experiments, KRA improves prediction quality over attention and recurrent baselines, strengthens natural associative recall over the evaluated recurrent models, and achieves practical distributed training throughput. These properties provide a principled foundation for nonlinear, adaptive sequence memory.
| Comments: | 15 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.23896 [cs.LG] |
| (or arXiv:2611.23896v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23896 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yichen Wang [view email]
[v1]
Wed, 18 Nov 2026 05:49:34 UTC (99 KB)
来源:arXiv cs.AI · arxiv.org