跳到正文
arXiv cs.AI· Zeman Li, Ali Behrouz, Yuan Deng, Praneeth Kacham, Rami Al-Rfou, Meisam Razaviyayn, Vahab Mirrokni·· 1 小时前AI 评分45

Toaster:用级联线性记忆与 target propagation 构建非线性记忆

Toaster: Nonlinear Memory from Cascaded Linear Memories and Target Propagation

AI 导读

一篇 arXiv 论文提出 Toaster,通过两个线性记忆中间加激活函数并用 target propagation 反传输出误差,构建两层非线性记忆,各层仍可在未修改的 delta-rule 内核上运行。

正文 · 原文

View PDF HTML (experimental)

Abstract:Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context.
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.23648 [cs.LG]
  (or arXiv:2611.23648v1 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.23648

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Zeman Li [view email]
[v1] Wed, 18 Nov 2026 00:14:27 UTC (233 KB)

来源:arXiv cs.AI · arxiv.org