arXiv cs.AI· Jonas Knupp, Jan Hendrik Metzen, Jeremias Bohn, Georg Groh, Kristian Kersting·· 4 小时前AI 评分39
深度循环注意力混合模型 Dreamer：让潜在推理获得应有的注意力
Depth-Recurrent Attention Mixture (Dreamer): Giving Latent Reasoning the Attention it Deserves
AI 导读
arXiv 论文提出模块化的深度循环注意力混合框架 Dreamer，结合序列注意力、深度注意力与稀疏专家注意力，通过沿深度方向的注意力缓解固定隐层尺寸对多步潜在推理的瓶颈。
正文 · 原文
Abstract:Depth-recurrence facilitates latent reasoning by sharing parameters across depths. However, prior work lacks combined FLOP-, parameter-, and memory-matched baselines, underutilizes depth-recurrence due to partially fixed layer stacks, and ignores the bottleneck of constant hidden-sizes that restricts many-step latent reasoning. To address this, we introduce a modular framework of depth-recurrent attention mixtures (Dreamer), combining sequence attention, depth attention, and sparse expert attention. It alleviates the hidden-size bottleneck through attention along depth, decouples scaling dimensions, and allows depth-recurrent models to scale efficiently and effectively. Across language reasoning benchmarks, our models require 2 to 8x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched SOTA, and outperform ca. 2x larger SOTA models with the same training tokens. We further present insights into knowledge usage across depths, e.g., showing 2 to 11x larger expert selection diversity than SOTA MoEs.
| Comments: | NeurIPS 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2601.21582 [cs.AI] |
| (or arXiv:2601.21582v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2601.21582 arXiv-issued DOI via DataCite |
Submission history
From: Jonas Knupp [view email]
[v1]
Thu, 29 Jan 2026 11:44:38 UTC (968 KB)
[v2]
Thu, 5 Nov 2026 10:23:39 UTC (271 KB)
来源：arXiv cs.AI · arxiv.org