arXiv cs.AI· Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai·· 4 小时前AI 评分40
Loopie：迄今最强的开源循环 Transformer 架构
Loop the Loopies!
AI 导读
Loopie 系列是迄今最强的开源循环 Transformer，包含两个 MoE 模型：20B 参数（2B 激活）和 6B 参数（0.6B 激活）。循环 Transformer 长期面临 N 倍预训练算力下增大参数量通常优于循环 N 次的难题，Loopie 解决了这一问题。
正文 · 原文
Abstract:We present Loopie, the most powerful open-source looped Transformer to date. The Loopie series consists of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N-fold pre-training compute, increasing the parameter count by a factor of N usually outperforms looping N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.16051 [cs.CL] |
| (or arXiv:2607.16051v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.16051 arXiv-issued DOI via DataCite |
Submission history
From: Zitian Gao [view email]
[v1]
Fri, 17 Jul 2026 15:28:43 UTC (829 KB)
[v2]
Mon, 20 Jul 2026 15:59:50 UTC (835 KB)
[v3]
Mon, 26 Oct 2026 17:12:57 UTC (1,055 KB)
来源：arXiv cs.AI · arxiv.org