ReLF：基于 Rectified Flow 的连续态语言生成模型，生成质量超越自回归基线
Rectified Language Flow
AI 导读
研究者提出 Rectified Language Flow（ReLF），用 rectified flow 把噪声传输到 token 嵌入来生成文本，结合 Hellinger-SVD 固定嵌入、交叉熵自条件引导和正则化 reflow 等设计。
正文 · 原文
Abstract:Continuous-state flow and diffusion models offer a promising approach to fast, parallel language generation, but matching the generation quality of autoregressive models remains challenging. We present Rectified Language Flow (ReLF), a language model that generates text by transporting noise to token embeddings using rectified flow. ReLF combines several complementary design choices: fixed token embeddings constructed with Hellinger-SVD, self-conditioning guidance trained with cross-entropy, and regularized reflow for few-step generation. We further introduce singular logits, which combine an analytically computed term that drives commitment to a valid token embedding as noise vanishes with a bounded neural residual that learns contextual token preferences. With 64 function evaluations on OpenWebText, ReLF achieves lower generative perplexity than autoregressive baselines at matched diversity across all three model sizes tested, while also outperforming prior continuous flow methods in this setting. After two reflow rounds, ReLF still outperforms autoregressive baselines with 16 function evaluations while generating 1,024-token sequences 15-19 times faster at batch size 1.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33811 [cs.AI] |
| (or arXiv:2610.33811v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33811 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lizhang Chen [view email]
[v1]
Sat, 24 Oct 2026 17:16:49 UTC (2,383 KB)
来源：arXiv cs.AI · arxiv.org