TIBF：基于 Token Identity 平方监督的语言流继续训练方法
Continued Training of Continuous Language Flows with Token Identity Squared Supervision
AI 导读
一篇 arXiv 论文提出 Token Identity Brier Flow（TIBF），在继续训练预训练语言流时直接对 token 身份施加平方监督，用 Brier loss 监督 token 概率并以概率加权嵌入做连续生成。
正文 · 原文
Abstract:Diffusion and flow models have recently been extended to language modeling, generating text through continuous denoising in token embedding spaces. Pretrained models provide token representations and contextual prediction capabilities, but improving generation with limited additional training requires an appropriate connection between supervision and continuous generation. Continuous MSE evaluates a predicted vector obtained by weighting token embeddings with predicted probabilities; different probability distributions can yield the same vector and are therefore indistinguishable under this objective. We propose Token Identity Brier Flow (TIBF), which applies squared supervision directly to token identities when continuing pretrained language flows. TIBF combines network scores with token-matching scores under a Gaussian observation model, supervises token probabilities using the Brier loss, and uses probability-weighted embeddings for continuous generation. With fixed token embeddings, we show that the squared error between predicted token probabilities and the true posterior bounds the squared errors of continuous denoising predictions and flow velocities. A risk decomposition separates uncertainty inherent in the observation from posterior fitting error, motivating continued training that preserves pretrained embeddings and focuses on probability prediction. We also establish convergence to token embeddings under ideal predictions and sufficient conditions for local convergence with prediction error. From common pretrained initializations and under matched continuation budgets, frozen-embedding TIBF achieves this http URL values of 33.96 on OWT and 81.17 on LM1B, outperforming continuous-MSE and CE continuation controls. It attains the lowest this http URL at every tested sampling budget on both datasets, together with the lowest repetition rate on LM1B.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.07721 [cs.CL] |
| (or arXiv:2611.07721v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07721 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhaoxiang Zang [view email]
[v1]
Thu, 5 Nov 2026 09:38:06 UTC (1,249 KB)
来源：arXiv cs.AI · arxiv.org