arXiv cs.AI· Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang·· 3 小时前AI 评分42
LFPO：基于 Velocity 对齐的掩码扩散模型 Token 级高效策略优化
Efficient Token-Level Policy Optimization for Masked Diffusion Models via Velocity Alignment
AI 导读
arXiv 论文提出序列似然无关策略优化框架 LFPO，将扩散大语言模型（dLLM）的策略对齐转化为离散 velocity 对齐，绕开难以求解的序列级似然。该方法通过对比式 token 级更新直接优化去噪 logits，提供稳定、低方差的替代目标，并隐式拉直生成轨迹以减少去噪迭代。在推理与代码评测上，LFPO 匹配或超越基于似然的基线准确率，同时显著加速推理。
正文 · 原文
Authors:Chenxing Wei, Jiazhen Kang, Hong Wang, Jianqing Zhang, Hao Jiang, Xiaolong Xu, Ningyuan Sun, Ying He, F. Richard Yu, Yao Shu, Bo Jiang
Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) successfully aligns autoregressive models in rigorous domains like mathematical reasoning and code generation. However, applying RLVR to Diffusion Large Language Models (dLLMs) is hindered by intractable sequence-level likelihoods, forcing existing methods to rely on unstable, high-variance approximations. We propose Sequence Likelihood-Free Policy Optimization (LFPO), an efficient framework that circumvents this bottleneck by formulating policy alignment as discrete velocity alignment. Inspired by continuous flow matching, LFPO eschews trajectory-level estimation and directly optimizes denoising logits via contrastive token-level updates, providing a stable, low-variance surrogate. Furthermore, predicting terminal solutions from intermediate states implicitly straightens the generation trajectory, enabling high-quality sampling with significantly fewer denoising iterations. Extensive experiments on reasoning and coding benchmarks demonstrate that LFPO matches or exceeds the accuracy of likelihood-based baselines while substantially accelerating inference.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.01563 [cs.LG] |
| (or arXiv:2603.01563v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01563 arXiv-issued DOI via DataCite |
Submission history
From: Chenxing Wei [view email]
[v1]
Mon, 2 Mar 2026 07:42:55 UTC (272 KB)
[v2]
Sun, 27 Sep 2026 05:00:38 UTC (300 KB)
[v3]
Tue, 27 Oct 2026 16:51:22 UTC (300 KB)
来源：arXiv cs.AI · arxiv.org