跳到正文
arXiv cs.AI· Haocheng Tang, Tianchi Xie, Xingqiao Lin·· 4 小时前AI 评分29

MeanFlowAdvantage:面向少步平均速度生成器的稳定奖励微调方法

MeanFlowAdvantage: Stable Reward Fine-Tuning for Few-Step Average-Velocity Generators

AI 导读

研究者提出 MeanFlowAdvantage,一种带符号优势加权的最小二乘目标,解决 MeanFlow 平均速度表示与现有优势型奖励微调目标的错配问题。

正文 · 原文

View PDF HTML (experimental)

Abstract:MeanFlow enables efficient few-step generation by predicting interval-average velocities, but this representation creates a mismatch for reward fine-tuning: existing advantage-based objectives are typically defined on instantaneous velocities or equivalent $x_0$-space predictions, whereas inference directly uses the learned average-velocity map. We introduce MeanFlowAdvantage, a signed advantage-weighted least-squares objective for average-velocity generators. Our key construction uses a shared, detached MeanFlow derivative correction to express the reward objective in prediction space while making rollout and reference regularization exact penalties on the average-velocity network deployed at inference. The resulting formulation preserves MeanFlow's native few-step sampler and provides a direct mechanism for transferring reward improvements to the deployed flow map. On SD3.5-Medium, MeanFlowAdvantage improves all eight reported metrics over the matched four-step MeanFlowNFT baseline and, with only four NFEs, matches or exceeds the 40-step DiffusionNFT baseline on six of eight metrics. The same objective also transfers to DNA promoter design, where it supports both teacher-free on-policy RL for a generator defined on a manifold and teacher-guided reward-graded distillation, with the latter yielding the lowest one-step Sei profile MSE among the compared configurations.
Subjects: Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Computer Vision and Pattern Recognition (cs.CV)
Cite as: arXiv:2609.37670 [cs.AI]
  (or arXiv:2609.37670v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2609.37670

arXiv-issued DOI via DataCite

Submission history

From: Haocheng Tang [view email]
[v1] Tue, 29 Sep 2026 14:25:03 UTC (2,277 KB)
[v2] Fri, 13 Nov 2026 01:30:10 UTC (2,277 KB)

来源:arXiv cs.AI · arxiv.org