LeNO：面向少步生成模型奖励对齐的噪声优化学习方法
Learning Noise Optimization for Reward Alignment in Few-Step Generative Models
AI 导读
LeNO（Learning Noise Optimization）将基于奖励的初始噪声优化蒸馏为可学习的残差传输，预训练生成器保持冻结，无需测试时迭代优化。该方法用 Local GRPO 和标量奖励学习锚点条件下的噪声修正分布，并推导出修正噪声分布与原始先验之间 KL 散度的可解上界作为正则化目标。
正文 · 原文
Abstract:Reward alignment for diffusion- and flow-based generative models is often formulated over stochastic trajectories, making it less suitable for few-step deterministic generators. Initial noise optimization provides a trajectory-free alternative, but existing gradient-free methods require costly per-sample search at inference. We introduce LeNO (Learning Noise Optimization), which amortizes reward-guided initial noise optimization into a learned residual transport while keeping the pretrained generator frozen. LeNO learns an anchor-conditioned distribution over noise corrections using Local GRPO and scalar rewards, eliminating iterative test-time optimization. For Gaussian initial noise with norm-preserving projection, we derive a tractable upper bound on the KL divergence between the corrected noise distribution and the original prior, providing a principled regularization objective. We evaluate LeNO on few-step text-to-image and protein structure generation, including expensive black-box designability rewards. Across both domains, LeNO improves target rewards with a single noise correction at inference while largely preserving generation quality and diversity.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.06255 [cs.LG] |
| (or arXiv:2611.06255v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06255 arXiv-issued DOI via DataCite |
Submission history
From: Jeongsol Kim [view email]
[v1]
Wed, 4 Nov 2026 10:06:32 UTC (7,079 KB)
来源：arXiv cs.AI · arxiv.org