GDRO：适用于扩散模型的组级奖励后训练
GDRO: Group-level Reward Post-training Suitable for Diffusion Models
对做扩散/整流流模型奖励对齐的人，GDRO 给出一个可复用的后训练范式：用组级奖励构造完全离线的目标，省掉在线图像采样 rollout，并摆脱 ODE-to-SDE 近似对随机采样器的依赖；同时提出把奖励黑客趋势纳入评测的修正分数，可直接用于评估任何 RL 微调的文生图模型。
原文依据 GDRO supports full offline training that saves the large time cost for image rollout sampling
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期工作将大语言模型中的在线强化学习引入文本到图像的整流流扩散模型，以实现奖励对齐。使用组级奖励能够成功地将模型与目标奖励对齐。然而，该方法面临效率低、依赖随机采样器以及奖励黑客等问题。其根源在于整流流模型与大语言模型存在本质差异：1）在效率方面，在线图像采样耗时远多于训练时间，并主导了训练总时长；2）在随机性方面，一旦初始噪声固定，整流流便是确定性的。针对这些问题，并受大语言模型中组级奖励效果的启发，我们设计了组级直接奖励优化（GDRO）。GDRO 是一种新的后训练范式，用于组级奖励对齐，并结合了整流流模型的特点。通过严格的理论分析，我们指出 GDRO 支持完全离线的训练，从而省去图像 rollout 采样带来的大量时间开销。此外，它不依赖扩散采样器，因而无需通过 ODE 到 SDE 的近似来获得随机性。我们还通过实验研究了可能误导评估的奖励黑客陷阱，并将该因素纳入评估，采用一种修正分数，它不仅考虑原始评估奖励，还考虑奖励黑客的趋势。大量实验表明，GDRO 通过组级离线优化在 OCR 和 GenEval 任务上有效且高效地提升了扩散模型的奖励分数，同时在缓解奖励黑客方面展现出很强的稳定性与鲁棒性。
英文原文摘要 1676 字符
Recent advancements adopt online reinforcement learning (RL) from LLMs to text-to-image rectified flow diffusion models for reward alignment. The use of group-level rewards successfully aligns the model with the targeted reward. However, it faces challenges including low efficiency, dependency on stochastic samplers, and reward hacking. The problem is that rectified flow models are fundamentally different from LLMs: 1) For efficiency, online image sampling takes much more time and dominates the time of training. 2) For stochasticity, rectified flow is deterministic once the initial noise is fixed. Aiming at these problems and inspired by the effects of group-level rewards from LLMs, we design Group-level Direct Reward Optimization (GDRO). GDRO is a new post-training paradigm for group-level reward alignment that combines the characteristics of rectified flow models. Through rigorous theoretical analysis, we point out that GDRO supports full offline training that saves the large time cost for image rollout sampling. Also, it is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtain stochasticity. We also empirically study the reward hacking trap that may mislead the evaluation, and involve this factor in the evaluation using a corrected score that not only considers the original evaluation reward but also the trend of reward hacking. Extensive experiments demonstrate that GDRO effectively and efficiently improves the reward score of the diffusion model through group-wise offline optimization across the OCR and GenEval tasks, while demonstrating strong stability and robustness in mitigating reward hacking.