面向流匹配模型 GRPO 的逐步信用分配
Stepwise Credit Assignment for GRPO on Flow-Matching Models
把扩散/流匹配的采样步按「低频结构 vs 高频细节」分层做信用分配，可直接迁移到任何用 GRPO/RLHF 微调图像或视频生成模型的后训练流程，提升样本效率与收敛速度。
原文依据 assigns credit based on each step's reward improvement
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Flow-GRPO 成功地将强化学习应用于流模型，但在所有步骤上采用统一的信用分配。这忽略了扩散生成的时序结构：早期步骤决定构图与内容（低频结构），而后期步骤解决细节与纹理（高频细节）。此外，仅依据最终图像进行统一信用分配，可能会在扩散轨迹后期纠正错误的情况下，无意中奖励次优的中间步骤。我们提出 Stepwise-Flow-GRPO，它根据每一步的奖励提升来分配信用。通过利用 Tweedie 公式获得中间奖励估计，并引入基于增益的优势函数，我们的方法实现了更优的样本效率和更快的收敛。我们还引入了一种受 DDIM 启发的 SDE，它在为策略梯度保留随机性的同时提升了奖励质量。
英文原文摘要 918 字符
Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps, especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO, which assigns credit based on each step's reward improvement. By leveraging Tweedie's formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients.