细粒度 GRPO：面向流模型的精确偏好对齐
Fine-Grained GRPO for Precise Preference Alignment in Flow Models
把 GRPO 式在线 RL 的信用分配细化到单步 SDE 扰动，并跨多个扩散尺度聚合优势，可直接迁移到任何 flow/diffusion 模型的偏好对齐与奖励微调流程中。
原文依据 we propose a Singular Stochastic Sampling mechanism that supports step-wise stochastic exploration while ensuring strong correlation between injected noise and reward signals
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将在线强化学习（RL）引入扩散与基于流的生成模型，近来作为使模型行为与人类偏好对齐的强大范式而受到关注。通过在去噪阶段利用随机微分方程（SDE）进行随机采样，这类模型能够探索多种去噪轨迹，从而增强 RL 的探索能力。然而，尽管现有方法能够发现潜在的高奖励样本，却常常因奖励反馈的稀疏性与狭窄性而难以有效对齐偏好。为克服这一局限，我们提出一个名为 Granular-GRPO（G^2RPO）的新框架，它能够在流模型的 RL 训练中对采样方向进行细粒度且全面的评估。具体而言，我们提出一种奇异随机采样机制，在支持逐步随机探索的同时确保注入噪声与奖励信号之间具有强相关性，从而对每一次 SDE 扰动实现更准确的信用分配。此外，为缓解固定粒度去噪所带来的偏差，我们设计了一个多粒度优势集成模块，聚合在多个扩散尺度上计算得到的优势，从而对采样轨迹给出更稳健、更全面的评估。在包括域内与域外设置在内的多种奖励模型上的大量实验表明，我们的 G^2RPO 优于现有的基于流的 GRPO 基线，凸显了其有效性与泛化能力。
英文原文摘要 1607 字符
The incorporation of online reinforcement learning (RL) into diffusion and flow-based generative models has recently gained attention as a powerful paradigm for aligning model behavior with human preferences. By leveraging stochastic sampling via Stochastic Differential Equations (SDEs) during the denoising phase, these models can explore a variety of denoising trajectories, enhancing the exploratory capacity of RL. However, despite their ability to discover potentially high-reward samples, current approaches often struggle to effectively align with preferences due to the sparsity and narrowness of reward feedback. To overcome this limitation, we introduce a novel framework called Granular-GRPO (G^2RPO), which enables fine-grained and comprehensive evaluation of sampling directions in the RL training of flow models. Specifically, we propose a Singular Stochastic Sampling mechanism that supports step-wise stochastic exploration while ensuring strong correlation between injected noise and reward signals, enabling more accurate credit assignment to each SDE perturbation. Additionally, to mitigate the bias introduced by fixed-granularity denoising, we design a Multi-Granularity Advantage Integration module that aggregates advantages computed across multiple diffusion scales, resulting in a more robust and holistic assessment of sampling trajectories. Extensive experiments on various reward models, including both in-domain and out-of-domain settings, demonstrate that our G^2RPO outperforms existing flow-based GRPO baselines, highlighting its effectiveness and generalization capability.