Neighbor GRPO：对比式常微分方程策略优化对齐流模型
Neighbor GRPO: Contrastive ODE Policy Optimization Aligns Flow Models
对做扩散/流匹配模型偏好对齐的人，这篇给出了绕开 SDE 的 GRPO 方案：直接扰动 ODE 初始噪声构造候选轨迹，用基于距离的 softmax 代理目标做策略梯度，从而保留确定性采样与高阶求解器的少步采样优势，可直接替换现有 SDE-GRPO 训练流程。
原文依据 Neighbor GRPO generates a diverse set of candidate trajectories by perturbing the initial noise conditions of the ODE
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组相对策略优化（GRPO）在使图像与视频生成模型对齐人类偏好方面已展现出潜力。然而，由于其确定性采样范式，将其应用于现代流匹配模型颇具挑战。现有方法通过将常微分方程（ODE）转换为随机微分方程（SDE）来引入随机性，以此解决该问题。但基于 SDE 的 GRPO 存在信用分配效率低下、且与用于少步采样的高阶求解器不兼容的问题。本文首先从距离优化的视角重新解读现有的基于 SDE 的 GRPO 方法，揭示其内在机制本质上是一种对比学习。基于这一洞见，我们提出 Neighbor GRPO，一种完全无需 SDE 的新型对齐算法。Neighbor GRPO 通过扰动 ODE 的初始噪声条件生成一组多样化的候选轨迹，并使用基于 softmax 距离的代理跳跃策略对模型进行优化。我们建立了该基于距离的目标与策略梯度优化之间的理论联系，从而将我们的方法严格纳入 GRPO 框架。我们的方法完整保留了确定性 ODE 采样的优势，包括高效性以及与高阶求解器的兼容性。我们进一步引入对称锚框采样以提升计算效率，并引入组内拟范数重加权以缓解奖励平坦化问题。大量实验表明，Neighbor GRPO 在训练成本、收敛速度和生成质量方面均显著优于基于 SDE 的同类方法。
英文原文摘要 1685 字符
Group Relative Policy Optimization (GRPO) has shown promise in aligning image and video generative models with human preferences. However, applying it to modern flow matching models is challenging because of its deterministic sampling paradigm. Current methods address this issue by converting Ordinary Differential Equations (ODEs) to Stochastic Differential Equations (SDEs), which introduce stochasticity. However, this SDE-based GRPO suffers from issues of inefficient credit assignment and incompatibility with high-order solvers for fewer-step sampling. In this paper, we first reinterpret existing SDE-based GRPO methods from a distance optimization perspective, revealing their underlying mechanism as a form of contrastive learning. Based on this insight, we propose Neighbor GRPO, a novel alignment algorithm that completely bypasses the need for SDEs. Neighbor GRPO generates a diverse set of candidate trajectories by perturbing the initial noise conditions of the ODE and optimizes the model using a softmax distance-based surrogate leaping policy. We establish a theoretical connection between this distance-based objective and policy gradient optimization, rigorously integrating our approach into the GRPO framework. Our method fully preserves the advantages of deterministic ODE sampling, including efficiency and compatibility with high-order solvers. We further introduce symmetric anchor sampling for computational efficiency and group-wise quasi-norm reweighting to address reward flattening. Extensive experiments demonstrate that Neighbor GRPO significantly outperforms SDE-based counterparts in terms of training cost, convergence speed, and generation quality.