扩展与剪枝：最大化轨迹多样性以实现生成模型中有效的 GRPO
Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models
Pro-GRPO 将基于潜变量的轨迹剪枝嵌入采样过程，通过提前终止奖励聚集的轨迹来降低计算开销，使大 group size 的 GRPO 训练在扩散/流模型中变得可行；其 Expand-and-Prune 策略可直接用于视觉生成模型的强化学习对齐。
原文依据 we propose Pro-GRPO (Proactive GRPO), a novel dynamic framework that integrates latent feature-based trajectory pruning into the sampling process.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组相对策略优化（GRPO）是一种用于对齐生成模型的强大技术，但其有效性受到大组规模与高昂计算成本之间冲突的瓶颈制约。在本工作中，我们通过实证研究考察了这一权衡，并得出两个关键观察。首先，我们发现了奖励聚集现象，即许多轨迹向组平均奖励坍缩，提供的优化价值有限。其次，我们设计了一种名为最优方差过滤（OVF）的启发式策略，并验证了由 OVF 选出的高方差轨迹子集能够优于更大的未过滤组。然而，这种静态的、采样后执行的 OVF 方法仍然需要关键的计算开销，因为它对最终被丢弃的轨迹执行了不必要的采样。为解决这一问题，我们提出了 Pro-GRPO（主动 GRPO），一种新颖的动态框架，将基于潜变量特征的轨迹剪枝集成到采样过程中。通过提前终止奖励聚集的轨迹，Pro-GRPO 降低了计算开销。利用其效率，Pro-GRPO 采用“扩展与剪枝”策略。该策略首先扩展初始采样组的规模以最大化轨迹多样性，然后对潜变量应用多步 OVF，从而避免高昂的计算成本。在基于扩散和基于流的模型上进行的大量实验证明了我们 Pro-GRPO 框架的通用性和有效性。
英文原文摘要 1521 字符
Group Relative Policy Optimization (GRPO) is a powerful technique for aligning generative models, but its effectiveness is bottlenecked by the conflict between large group sizes and prohibitive computational costs. In this work, we investigate the trade-off through empirical studies, yielding two key observations. First, we discover the reward clustering phenomenon in which many trajectories collapse toward the group-mean reward, offering limited optimization value. Second, we design a heuristic strategy named Optimal Variance Filtering (OVF), and verify that a high-variance subset of trajectories selected by OVF can outperform the larger, unfiltered group. However, this static, post-sampling OVF approach still necessitates critical computational overhead, as it performs unnecessary sampling for trajectories that are ultimately discarded. To resolve this, we propose Pro-GRPO (Proactive GRPO), a novel dynamic framework that integrates latent feature-based trajectory pruning into the sampling process. Through the early termination of reward-clustered trajectories, Pro-GRPO reduces computational overhead. Leveraging its efficiency, Pro-GRPO employs an "Expand-and-Prune" strategy. This strategy first expands the size of initial sampling group to maximize trajectory diversity, then it applies multi-step OVF to the latents, avoiding prohibitive computational costs. Extensive experiments on both diffusion-based and flow-based models demonstrate the generality and effectiveness of our Pro-GRPO framework.