GRPO-Guard：通过受调控的裁剪缓解流匹配中的隐式过度优化
GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping
对做扩散/流匹配模型 RL 后训练的人，这篇给出了一个可直接替换 PPO clipping 的比值归一化 + 梯度重加权模块，用来防止代理奖励上涨而图像质量与文本对齐崩塌；可作为 SD3.5、Flux 等骨干上 GRPO 微调的稳定化插件。
原文依据 we introduce GRPO-Guard, a simple yet effective enhancement to existing GRPO frameworks
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期，基于 GRPO 的强化学习在优化流匹配模型方面取得了显著进展，有效提升了其与任务特定奖励的对齐程度。在这些框架中，策略更新依赖重要性比值裁剪来约束过度自信的正梯度与负梯度。然而在实际中，我们观察到重要性比值分布出现系统性偏移——其均值低于 1，且在不同时间步上方差差异显著。这种左偏且不一致的分布使正优势样本无法进入裁剪区域，导致该机制无法约束过度自信的正向更新。结果，策略模型不可避免地进入隐式过度优化阶段——代理奖励持续上升，而图像质量与文本提示对齐等关键指标却急剧恶化，最终使所学策略难以在实际场景中使用。为解决该问题，我们提出 GRPO-Guard，这是对现有 GRPO 框架的一种简单而有效的增强。该方法引入比值归一化，恢复平衡且跨步一致的重要性比值，确保 PPO 裁剪能在各去噪时间步上正确约束有害更新。此外，一种梯度重加权策略均衡了不同噪声条件下的策略梯度，防止来自特定时间步区域的过度更新。这些设计共同构成受调控的裁剪机制，在不依赖繁重 KL 正则的情况下稳定优化并显著缓解隐式过度优化。在多种扩散骨干（如 SD3.5M、Flux.1-dev）与多种代理任务上的大量实验表明，GRPO-Guard 显著减少过度优化，同时保持甚至提升生成质量。我们在 https://jingw193.github.io/GRPO-Guard/ 提供了过度优化过程的详细演示与相应可视化。
英文原文摘要 2006 字符
Recently, GRPO-based reinforcement learning has shown remarkable progress in optimizing flow-matching models, effectively improving their alignment with task-specific rewards. Within these frameworks, the policy update relies on importance-ratio clipping to constrain overconfident positive and negative gradients. However, in practice, we observe a systematic shift in the importance-ratio distribution--its mean falls below 1 and its variance differs substantially across timesteps. This left-shifted and inconsistent distribution prevents positive-advantage samples from entering the clipped region, causing the mechanism to fail in constraining overconfident positive updates. As a result, the policy model inevitably enters an **implicit over-optimization stage**--while the proxy reward continues to increase, essential metrics such as image quality and text-prompt alignment deteriorate sharply, ultimately making the learned policy impractical for real-world use. To address this issue, we introduce **GRPO-Guard**, a simple yet effective enhancement to existing GRPO frameworks. Our method incorporates ratio normalization, which restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates across denoising timesteps. In addition, a gradient reweighting strategy equalizes policy gradients over noise conditions, preventing excessive updates from particular timestep regions. Together, these designs act as a regulated clipping mechanism, stabilizing optimization and substantially mitigating implicit over-optimization without relying on heavy KL regularization. Extensive experiments on multiple diffusion backbones (e.g., SD3.5M, Flux.1-dev) and diverse proxy tasks demonstrate that GRPO-Guard significantly reduces over-optimization while maintaining or even improving generation quality. We provide detailed demonstrations of the over-optimization process and corresponding visualizations in https://jingw193.github.io/GRPO-Guard/.