图像即奖励：面向图像生成的对抗奖励强化学习
The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
用参考图像和视觉基础模型（如 DINO）提供密集视觉奖励，替代标量偏好奖励，可缓解奖励攻击并提升生成质量；该对抗奖励框架可复用于扩散模型或流匹配模型的强化学习微调。
原文依据 we take the image itself as a reward, using reference images and vision foundation models (e.g., DINO) to provide rich visual rewards.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论可靠的奖励函数对于图像生成中的强化学习（RL）至关重要。当前大多数强化学习方法依赖预训练的偏好模型输出标量奖励来近似人类偏好。然而，这些奖励往往无法捕捉人类感知，并且容易受到奖励攻击，即更高的分数并不对应更好的图像。为了解决这个问题，我们引入了 Adv-GRPO，一个带有对抗奖励的强化学习框架，它迭代地更新奖励模型和生成器。奖励模型以参考图像作为正样本进行监督，并且能够在很大程度上避免被攻击。与约束参数更新的 KL 正则化不同，我们学习到的奖励通过其视觉输出直接引导生成器，从而产生更高质量的图像。此外，虽然优化现有的奖励函数可以缓解奖励攻击，但它们固有的偏差仍然存在。例如，PickScore 可能会降低图像质量，而基于 OCR 的奖励通常会降低美学保真度。为了解决这个问题，我们将图像本身作为奖励，使用参考图像和视觉基础模型（例如 DINO）来提供丰富的视觉奖励。这些密集的视觉信号而非单个标量，在图像质量、美学和任务特定指标上带来了一致的提升。最后，我们展示了将参考样本与基础模型奖励相结合可以实现分布迁移和灵活的风格定制。在人类评估中，我们的方法优于 Flow-GRPO 和 SD3，在图像质量和美学方面分别达到了 70.0% 和 72.4% 的胜率。代码和模型已发布在 https://github.com/showlab/Adv-GRPO。
英文原文摘要 1687 字符
A reliable reward function is essential for reinforcement learning (RL) in image generation. Most current RL approaches depend on pre-trained preference models that output scalar rewards to approximate human preferences. However, these rewards often fail to capture human perception and are vulnerable to reward hacking, where higher scores do not correspond to better images. To address this, we introduce Adv-GRPO, an RL framework with an adversarial reward that iteratively updates both the reward model and the generator. The reward model is supervised using reference images as positive samples and can largely avoid being hacked. Unlike KL regularization that constrains parameter updates, our learned reward directly guides the generator through its visual outputs, leading to higher-quality images. Moreover, while optimizing existing reward functions can alleviate reward hacking, their inherent biases remain. For instance, PickScore may degrade image quality, whereas OCR-based rewards often reduce aesthetic fidelity. To address this, we take the image itself as a reward, using reference images and vision foundation models (e.g., DINO) to provide rich visual rewards. These dense visual signals, instead of a single scalar, lead to consistent gains across image quality, aesthetics, and task-specific metrics. Finally, we show that combining reference samples with foundation-model rewards enables distribution transfer and flexible style customization. In human evaluation, our method outperforms Flow-GRPO and SD3, achieving 70.0% and 72.4% win rates in image quality and aesthetics, respectively. Code and models has been released in https://github.com/showlab/Adv-GRPO.