GDPO-SR：面向单步生成式图像超分辨率的组直接偏好优化
GDPO-SR: Group Direct Preference Optimization for One-Step Generative Image Super-Resolution
将GRPO的组相对优势思想引入DPO，实现在线生成样本的偏好优化，并设计属性感知奖励函数动态评估局部细节，可复用于其他单步扩散复原任务（如去噪、去模糊）的微调。
原文依据 we propose Group Direct Preference Optimization (GDPO), a novel approach to integrate RL into one-step generative ISR model training.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期，强化学习（RL）已被用于提升生成式图像超分辨率（ISR）的性能。然而，当前工作主要集中于多步生成式ISR，而单步生成式ISR因其随机性有限仍探索不足。此外，直接偏好优化（DPO）等RL方法需要离线生成正负样本对，导致样本数量有限，而组相对策略优化（GRPO）仅计算整幅图像的似然，忽略了对于ISR至关重要的局部细节。本文提出组直接偏好优化（GDPO），一种将RL融入单步生成式ISR模型训练的新方法。首先，我们引入一种噪声感知的单步扩散模型，能够生成多样化的ISR输出。为防止噪声注入导致的性能下降，我们采用不等时间步策略，将噪声添加的时间步与扩散的时间步解耦。然后，我们提出GDPO策略，将GRPO的原理融入DPO，计算每个在线生成样本的组相对优势以进行模型优化。同时，设计了一种属性感知奖励函数，根据样本的平滑区域和纹理区域的统计量动态评估每个样本的得分。实验证明了GDPO在提升单步生成式ISR模型性能方面的有效性。代码：https://github.com/Joyies/GDPO。
英文原文摘要 1519 字符
Recently, reinforcement learning (RL) has been employed for improving generative image super-resolution (ISR) performance. However, the current efforts are focused on multi-step generative ISR, while one-step generative ISR remains underexplored due to its limited stochasticity. In addition, RL methods such as Direct Preference Optimization (DPO) require the generation of positive and negative sample pairs offline, leading to a limited number of samples, while Group Relative Policy Optimization (GRPO) only calculates the likelihood of the entire image, ignoring local details that are crucial for ISR. In this paper, we propose Group Direct Preference Optimization (GDPO), a novel approach to integrate RL into one-step generative ISR model training. First, we introduce a noise-aware one-step diffusion model that can generate diverse ISR outputs. To prevent performance degradation caused by noise injection, we introduce an unequal-timestep strategy to decouple the timestep of noise addition from that of diffusion. We then present the GDPO strategy, which integrates the principle of GRPO into DPO, to calculate the group-relative advantage of each online generated sample for model optimization. Meanwhile, an attribute-aware reward function is designed to dynamically evaluate the score of each sample based on its statistics of smooth and texture areas. Experiments demonstrate the effectiveness of GDPO in enhancing the performance of one-step generative ISR models. Code: https://github.com/Joyies/GDPO.