DRM：基于扩散模型并带逐步引导的奖励模型
DRM: Diffusion-based Reward Model With Step-wise Guidance
把预训练扩散模型本身当作奖励骨干，利用其对含噪中间潜变量的逐步评估能力，为扩散模型对齐提供密集的每步奖励与推理时的路径引导，可直接迁移到任意扩散/流匹配模型的偏好优化与采样搜索。
原文依据 a novel paradigm that use the pre-trained diffusion model as a powerful evaluative backbone
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前将扩散模型与人类偏好对齐的主流方法通常采用基于视觉语言模型（VLM）的奖励模型。然而，这些为语义对齐而预训练的奖励模型难以捕捉关键的感知品质——例如美学、构图与视觉和谐。在本工作中，我们认为一个能够实现高保真生成的模型必然对这些视觉属性具有深刻理解。基于这一洞见，我们提出了基于扩散模型的奖励模型（DRM），这是一种以预训练扩散模型作为强大评估骨干的新范式。DRM 的一个关键优势在于其独特能力：它不仅能评估最终图像，还能评估生成过程任意阶段中含噪的中间潜变量。我们以两种方式利用这种逐步评估能力。首先，我们提出了逐步 GRPO（Step-wise GRPO），一种强化学习算法，它提供密集的每步奖励，以解决 GRPO 算法中信用分配不精确的问题，从而实现更稳定、更有效的对齐。其次，我们引入了逐步采样（Step-wise Sampling），一种新颖的推理策略，它将 DRM 用作动态引导，在每一步评估多条生成路径，从而将生成过程导向更高质量的结果。大量实验证实，我们的方法显著提升了生成图像的最终质量。
英文原文摘要 1368 字符
Current mainstream methods of aligning diffusion models with human preferences typically employ VLM-based reward models.However, these reward models, pre-trained for semantic alignment, struggle to capture the essential perceptual qualities--such as aesthetics, composition, and visual harmony.In this work, we argue that a model capable of high-fidelity generation must possess a profound understanding of these visual attributes.Based on this insight, we introduce the Diffusion-based Reward Model (DRM), a novel paradigm that use the pre-trained diffusion model as a powerful evaluative backbone.A key advantage of the DRM is its unique ability to assess not only the final image but also the noisy intermediate latents at any stage of the generative process. We leverage this step-wise evaluative capacity in two ways.First, we propose Step-wise GRPO, a reinforcement learning algorithm that provides dense, per-step rewards to resolve the imprecise credit assignment problem in GRPO algorithm, leading to more stable and effective alignment.Second, we introduce Step-wise Sampling, a novel inference strategy that employs the DRM as a dynamic guide to evaluate multiple generation paths at each step, steering the process towards higher-quality outcomes.Extensive experiments confirm that our approach significantly enhances the final quality of generated images.