面向扩散模型的奖励锐度感知微调
Reward Sharpness-Aware Fine-Tuning for Diffusion Models
针对扩散模型 RLHF 微调中的奖励攻击问题，提出用参数扰动与样本扰动构造平坦化奖励模型的梯度，无需重训奖励模型即可提升生成质量与奖励对齐的可靠性，可直接嵌入现有扩散强化学习流程。
原文依据 we employ gradients from a flattened reward model, obtained through parameter perturbations of the diffusion model and perturbations of its generated samples
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于人类反馈的强化学习（RLHF）已被证明能有效使大语言模型与人类偏好对齐，这启发了以奖励为中心的扩散强化学习（RDRL）的发展，以实现类似的对齐与可控性。尽管扩散模型能够生成高质量输出，RDRL 仍易受奖励攻击影响，即奖励分数上升而感知质量并未相应提升。我们证明这一脆弱性源于奖励模型梯度的非鲁棒性，尤其是当相对于输入图像的奖励景观较为尖锐时。为缓解该问题，我们引入了利用鲁棒化奖励模型梯度的方法，且无需对其重新训练。具体而言，我们采用来自平坦化奖励模型的梯度，该模型通过对扩散模型进行参数扰动以及对其生成样本进行扰动而获得。实验上，每种方法都能独立缓解奖励攻击并提升鲁棒性，而联合使用则进一步放大这些收益。我们由此得到的框架 RSA-FT（奖励锐度感知微调）简单、广泛兼容，并持续提升 RDRL 的可靠性。
英文原文摘要 1231 字符
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models with human preferences, inspiring the development of reward-centric diffusion reinforcement learning (RDRL) to achieve similar alignment and controllability. While diffusion models can generate high-quality outputs, RDRL remains susceptible to reward hacking, where the reward score increases without corresponding improvements in perceptual quality. We demonstrate that this vulnerability arises from the non-robustness of reward model gradients, particularly when the reward landscape with respect to the input image is sharp. To mitigate this issue, we introduce methods that exploit gradients from a robustified reward model, without requiring its retraining. Specifically, we employ gradients from a flattened reward model, obtained through parameter perturbations of the diffusion model and perturbations of its generated samples. Empirically, each method independently alleviates reward hacking and improves robustness, while their joint use amplifies these benefits. Our resulting framework, RSA-FT (Reward Sharpness-Aware Fine-Tuning), is simple, broadly compatible, and consistently enhances the reliability of RDRL.