通过方向解耦对齐驯服扩散强化学习中的偏好模式坍缩
Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learning
针对文生图扩散模型做 RLHF 时出现的偏好模式坍缩（奖励黑客）问题，提出在奖励模型嵌入空间中学习一个方向性修正并作用于奖励信号，可直接用于任何基于奖励优化的扩散微调流程以保住生成多样性；配套的 DivGenBench 也提供了衡量多样性坍缩的评测手段。
原文依据 we propose Directional Decoupling Alignment (D^2-Align), a novel framework that mitigates PMC by directionally correcting the reward signal
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期研究表明，通过人类反馈强化学习将文生图扩散模型与人类偏好对齐已取得显著进展。然而，现有方法虽然在自动奖励指标上取得高分，却往往导致偏好模式坍缩（PMC）——这是奖励黑客的一种特定形式，模型会收敛到狭窄的高分输出（例如风格单一或普遍过曝的图像），严重损害生成多样性。在本工作中，我们引入并量化了这一现象，提出 DivGenBench，一个用于衡量 PMC 程度的新基准。我们认为这种坍缩是由沿奖励模型固有偏置的过度优化所驱动的。基于这一分析，我们提出方向解耦对齐（D^2-Align），一个通过对奖励信号进行方向性修正来缓解 PMC 的新框架。具体而言，我们的方法首先在奖励模型的嵌入空间中学习一种方向性修正，同时保持模型冻结。随后在优化过程中将该修正施加于奖励信号，防止模型坍缩到特定模式，从而保持多样性。我们结合定性分析与针对质量与多样性的定量指标进行综合评估，结果表明 D^2-Align 实现了与人类偏好更优的对齐。
英文原文摘要 1403 字符
Recent studies have demonstrated significant progress in aligning text-to-image diffusion models with human preference via Reinforcement Learning from Human Feedback. However, while existing methods achieve high scores on automated reward metrics, they often lead to Preference Mode Collapse (PMC)-a specific form of reward hacking where models converge on narrow, high-scoring outputs (e.g., images with monolithic styles or pervasive overexposure), severely degrading generative diversity. In this work, we introduce and quantify this phenomenon, proposing DivGenBench, a novel benchmark designed to measure the extent of PMC. We posit that this collapse is driven by over-optimization along the reward model's inherent biases. Building on this analysis, we propose Directional Decoupling Alignment (D^2-Align), a novel framework that mitigates PMC by directionally correcting the reward signal.Specifically, our method first learns a directional correction within the reward model's embedding space while keeping the model frozen.This correction is then applied to the reward signal during the optimization process, preventing the model from collapsing into specific modes and thereby maintaining diversity. Our comprehensive evaluation, combining qualitative analysis with quantitative metrics for both quality and diversity, reveals that D^2-Align achieves superior alignment with human preference.