迈向细粒度归因：用于对齐扩散模型的实例感知偏好优化
Towards Fine-Grained Attribution: Instance-Aware Preference Optimization for Aligning Diffusion Models
将扩散模型对齐的偏好优化从图像级推进到实例级，通过自动构建实例级偏好数据集和动态重加权掩码，为视觉生成模型提供更细粒度的反馈信号，可复用于其他需要局部控制的生成任务。
原文依据 we propose IAPO, an Instance-Aware Preference Optimization that introduces instance-level credit assignment to advance alignment from image-level to instance-level.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论直接偏好优化在使扩散模型与人类反馈对齐方面取得了显著成功。然而，现有方法严重依赖图像级偏好，在空间维度上存在奖励稀疏的问题。这造成了一种根本性的错位：一幅图像可能在全局上更受偏好，但其中可能包含局部较差的实例。因此，将相同的正偏好应用于这些区域，会不公平地奖励干扰区域而惩罚信息丰富的区域，导致性能次优和学习效率低下。为了解决这个问题，我们提出了IAPO，一种实例感知偏好优化方法，它引入实例级信用分配，将对齐从图像级推进到实例级。我们首先利用视觉语言模型和目标检测模型自动识别并重新标注图像对中对应的实例，从而构建一个高质量的实例级偏好数据集。利用这个细粒度数据集，我们设计了一种新颖的实例对齐损失，带有动态重加权掩码，该掩码在标注的边界框内调制实例级损失，抑制干扰物以强制执行细粒度的人类偏好对齐。大量实验表明，我们的方法不仅在多个基准测试中达到了最优性能，而且由于细粒度的实例级偏好标签，还获得了更高的训练效率。
英文原文摘要 1425 字符
Direct Preference Optimization has achieved remarkable success in aligning diffusion models with human feedback. However, existing methods heavily rely on image-level preferences, which suffer from sparse rewards in the spatial dimension. This creates a fundamental misalignment: while an image may be globally preferred, it can contain locally inferior instances. Applying the same positive preference to these areas thus unfairly credits distracting regions while penalizing informative ones, leading to suboptimal performance and inefficient learning. To resolve this issue, we propose IAPO, an Instance-Aware Preference Optimization that introduces instance-level credit assignment to advance alignment from image-level to instance-level. We first construct a high-quality instance-level preference dataset by automatically identifying and relabeling corresponding instances in image pairs using vision-language models and object detection models. Leveraging this fine-grained dataset, we design a novel instance alignment loss with a dynamic reweighting mask that modulates instance-level loss within annotated bounding boxes, suppressing distractors to enforce fine-grained human preference alignment. Extensive experiments demonstrate that our method not only achieves state-of-the-art performance in multiple benchmarks but also attains higher training efficiency due to fine-grained instance-level preference labels.