面向多角色叙事图像生成的多方面人类偏好对齐
Aligning Multi-Character Narrative Image Generation with Multi-Aspect Human Preferences
提供了面向多角色叙事图像生成的细粒度人类偏好数据集 NI-RLHF 与可解释奖励模型 NIReward，以及按偏好维度自适应平衡的 ADPO 优化策略，可直接迁移到其他需要多维人类偏好对齐的扩散生成任务（如个性化、可控生成）的微调阶段。
原文依据 we propose Adaptive Dominance-based Preference Optimization (ADPO) to balance learning across diverse preference dimensions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论叙事图像生成旨在创建包含多个不同角色并刻画其相互关系的图像，这对当前文本到图像扩散模型构成了重大挑战。因此，通用的个性化方法往往存在语义对齐差、身份混淆和美学不合理等问题。现有评估指标（如 CLIP、ArcFace 和常规奖励模型）无法充分捕捉这些问题，从根本上无法与人类感知偏好保持一致。为与人类偏好对齐，我们首先构建了一个细粒度人类偏好数据集 NI-RLHF，围绕提示词遵循、身份一致性和视觉质量三个核心维度，同时收集了详细的人类评语与偏好判断。该全面数据集有助于训练 NIReward，这是一个基于评语的奖励模型，能够生成可解释的图像评估。基于 NIReward 提供的可解释奖励信号，我们提出了自适应主导偏好优化（ADPO），以平衡不同偏好维度上的学习，同时动态适应奖励边际。实验结果表明，NIReward 显著优于现有评估模型和奖励模型，ADPO 在三个关键偏好维度上均带来显著提升。通过引入 NIReward 和 ADPO，我们的工作为生成符合真实人类偏好的叙事图像铺平了道路。
英文原文摘要 1499 字符
Narrative image generation aims to create images featuring multiple distinct characters while capturing their interrelationships, posing significant challenges for current text-to-image diffusion models. As a result, general personalized methods often suffer from poor semantic alignment, identity blending, and aesthetic implausibility.These issues are inadequately captured by existing evaluation metrics such as CLIP, ArcFace, and conventional reward models, which fundamentally fail to align with human perceptual preferences. To align with human preferences, we first construct a fine-grained human preference dataset, NI-RLHF, by collecting both detailed human critiques and preference judgments across three core dimensions: prompt following, identity consistency, and visual quality.This comprehensive dataset facilitates the training of NIReward, a critique-based reward model capable of generating interpretable image evaluations.Building upon the interpretable reward signal from NIReward, we propose Adaptive Dominance-based Preference Optimization (ADPO) to balance learning across diverse preference dimensions while dynamically adapting to reward margins.Experimental results indicate that NIReward significantly outperforms existing evaluation models and reward models, and ADPO yields a significant improvement across the three key preference dimensions. By introducing NIReward and ADPO, our work paves the way for generating narrative images aligned with actual human preferences.