CRAFT：对齐扩散模型，微调比你想象的更容易
CRAFT: Aligning Diffusion Models with Fine-Tuning Is Easier Than You Think
用复合奖励过滤构造小而一致的偏好数据集，再做增强版 SFT，就能以 100 个样本超越数千对偏好数据的 DPO 类方法；做扩散模型人类偏好对齐时，可把数据筛选环节当作主要杠杆，而不是堆偏好对数量。
原文依据 CRAFT with only 100 samples can easily outperform recent SOTA preference optimization methods with thousands of preference-paired samples
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论对齐扩散模型已在生成高质量、符合人类偏好的图像方面取得显著突破。现有技术，如有监督微调（SFT）和 DPO 式偏好优化，已成为微调扩散模型的原则性工具。然而，SFT 依赖获取成本高昂的高质量图像，而 DPO 式方法依赖大规模偏好数据集，其质量往往不一致。除数据依赖外，这些方法还受计算效率低下的制约。为应对这两项挑战，我们提出复合奖励辅助微调（CRAFT），一种轻量却强大的微调范式，它显著减少了所需训练数据，同时保持计算效率。该方法首先利用复合奖励过滤（CRF）技术构建高质量且一致的训练数据集，然后执行 SFT 的增强变体。我们还从理论上证明，CRAFT 实际上优化了基于组的强化学习的下界，从而在带数据筛选的 SFT 与强化学习之间建立了原则性联系。我们的大量实证结果表明，仅用 100 个样本的 CRAFT 即可轻松超越近期使用数千个偏好配对样本的 SOTA 偏好优化方法。此外，CRAFT 的收敛速度甚至可比基线偏好优化方法快 11–220 倍，凸显其极高的效率。
英文原文摘要 1509 字符
Aligning Diffusion models has achieved remarkable breakthroughs in generating high-quality, human preference-aligned images. Existing techniques, such as supervised fine-tuning (SFT) and DPO-style preference optimization, have become principled tools for fine-tuning diffusion models. However, SFT relies on high-quality images that are costly to obtain, while DPO-style methods depend on large-scale preference datasets, which are often inconsistent in quality. Beyond data dependency, these methods are further constrained by computational inefficiency. To address these two challenges, we propose Composite Reward Assisted Fine-Tuning (CRAFT), a lightweight yet powerful fine-tuning paradigm that requires significantly reduced training data while maintaining computational efficiency. It first leverages a Composite Reward Filtering (CRF) technique to construct a high-quality and consistent training dataset and then perform an enhanced variant of SFT. We also theoretically prove that CRAFT actually optimizes the lower bound of group-based reinforcement learning, establishing a principled connection between SFT with selected data and reinforcement learning. Our extensive empirical results demonstrate that CRAFT with only 100 samples can easily outperform recent SOTA preference optimization methods with thousands of preference-paired samples. Moreover, CRAFT can even achieve 11-220xfaster convergences than the baseline preference optimization methods, highlighting its extremely high efficiency.