基于扩散模型的面部区域感知妆容特征妆容迁移
Diffusion-Based Makeup Transfer with Facial Region-Aware Makeup Features
用可学习 token 查询微调后的 CLIP 编码器，提取眼、嘴等面部区域级妆容特征，并以注意力损失实现区域可控注入，为扩散编辑中的局部属性控制提供了可复用范式；同时用 GPT-o3 加文本驱动编辑模型合成带标注风格数据来微调领域编码器，可迁移到其他细粒度风格迁移任务。
原文依据 we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前基于扩散模型的妆容迁移方法通常以现成基础模型（如 CLIP）编码的妆容信息作为条件，在生成过程中保持参考图像的妆容风格。尽管这类方法有效，但主要存在两点局限：（1）为通用任务预训练的基础模型难以捕捉妆容风格；（2）参考图像的妆容特征被整体注入扩散去噪模型以进行全局妆容迁移，忽视了面部区域感知的妆容特征（即眼睛、嘴部等），限制了对特定区域进行妆容迁移的区域可控性。为解决上述问题，本文提出面部区域感知妆容特征（FRAM），包含两个阶段：（1）妆容 CLIP 微调；（2）身份与面部区域感知的妆容注入。在妆容 CLIP 微调方面，与先前使用现成 CLIP 的工作不同，我们利用 GPT-o3 与文本驱动的图像编辑模型合成带标注的妆容风格数据，再通过自监督学习与图像-文本对比学习在该数据上训练妆容 CLIP 编码器。在身份与面部区域感知的妆容注入方面，我们由第一阶段编辑后的图像构建妆前妆后图像对，并利用这些图像对学习将源图像的身份与参考图像的妆容注入扩散去噪模型以完成妆容迁移。具体而言，我们使用可学习 token 查询妆容 CLIP 编码器，提取面部区域感知的妆容特征用于妆容注入，并通过注意力损失进行学习以实现区域控制。在身份注入方面，我们使用 ControlNet Union 同时编码源图像及其三维网格。实验结果验证了我们在区域可控性与妆容迁移性能上的优越性。
英文原文摘要 1840 字符
Current diffusion-based makeup transfer methods commonly use the makeup information encoded by off-the-shelf foundation models (e.g., CLIP) as condition to preserve the makeup style of reference image in the generation. Although effective, these works mainly have two limitations: (1) foundation models pre-trained for generic tasks struggle to capture makeup styles; (2) the makeup features of reference image are injected to the diffusion denoising model as a whole for global makeup transfer, overlooking the facial region-aware makeup features (i.e., eyes, mouth, etc) and limiting the regional controllability for region-specific makeup transfer. To address these, in this work, we propose Facial Region-Aware Makeup features (FRAM), which has two stages: (1) makeup CLIP fine-tuning; (2) identity and facial region-aware makeup injection. For makeup CLIP fine-tuning, unlike prior works using off-the-shelf CLIP, we synthesize annotated makeup style data using GPT-o3 and text-driven image editing model, and then use the data to train a makeup CLIP encoder through self-supervised and image-text contrastive learning. For identity and facial region-aware makeup injection, we construct before-and-after makeup image pairs from the edited images in stage 1 and then use them to learn to inject identity of source image and makeup of reference image to the diffusion denoising model for makeup transfer. Specifically, we use learnable tokens to query the makeup CLIP encoder to extract facial region-aware makeup features for makeup injection, which is learned via an attention loss to enable regional control. As for identity injection, we use a ControlNet Union to encode source image and its 3D mesh simultaneously. The experimental results verify the superiority of our regional controllability and our makeup transfer performance.