面向说话人脸视频情感编辑的跨模态情感迁移
Cross-Modal Emotion Transfer for Emotion Editing in Talking Face Video
用大规模预训练音频编码器与解耦表情编码器在跨模态空间中学习情感语义向量，把「情感」从语音内容中剥离出来，为语音驱动的可控人脸动画提供了一种可迁移的条件控制手段，尤其适合处理未见过的扩展情感类别。
原文依据 modeling emotion semantic vectors between speech and visual feature spaces
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论说话人脸生成作为生成模型的核心应用已受到广泛关注。为提升合成视频的表现力与真实感，说话人脸视频中的情感编辑起着关键作用。然而，现有方法往往限制了表情的灵活性，且难以生成扩展情感。基于标签的方法用离散类别表示情感，无法捕捉广泛的情感范围。基于音频的方法可以利用情感丰富的语音信号——甚至受益于富有表现力的文本到语音（TTS）合成——但由于情感与语言内容在情感语音中相互纠缠，它们无法表达目标情感。另一方面，基于图像的方法依赖目标参考图像来引导情感迁移，但需要高质量的正脸视图，并且在获取扩展情感（例如讽刺）的参考数据方面面临挑战。为解决这些局限，我们提出跨模态情感迁移（C-MET），一种通过对语音与视觉特征空间之间的情感语义向量建模、基于语音生成面部表情的新方法。C-MET 利用大规模预训练音频编码器与解耦的面部表情编码器，学习表示跨模态两个不同情感嵌入之间差异的情感语义向量。在 MEAD 与 CREMA-D 数据集上的大量实验表明，我们的方法相较最优方法将情感准确率提升 14%，同时生成富有表现力的说话人脸视频——即使对未见过的扩展情感也是如此。代码、检查点与演示见 https://chanhyeok-choi.github.io/C-MET/。
英文原文摘要 1692 字符
Talking face generation has gained significant attention as a core application of generative models.To enhance the expressiveness and realism of synthesized videos, emotion editing in talking face video plays a crucial role.However, existing approaches often limit expressive flexibility and struggle to generate extended emotions.Label-based methods represent emotions with discrete categories, which fail to capture a wide range of emotions.Audio-based methods can leverage emotionally rich speech signals--and even benefit from expressive text-to-speech (TTS) synthesis--but they fail to express the target emotions because emotions and linguistic contents are entangled in emotional speeches.Images-based methods, on the other hand, rely on target reference images to guide emotion transfer, yet they require high-quality frontal views and face challenges in acquiring reference data for extended emotions (e.g., sarcasm).To address these limitations, we propose Cross-Modal Emotion Transfer (C-MET), a novel approach that generates facial expressions based on speeches by modeling emotion semantic vectors between speech and visual feature spaces.C-MET leverages a large-scale pretrained audio encoder and a disentangled facial expression encoder to learn emotion semantic vectors that represent the difference between two different emotional embeddings across modalities.Extensive experiments on the MEAD and CREMA-D datasets demonstrate that our method improves emotion accuracy by 14% over state-of-the-art methods, while generating expressive talking face videos--even for unseen extended emotions. Code, checkpoint, and demo are available at https://chanhyeok-choi.github.io/C-MET/.