PC-Talk：音频驱动说话人脸生成的精确面部动画控制
PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation
通过隐式关键点变形实现唇音对齐与情感解耦控制，为音频驱动人脸生成提供可复用的精细控制手段，支持说话风格编辑与情感强度/混合调控。
原文依据 we propose a novel and efficient framework, PC-Talk, which enables lip-audio alignment and emotion control through implicit keypoint deformations.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期音频驱动说话人脸生成在唇形同步方面取得了巨大进展。然而，当前方法往往缺乏对说话人脸的充分控制，例如说话风格和情感表达，导致面部运动单一。本文聚焦于改进两个关键因素：唇音对齐控制（LAC）和情感控制（EMC），以增强说话视频的多样性和用户友好性。唇音对齐控制确保在不同说话风格下实现准确的唇形同步，以模拟不同的说话习惯；而情感控制旨在生成具有不同强度和混合情感状态的真实情感表情。为实现精确的面部动画控制，我们提出了一种新颖且高效的框架PC-Talk，通过隐式关键点变形实现唇音对齐和情感控制。首先，我们的LAC模块生成具有特定说话风格的唇形同步说话人脸，该风格可源自视频参考或预设选项。它还支持唇动尺度调整以及对特定发音的说话风格进行细粒度编辑。其次，我们的EMC模块通过纯情感变形生成生动的情感面部表情。它进一步能够精确控制情感强度以及不同面部区域的复合情感。我们的方法在实验中展示了出色的控制能力，并在HDTF和MEAD数据集上取得了最先进的性能。项目页面：https://bq-wang0511.github.io/PC-Talk/
英文原文摘要 2061 字符
Recent advancements in audio-driven talking face generation have made great progress in lip synchronization. However, current methods often lack sufficient control over talking face, such as speaking style and emotional expression, resulting in uniform facial motion. In this paper, we focus on improving two key factors: lip-audio alignment control(LAC) and emotion control(EMC), to enhance the diversity and user-friendliness of talking videos. Lip-audio alignment control ensures accurate lip-sync across Recent advancements in audio-driven talking face generation have made great progress in lip synchronization. However, current methods often lack sufficient control over talking face, such as speaking style and emotional expression, resulting in uniform facial motion. In this paper, we focus on improving two key factors: lip-audio alignment control(LAC) and emotion control(EMC), to enhance the diversity and user-friendliness of talking videos. Lip-audio alignment control ensures accurate lip-sync across varied speaking styles to simulate different talking habits, whereas emotion control aims to generate realistic emotional expressions with varying intensities and mixed emotional states. To achieve precise facial animation control, we propose a novel and efficient framework, PC-Talk, which enables lip-audio alignment and emotion control through implicit keypoint deformations. First, our LAC module generates lip-synced talking faces with a specific speaking style, derived from either a video reference or preset options. It also supports lip movement scale adjustment and fine-grained editing of speaking styles for specific articulations. Second, our EMC module produces vivid emotional facial expressions through pure emotional deformation. It further enables precise control over emotion intensity and the compound emotions across different facial regions. Our method demonstrates outstanding control capabilities and achieves SOTA performance on HDTF and MEAD datasets in experiments. Project page: https://bq-wang0511.github.io/PC-Talk/