PerformRecast：面向人像视频编辑的表情与头部姿态解耦
PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing
用 3DMM 的分离参数（身份/表情/头姿）改造关键点变换公式，实现表情与头姿的解耦控制；并把人脸/非人脸区域解耦、用教师模型分别监督以消除边界错位，这两点可直接迁移到任何人脸驱动编辑或重演管线。
原文依据 we improve the keypoints transformation formula in previous methods to make it more consistent with 3DMM model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文主要研究基于驱动视频的、仅编辑表情的人像视频表演编辑任务，该任务在动画与电影工业中至关重要。现有研究大多集中于人像动画，即根据驱动视频中的面部运动来驱动一张静态人像图像。因此，这些方法难以将面部表情与头部姿态旋转解耦，从而缺乏独立编辑面部表情的能力。本文提出 PerformRecast，一种多功能的仅表情视频编辑方法，致力于重塑现有电影与动画中的表演。该方法的关键洞见来自三维可变形人脸模型（3DMM）的特性：它以相互独立的参数对三维人脸网格的身份、面部表情与头部姿态进行建模。因此，我们改进了先前方法中的关键点变换公式，使其与 3DMM 模型更加一致，从而实现更好的解耦，并为用户提供更细粒度的控制。此外，为避免生成结果中人脸边界附近的错位，我们将输入人像图像的面部区域与非面部区域解耦，并预训练一个教师模型为其分别提供监督。大量实验表明，本方法能够生成更忠实于驱动视频的高质量结果，在可控性与效率上均优于现有方法。代码、数据与训练模型见 https://youku-aigc.github.io/PerformRecast。
英文原文摘要 1628 字符
This paper primarily investigates the task of expression-only portrait video performance editing based on a driving video, which plays a crucial role in animation and film industries. Most existing research mainly focuses on portrait animation, which aims to animate a static portrait image according to the facial motion from the driving video. As a consequence, it remains challenging for them to disentangle the facial expression from head pose rotation and thus lack the ability to edit facial expression independently. In this paper, we propose PerformRecast, a versatile expression-only video editing method which is dedicated to recast the performance in existing film and animation. The key insight of our method comes from the characteristics of 3D Morphable Face Model (3DMM), which models the face identity, facial expression and head pose of 3D face mesh with separate parameters. Therefore, we improve the keypoints transformation formula in previous methods to make it more consistent with 3DMM model, which achieves a better disentanglement and provides users with much more fine-grained control. Furthermore, to avoid the misalignment around the boundary of face in generated results, we decouple the facial and non-facial regions of input portrait images and pre-train a teacher model to provide separate supervision for them. Extensive experiments show that our method produces high-quality results which are more faithful to the driving video, outperforming existing methods in both controllability and efficiency. Our code, data and trained models are available at https://youku-aigc.github.io/PerformRecast.