MoCha：无需结构引导的端到端视频角色替换
MoCha: End-to-End Video Character Replacement without Structural Guidance
把视频扩散模型自身的时序跟踪能力当作隐式结构先验，从而摆脱逐帧掩码与骨架/深度等显式结构引导，只需单帧任意掩码即可完成角色替换；条件感知 RoPE 与 RL 后训练可迁移到其他多模态条件注入的身份保持生成任务。
原文依据 harnessing the inherent tracking ability of the video diffusion model, therefore requiring only a single arbitrary frame mask and no structural guidance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在给定用户提供身份的前提下进行可控视频角色替换仍是一个具有挑战性的问题，原因在于缺乏成对的视频数据。已有工作主要依赖基于重建的范式，该范式需要逐帧分割掩码以及显式的结构引导（例如骨架、深度）。然而，这种依赖严重限制了其在涉及遮挡、角色与物体交互、非常规姿态或复杂光照等复杂场景中的泛化能力，往往导致视觉伪影和时序不一致。本文提出 MoCha，一个新框架，通过利用视频扩散模型固有的跟踪能力来缓解上述限制，因而仅需单帧任意掩码且无需结构引导。为有效适配多模态输入条件并增强面部身份，我们引入条件感知 RoPE，并采用基于强化学习的后训练阶段。此外，为克服高质量成对训练数据的稀缺，我们提出一套完整的数据构建流程。具体而言，我们设计三个专用数据集：一个使用 Unreal Engine 5（UE5）构建的高保真渲染数据集、一个由当前人像动画技术合成的表情驱动数据集，以及一个由现有视频-掩码对衍生的增强数据集。大量实验表明，本方法显著优于现有最优方法。我们将公开代码与数据集以促进后续研究。
英文原文摘要 1536 字符
Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a new framework that mitigates these limitations by harnessing the inherent tracking ability of the video diffusion model, therefore requiring only a single arbitrary frame mask and no structural guidance. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code and dataset to facilitate further research.