Soul：为数字人注入生命，实现高保真长期多模态动画
Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
提出基于Wan2.2-5B骨干的多模态驱动数字人动画框架，通过音频注入层和阈值感知码本替换实现长期一致性，并利用步数/CFG蒸馏与轻量VAE实现11.4倍推理加速，为视频生成中的长时一致性和效率优化提供可复用方案。
原文依据 integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出了一个多模态驱动的框架，用于高保真长期数字人动画，称为Soul，它从单帧肖像图像、文本提示和音频生成语义连贯的视频，实现精确的唇形同步、生动的面部表情和稳健的身份保持。我们构建了Soul-1M，包含100万个精细标注的样本，并配有精确的自动标注流程（涵盖肖像、上半身、全身和多人场景），以缓解数据稀缺问题，同时我们精心策划了Soul-Bench，用于全面公平地评估音频/文本引导的动画方法。该模型基于Wan2.2-5B骨干构建，集成了音频注入层和多种训练策略，并结合阈值感知的码本替换以确保长期生成的一致性。同时，采用步数/CFG蒸馏和轻量级变分自编码器来优化推理效率，在几乎不损失质量的情况下实现了11.4倍加速。大量实验表明，Soul在视频质量、视频-文本对齐、身份保持和唇形同步准确性方面显著优于当前领先的开源和商业模型，展示了在虚拟主播和电影制作等现实场景中的广泛适用性。
英文原文摘要 1308 字符
We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed Soul, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4xspeedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production.