SyncDreamer：超越说话头部的可控且富有表现力的虚拟人生成
SyncDreamer: Controllable and Expressive Avatar Generation Beyond the Talking Head
用 Diffusion Transformer 直接从单图+语音+文本提示词生成全身说话数字人，其视觉适配器与注意力定位损失可复用于任何需要身份保持的条件生成任务；音频动力学编码器把节奏/能量等连续语音特征注入扩散过程，为语音驱动的细粒度运动控制提供了可借鉴的条件注入方式。
原文依据 a unified Diffusion Transformer framework that generates identity-preserving and emotionally expressive talking avatars from only a single image, speech audio, and text prompt
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论生成逼真且富有表现力的音频驱动说话虚拟人仍是数字人合成中的核心挑战。现有方法往往依赖姿态估计等中间表示来获得自然身体运动，这限制了灵活性并引入视觉失真。此外，多数音频驱动方法依赖离散情绪分类器或文本标签来调控面部表情，将复杂的情感动态简化为高兴、悲伤、愤怒等粗粒度类别。这种类别式监督无法捕捉连续且细粒度的语音动态（节奏、能量、强度），导致同步性有限、运动在情感上流于表面。为克服这些局限，我们提出 SyncDreamer，一个统一的扩散 Transformer 框架，仅凭单张图像、语音音频和文本提示词即可生成保持身份且情感表现力强的说话虚拟人。我们提出带有注意力定位损失的视觉适配器以维持身份保真度，进一步引入音频动力学编码器以实现节奏与情绪感知的运动，并采用基于强化学习的跨模态提示词增强器，将文本线索视觉定位到视觉上下文中以实现细粒度运动控制。在肖像与全身基准上的大量实验表明，该方法在真实感、同步精度和语义可控性上达到最优性能，为交互式与创意应用中的富有表现力的数字虚拟人奠定了可扩展的基础。
英文原文摘要 1492 字符
Generating realistic and expressive audio-driven talking avatars remains a central challenge in digital human synthesis. Existing methods often depend on intermediate representations such as pose estimations for natural body motion, which restricts flexibility and adds visual distortions. Moreover, most audio-driven approaches rely on discrete emotion classifiers or text labels to regulate facial expression, reducing complex affective dynamics to coarse categories such as happy, sad, or angry. Such categorical supervision fails to capture the continuous and fine-grained speech dynamics (rhythm, energy, intensity) resulting in limited synchronization and emotionally shallow motion. To overcome these limitations, we present SyncDreamer, a unified Diffusion Transformer framework that generates identity-preserving and emotionally expressive talking avatars from only a single image, speech audio, and text prompt.We propose a visual adapter with Attention Localization Loss to maintain identity fidelity, further incorporating an audio dynamics encoder for rhythm- and emotion-aware motion, and an RL-based Cross-Modal Prompt Enhancer grounding textual cues in visual context for fine-grained motion control. Extensive experiments on portrait and full-body benchmarks demonstrate state-of-the-art performance in realism, synchronization accuracy, and semantic controllability, establishing a scalable foundation for expressive digital avatars in interactive and creative applications.