UniLS：面向统一听与说的端到端音频驱动虚拟人
UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking
两阶段训练范式（先无音频自回归学习内在运动先验，再用双轨音频微调调制）为音频驱动的人脸/头部生成提供了可复用思路，尤其适合解决弱音频驱动信号下生成僵硬的问题；双轨音频端到端设定也可直接用于交互式数字人。
原文依据 Stage 1 first learns the internal motion prior by training an audio-free autoregressive generator
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论生成逼真的对话虚拟人，不仅需要建模孤立的说话者，还需要建模说话与倾听之间动态、相互的交互。然而，对倾听者的建模极具挑战：直接的音频驱动训练会失败，产生僵硬、静止的倾听动作。这一失败源于一种根本性的不平衡：说话者的动作受语音音频强烈驱动，而倾听者的动作主要遵循内在的运动先验，仅受外部语音的松散引导。这一挑战导致大多数方法只关注仅说话生成。此前唯一一次联合生成的尝试依赖额外的说话者动作来生成倾听者。这种设计并非端到端，因而阻碍了实时适用性。为解决这一局限，我们提出 UniLS，这是首个仅由双轨音频驱动、用于生成统一说-听表情的端到端框架。我们的方法引入了一种新颖的两阶段训练范式。第一阶段先通过训练一个无音频的自回归生成器来学习内在运动先验，捕捉自然面部运动的自发动态。第二阶段再引入双轨音频，微调生成器以根据外部语音线索调制已学到的运动先验。大量评估表明，UniLS 达到了最优的说话准确度。更重要的是，它在倾听指标上带来高达 44.1% 的提升，生成显著更多样、更自然的倾听表情。这有效缓解了僵硬问题，并为交互式数字人提供了一种实用、高保真的音频驱动方案。
英文原文摘要 1651 字符
Generating lifelike conversational avatars requires modeling not just isolated speakers, but the dynamic, reciprocal interaction of speaking and listening.However, modeling the listener is exceptionally challenging: direct audio-driven training fails, producing stiff, static listening motions. This failure stems from a fundamental imbalance: the speaker's motion is strongly driven by speech audio, while the listener's motion primarily follows an internal motion prior and is only loosely guided by external speech. This challenge has led most methods to focus on speak-only generation. The only prior attempt at joint generation relies on extra speaker's motion to produce the listener.This design is not end-to-end, thereby hindering the real-time applicability.To address this limitation, we present UniLS, the first end-to-end framework for generating unified speak-listen expressions, driven by only dual-track audio.Our method introduces a novel two-stage training paradigm.Stage 1 first learns the internal motion prior by training an audio-free autoregressive generator, capturing the spontaneous dynamics of natural facial motion. Stage 2 then introduces the dual-track audio, fine-tuning the generator to modulate the learned motion prior based on external speech cues.Extensive evaluations show UniLS achieves state-of-the-art speaking accuracy. More importantly, it delivers up to 44.1% improvement in listening metrics, generating significantly more diverse and natural listening expressions. This effectively mitigates the stiffness problem and provides a practical, high-fidelity audio-driven solution for interactive digital humans.