共同交谈：从音频合成共处同一空间的 3D 对话
Talking Together: Synthesizing Co-Located 3D Conversations from Audio
从混合音频中同时生成两个共处同一空间的人物的完整 3D 面部动画，并显式建模两人之间的相对位置、朝向与相互注视；其双流架构、说话人角色嵌入与说话人之间的交叉注意力，以及视线损失，为多人语音驱动动画与共处场景合成提供了可复用的建模思路。
原文依据 we propose a dual-stream architecture where each stream is responsible for one participant's output
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文研究一项具有挑战性的任务：从混合音频流中为两位相互交流、共处同一空间的参与者生成完整的 3D 面部动画。现有方法往往生成如同视频会议通话中那样脱离身体的“会说话的头像”，而本文首次显式建模了动态的 3D 空间关系，包括相对位置、朝向与相互注视，这些对真实的面对面交谈至关重要。本系统合成两位个体的完整表演，包括精确的唇形同步，并独特地支持通过文本描述控制两人的相对头部姿态。为此，本文提出一种双流架构，其中每一路流负责一位参与者的输出。本文采用说话人角色嵌入，并设计说话人之间的交叉注意力机制，以解耦混合音频并建模交互。此外，本文引入一种新的视线损失，以促进自然的目光接触。为支撑这一数据需求较大的方法，本文提出一条新的数据处理流程，从真实场景视频中整理出一个包含超过 200 万对双人对话样本的大规模对话数据集。本方法生成的二元动画流畅、可控且具备空间感知能力，适用于 VR 与远程呈现等沉浸式应用，在感知真实感与交互连贯性上显著优于现有基线。
英文原文摘要 1387 字符
We tackle the challenging task of generating complete 3D facial animations for two interacting, co-located participants from a mixed audio stream. While existing methods often produce disembodied "talking heads" akin to a video conference call, our work is the first to explicitly model the dynamic 3D spatial relationship--including relative position, orientation, and mutual gaze--that is crucial for realistic in-person dialogues. Our system synthesizes the full performance of both individuals, including precise lip-sync, and uniquely allows their relative head poses to be controlled via textual descriptions. To achieve this, we propose a dual-stream architecture where each stream is responsible for one participant's output. We employ speaker's role embeddings and inter-speaker cross-attention mechanisms are designed to disentangle the mixed audio and model the interaction. Furthermore, we introduce a novel eye gaze loss to promote natural, mutual eye contact. To power our data-hungry approach, we introduce a novel pipeline to curate a large-scale conversational dataset consisting of over 2 million dyadic pairs from in-the-wild videos. Our method generates fluid, controllable, and spatially aware dyadic animations suitable for immersive applications in VR and telepresence, significantly outperforming existing baselines in perceived realism and interaction coherence.