DyaDiT：面向社交友好双人手势生成的多模态扩散 Transformer
DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation
把扩散 Transformer 用于双人对话场景的联合运动生成，通过融合双方音频与可选的社会语境 token、并允许以对话伙伴的手势作为条件，为多人交互式人体动作生成提供了可复用的条件融合与响应式建模思路。
原文依据 DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论生成逼真的对话手势对于实现与数字人之间自然、具有社交吸引力的交互至关重要。然而，现有方法通常将单一音频流映射到单个说话者的动作，既未考虑社交语境，也未对两人对话过程中的相互动态进行建模。本文提出 DyaDiT，一种多模态扩散 Transformer，可从双人音频信号生成符合语境的人体动作。DyaDiT 在 Seamless Interaction 数据集上训练，输入双人音频以及可选的社会语境 token，生成符合语境的动作。它融合来自双方说话者的信息以捕捉交互动态，使用运动字典编码运动先验，并可选地利用对话伙伴的手势来生成更具响应性的动作。我们在标准运动生成指标上评估 DyaDiT，并开展定量用户研究，结果表明它不仅在客观指标上超越现有方法，而且受到用户的强烈偏好，凸显了其鲁棒性与社交友好的动作生成能力。代码与模型将在论文被接收后公开。
英文原文摘要 1173 字符
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a single audio stream to a single speaker's motion, without considering social context or modeling the mutual dynamics between two people engaging in conversation. We present DyaDiT, a multi-modal diffusion transformer that generates contextually appropriate human motion from dyadic audio signals. Trained on Seamless Interaction Dataset, DyaDiT takes dyadic audio with optional social-context tokens to produce context-appropriate motion. It fuses information from both speakers to capture interaction dynamics, uses a motion dictionary to encode motion priors, and can optionally utilize the conversational partner's gestures to produce more responsive motion. We evaluate DyaDiT on standard motion generation metrics and conduct quantitative user studies, demonstrating that it not only surpasses existing methods on objective metrics but is also strongly preferred by users, highlighting its robustness and socially favorable motion generation. Code and models will be released upon acceptance.