CoordSpeaker：利用手势描述实现协调描述赋能的多语音手势生成
CoordSpeaker: Exploiting Gesture Captioning for Coordinated Caption-Empowered Co-Speech Gesture Generation
提出基于运动-语言模型的手势描述生成框架，为扩散模型提供细粒度语义条件；其统一跨数据集运动表示与分层控制去噪器可复用于其他条件运动生成任务。
原文依据 we propose a conditional latent diffusion model with unified cross-dataset motion representation and a hierarchically controlled denoiser
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多语音手势生成显著推动了人机交互的发展，然而由于忽略了文本驱动的非自发性手势（例如说话时鞠躬），说话者的动作仍受到限制。现有方法面临两个关键挑战：1）手势数据集中缺乏描述性文本标注所导致的语义先验鸿沟；2）难以实现对手势生成的协调多模态控制。为解决这些挑战，本文提出CoordSpeaker，一个全面的框架，能够实现协调的描述赋能多语音手势合成。我们的方法首先通过一种新颖的手势描述框架弥合语义先验鸿沟，该框架利用运动-语言模型生成多粒度的描述性文本。在此基础上，我们提出一种条件潜变量扩散模型，结合统一的跨数据集运动表示和分层控制的去噪器，以实现高度受控、协调的手势生成。CoordSpeaker率先探索手势理解以解决手势生成中的语义鸿沟，同时提供了双向手势-文本映射的新视角。大量实验表明，我们的方法生成高质量手势，既与语音节奏同步，又与任意描述语义连贯，相比现有方法以更高效率实现了更优性能。
英文原文摘要 1481 字符
Co-speech gesture generation has significantly advanced human-computer interaction, yet speaker movements remain constrained due to the omission of text-driven non-spontaneous gestures (e.g., bowing while talking). Existing methods face two key challenges: 1) the semantic prior gap due to the lack of descriptive text annotations in gesture datasets, and 2) the difficulty in achieving coordinated multimodal control over gesture generation. To address these challenges, this paper introduces CoordSpeaker, a comprehensive framework that enables coordinated caption-empowered co-speech gesture synthesis. Our approach first bridges the semantic prior gap through a novel gesture captioning framework, leveraging a motion-language model to generate descriptive captions at multiple granularities. Building upon this, we propose a conditional latent diffusion model with unified cross-dataset motion representation and a hierarchically controlled denoiser to achieve highly controlled, coordinated gesture generation. CoordSpeaker pioneers the first exploration of gesture understanding to tackle the semantic gap in gesture generation while offering a novel perspective of bidirectional gesture-text mapping. Extensive experiments demonstrate that our method produces high-quality gestures that are both rhythmically synchronized with speeches and semantically coherent with arbitrary captions, achieving superior performance with higher efficiency compared to existing approaches.