MIBURI：迈向富有表现力的交互式手势合成
MIBURI: Towards Expressive Interactive Gesture Synthesis
把全身手势与面部表情编码成多层级离散 token，再用二维因果自回归框架在 LLM 语音-文本嵌入条件下实时生成，为视觉侧的人体动作合成提供了可复用的「部位感知运动编解码 + 因果自回归」范式，适合做实时虚拟人/对话智能体的动作驱动。
原文依据 the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论具身对话智能体（ECA）旨在通过语音、手势和面部表情来模拟人类面对面的交互。当前基于大语言模型（LLM）的对话智能体缺乏具身性，也缺乏自然交互所必需的表现力手势。现有的 ECA 方案往往产生僵硬、多样性低的动作，不适合类人交互。另一方面，用于协同语音手势合成的生成方法能产生自然的身体手势，但依赖未来语音上下文且运行时间长。为弥合这一差距，我们提出 MIBURI，这是首个与实时口语对话同步生成富有表现力的全身手势和面部表情的在线因果框架。我们采用部位感知的手势编解码器，将分层运动细节编码为多层级离散 token。这些 token 随后由一个二维因果框架在基于 LLM 的语音-文本嵌入条件下自回归生成，实时建模时序动态与部位级运动层级。此外，我们引入辅助目标以鼓励富有表现力且多样的手势，同时防止收敛到静态姿态。对比评估表明，与近期基线相比，我们的因果实时方法能产生自然且与上下文对齐的手势。我们建议读者访问 https://vcai.mpi-inf.mpg.de/projects/MIBURI/ 观看演示视频。
英文原文摘要 1445 字符
Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on https://vcai.mpi-inf.mpg.de/projects/MIBURI/