ViBES：具备行为智能 3D 虚拟身体的对话智能体
ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body
ViBES 将语音、文本和身体动作统一为交错的多模态词元流，并用按模态硬路由的混合专家 Transformer 联合建模，为视觉研究者提供了构建可控 3D 虚拟人对话系统的可复用架构与多轮对话-动作对齐评测思路。
原文依据 ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论人类交流本质上是多模态且社会性的：词语、韵律和肢体语言共同承载意图。然而，以往大多数系统将人类行为建模为翻译任务——如语音驱动手势或文本驱动动作，将固定话语映射到动作片段——而不要求智能体就何时移动、做什么或如何在多轮对话中适应做出决策。这导致时序脆弱、社会基础薄弱，以及语音、文本和动作在训练或推理时相互隔离的碎片化堆栈。我们提出 ViBES（行为表达与同步中的语音），一个对话式 3D 智能体，它联合规划语言和运动，并执行对话条件下的身体动作。具体而言，ViBES 是一个语音-语言-行为（SLB）模型，采用模态专家混合（MoME）主干：按模态划分的 Transformer 专家分别处理语音、面部表情和身体运动。该模型处理交错的多模态词元流，按模态进行硬路由（参数按专家拆分），同时通过跨专家注意力共享信息。通过利用强大的预训练语音-语言模型，该智能体支持混合主动交互：用户可以在对话中说话、打字或发出身体动作指令，系统暴露可控的行为钩子以支持流式响应。我们进一步在多轮对话上建立基准，使用对话-动作对齐和行为质量的自动指标，并观察到相较于强语音驱动手势和文本驱动动作基线的持续提升。ViBES 超越了“语音条件动作生成”，迈向语言、韵律和运动联合生成的智能体化虚拟身体，实现可控、具备社会能力的 3D 交互。
英文原文摘要 1748 字符
Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task--co-speech gesture or text-to-motion that maps a fixed utterance to motion clips--without requiring agentic decision-making about when to move, what to do, or how to adapt across multi-turn dialogue. This leads to brittle timing, weak social grounding, and fragmented stacks where speech, text, and motion are trained or inferred in isolation. We introduce ViBES (Voice in Behavioral Expression and Synchrony), a conversational 3D agent that jointly plans language and movement and executes dialogue-conditioned body actions. Concretely, ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone: modality-partitioned transformer experts for speech, facial expression, and body motion. The model processes interleaved multimodal token streams with hard routing by modality (parameters are split per expert), while sharing information through cross-expert attention. By leveraging strong pretrained speech-language models, the agent supports mixed-initiative interaction: users can speak, type, or issue body-action directives mid-conversation, and the system exposes controllable behavior hooks for streaming responses. We further benchmark on multi-turn conversation with automatic metrics of dialogue-motion alignment and behavior quality, and observe consistent gains over strong co-speech and text-to-motion baselines. ViBES goes beyond "speech-conditioned motion generation" toward agentic virtual bodies where language, prosody, and movement are jointly generated, enabling controllable, socially competent 3D interaction.