U-Mind：面向实时多模态交互与视听生成的统一框架
U-Mind: A Unified Framework for Real-Time Multimodal Interaction with Audiovisual Generation
论文提出以文本优先的思维链规划驱动语音、动作和视频的同步生成，并实现基于姿态与语音条件的实时视频渲染，为视觉研究者提供了将语言推理与视频生成闭环结合的可复用框架。
原文依据 a text-first decoding pipeline that performs internal chain-of-thought planning followed by temporally synchronized generation across modalities
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论全栈实时多模态交互是构建能够进行自然、动态交流的智能具身智能体的核心目标。然而，现有系统要么局限于单模态生成，要么存在推理能力下降和跨模态对齐不佳的问题，阻碍了连贯且具有感知基础的交互。在本工作中，我们提出 U-Mind，这是首个支持实时生成并在单一交互循环中联合建模语言、语音、动作和视频合成的高智能多模态对话统一系统。U-Mind 的核心是实现了一个统一对齐与推理框架，该框架解决两个关键挑战：通过分段对齐策略增强跨模态同步，以及通过演练驱动学习保持推理能力。在推理阶段，U-Mind 采用文本优先的解码流水线，先进行内部思维链规划，随后跨模态进行时间同步生成。为闭合循环，我们实现了一个以姿态和语音为条件的实时视频渲染框架，从而提供富有表现力且同步的视觉反馈。大量实验表明，U-Mind 在一系列多模态交互任务上取得了最优性能，包括问答、指令跟随和动作生成，为迈向智能、沉浸式对话智能体铺平了道路。
英文原文摘要 1440 字符
Full-stack multimodal interaction in real-time is a central goal in building intelligent embodied agents capable of natural, dynamic communication. However, existing systems are either limited to unimodal generation or suffer from degraded reasoning and poor cross-modal alignment, preventing coherent and perceptually grounded interactions. In this work, we introduce U-Mind, the first unified system for high-intelligence multimodal dialogue that supports real-time generation and jointly models language, speech, motion, and video synthesis within a single interactive loop. At its core, U-Mind implements a Unified Alignment and Reasoning Framework that addresses two key challenges: enhancing cross-modal synchronization via a segment-wise alignment strategy, and preserving reasoning abilities through Rehearsal-Driven Learning. During inference, U-Mind adopts a text-first decoding pipeline that performs internal chain-of-thought planning followed by temporally synchronized generation across modalities. To close the loop, we implement a real-time video rendering framework conditioned on pose and speech, enabling expressive and synchronized visual feedback. Extensive experiments demonstrate that U-Mind achieves state-of-the-art performance on a range of multimodal interaction tasks, including question answering, instruction following, and motion generation, paving the way toward intelligent, immersive conversational agents.