MimicTalker：面向实时双人 3D 头部生成的多模态交互与记忆增强框架
MimicTalker: A Multimodal Interactive and Memory-Enhanced Framework for Real-Time Dyadic 3D Head Generation
面向双人对话场景的实时 3D 头部动作生成，其多模态交互上下文提取与语义引导的运动风格记忆机制，可迁移到虚拟人、数字人驱动以及语音驱动的面部动画等需要长时一致性的生成任务。
原文依据 Dyadic interactive head generation aims to synthesize realistic head motions that respond both verbally and non-verbally to an interlocutor in real-time conversation.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论双人交互式头部生成旨在实时对话中合成能够对对话者做出言语与非言语回应的逼真头部动作。现有工作往往聚焦于离线场景，对多模态对话上下文的理解较为浅层，同时缺乏长期连贯性。为解决这些局限，我们提出 MimicTalker，一种用于生成实时、上下文感知且长期一致的交互式头部动作的新方法。为此，我们提出多模态交互上下文提取（MICE）模块，从对话者处捕获即时与长期的多模态交互信息。为增强对对话的深入理解，我们提出语义增强动态交互（SDI）模块，整合通过基于大语言模型的分析器自动提取的对话意图与话题。此外，我们提出语义引导的运动风格记忆（MSM）机制，使整个对话过程中保持长期的动作一致性。我们在短对话片段（25 秒）与长对话（6 分钟）上开展实验，全面的实验表明我们的方法显著优于现有方法。
英文原文摘要 1295 字符
Dyadic interactive head generation aims to synthesize realistic head motions that respond both verbally and non-verbally to an interlocutor in real-time conversation. The existing works often focus on offline scenarios, and struggle with a shallow understanding of the multimodal conversational context while also lacking long-term coherence. To address these limitations, we propose MimicTalker, a novel method for producing real-time, contextually-aware, and long-term consistent interactive head motions. To this end, we propose a Multimodal Interactive Context Extraction (MICE) module to capture both instantaneous and long-term multimodal interactive information from the interlocutor. To enhance in-depth conversational understanding, we propose a Semantic-enhanced Dynamic Interaction (SDI) module to integrate the intentions and topics of the conversation, which are automatically extracted through an LLM-based analyzer. Further, we propose a semantic-guided Motion Style Memory (MSM) mechanism, enabling the long-term motion consistency throughout the conversation. We conduct experiments on both short conversational segments (25 seconds) and extended dialogues (6 minutes), and the comprehensive experiments demonstrate that our method significantly outperforms existing approaches.