PolySLGen:多人交互中的在线多模态说-听反应生成
PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction
面向具身智能体的多人交互场景,提出姿态融合模块与社交线索编码器,将群体运动与社交信号聚合后生成目标参与者的语音、身体动作与说话状态;其多人运动聚合与语音-动作对齐的建模思路可迁移到人体动作预测、社交姿态生成与说话人状态估计等视觉任务。
原文依据 we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论类人的多模态反应生成对于人类与智能具身智能体之间自然的群体交互至关重要。然而,现有方法往往局限于双人交互中的单模态或仅说话方反应,难以适用于真实的社交场景。许多方法还忽视了非言语线索以及多人交互的复杂动态,而这两者对于参与度和对话连贯性都至关重要。在本工作中,我们提出 PolySLGen,一个用于多人多模态说与听反应生成的在线框架。给定所有参与者过去的对话与动作,PolySLGen 为目标参与者生成未来的说话或倾听反应,包括语音、身体动作和说话状态分数。为有效建模群体交互,我们提出姿态融合模块和社交线索编码器,联合聚合来自群体的动作与社交信号。大量实验以及定量与定性评估表明,PolySLGen 能够生成上下文适当且时间连贯的多模态反应,在动作质量、动作-语音对齐、说话状态预测和人类感知真实感方面优于若干适配的基线方法与最优基线方法。
英文原文摘要 1269 字符
Human-like multimodal reaction generation is essential for natural group interactions between humans and intelligent embodied AI. However, existing approaches are often limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and the complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen, an online framework for Polyadic multimodal Speaking and Listening reaction Generation. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score.To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multimodal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism.