MoLingo：面向文本到人体动作生成的动作-语言对齐
MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation
提出用帧级文本标签训练语义对齐的运动编码器，使潜空间更利于扩散；并验证多token交叉注意力注入文本条件优于单token，可迁移到其他条件生成任务（如文本到图像/视频）的潜空间设计与条件注入。
原文依据 We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 MoLingo，一种文本到动作（T2M）模型，通过在连续潜空间中执行去噪来生成逼真、生动的人体动作。近期工作采用潜空间扩散，要么一次性对整个潜变量进行扩散，要么在多个潜变量上自回归地进行扩散。本文研究如何使连续动作潜变量上的扩散达到最佳效果。我们聚焦两个问题：（1）如何构建语义对齐的潜空间以使扩散更有效；（2）如何最佳地注入文本条件以使动作紧密跟随描述。我们提出一种语义对齐的动作编码器，使用帧级文本标签进行训练，使得具有相似文本含义的潜变量彼此靠近，从而使潜空间更利于扩散。我们还比较了单 token 条件与多 token 交叉注意力方案，发现交叉注意力能带来更好的动作真实感和文本-动作对齐。借助语义对齐的潜变量、自回归生成和交叉注意力文本条件，我们的模型在标准指标和用户研究中均取得了人体动作生成的最优性能。我们将发布代码和模型以促进进一步研究和下游应用。
英文原文摘要 1206 字符
We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state-of-the-art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.