FrankenMotion:部位级人体动作生成与组合
FrankenMotion: Part-level Human Motion Generation and Composition
把生成过程按身体部位解耦、每个部位由独立时序文本提示驱动,这种「部位级条件注入 + 原子动作组合」的思路可迁移到细粒度可控的人体/手部/面部动作生成与编辑,也提示了用 LLM 自动构建细粒度时序标注的可行路径。
原文依据 we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion, where each body part is guided by its own temporally-structured textual prompt
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近年来,基于文本提示的人体动作生成取得了显著进展。然而,由于缺乏细粒度、部位级的动作标注,现有方法主要依赖序列级或动作级的描述,这限制了对单个身体部位的可控性。在本工作中,我们借助大语言模型(LLM)的推理能力,构建了一个高质量的、带有原子化且具备时序感知能力的部位级标注的动作数据集。与以往要么提供固定时间段内同步部位描述、要么仅依赖全局序列标签的数据集不同,我们的数据集在细粒度时间分辨率下刻画了异步且语义上彼此不同的部位运动。基于该数据集,我们提出了一个基于扩散模型的部位感知动作生成框架,即 FrankenMotion,其中每个身体部位都由其自身具有时序结构的文本提示引导。据我们所知,这是首个提供原子化、时序感知的部位级动作标注、并拥有一个同时支持空间(身体部位)与时序(原子动作)控制动作生成的模型的工作。实验表明,FrankenMotion 优于所有为适配我们的设定而调整并重新训练的先前基线模型,并且我们的模型能够组合训练中未出现过的动作。我们的代码与数据集将在论文发表后公开。
英文原文摘要 1399 字符
Human motion generation from text prompts has made remarkable progress in recent years. However, existing methods primarily rely on either sequence-level or action-level descriptions due to the absence of fine-grained, part-level motion annotations. This limits their controllability over individual body parts. In this work, we construct a high-quality motion dataset with atomic, temporally-aware part-level annotations, leveraging the reasoning capabilities of large language models (LLMs). Unlike prior datasets that either provide synchronized part captions with fixed time segments or rely solely on global sequence labels, our dataset captures asynchronous and semantically distinct part movements at fine temporal resolution. Based on this dataset, we introduce a diffusion-based part-aware motion generation framework, namely FrankenMotion, where each body part is guided by its own temporally-structured textual prompt. This is, to our knowledge, the first work to provide atomic, temporally-aware part-level motion annotations and have a model that allows motion generation with both spatial (body part) and temporal (atomic action) control. Experiments demonstrate that FrankenMotion outperforms all previous baseline models adapted and retrained for our setting, and our model can compose motions unseen during training. Our code and dataset will be publicly available upon publication.