面向组合式视频生成的免训练运动分解
Training-free Motion Factorization for Compositional Video Generation
把复杂运动拆成静止/刚体/非刚体三类，并用免训练的引导分支在扩散采样中分别约束外观稳定、刚体几何与局部形变，可直接迁移到任何视频扩散骨干上做可控运动合成。
原文依据 our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组合式视频生成旨在合成具有多样外观与运动的多个实例。然而，现有方法主要关注语义绑定，忽视了对提示词中指定的多样运动类别的理解。本文提出一个运动分解框架，将复杂运动分解为三类基本运动：静止、刚体运动与非刚体运动。具体而言，该框架遵循“先规划、后生成”的范式。(1) 在规划阶段，我们在运动图上推理运动规律，以获得每个实例形状与位置的逐帧变化。这通过将用户提示词组织为实例及其交互的结构化表示，缓解了其中的语义歧义。(2) 在生成阶段，我们以解耦的方式调制不同运动类别的合成。以运动线索为条件，各引导分支分别稳定静止区域的外观、保持刚体几何、并正则化局部非刚体形变。关键在于，我们的两个模块均与模型无关，可无缝集成到多种扩散模型架构中。大量实验表明，该框架在真实世界基准上的运动合成中取得了令人印象深刻的性能。代码见 https://github.com/ZixuanWang0525/MF-CVG。
英文原文摘要 1369 字符
Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.