MusicInfuser：让视频扩散模型听懂音乐并起舞
MusicInfuser: Making Video Diffusion Listen and Dance
提出基于引导式构造影响函数的逐层可适配性准则，可在单卡一天内把预训练文生视频扩散模型对齐到音乐条件，为视觉研究者提供一种低成本、保留先验的扩散模型新条件模态适配范式。
原文依据 a novel layer-wise adaptability criterion based on a guidance-inspired constructive influence function to select adaptable layers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 MusicInfuser，一种将预训练文本到视频扩散模型对齐以生成与指定音乐轨道同步的高质量舞蹈视频的方法。该方法无需从头训练多模态音视频或音频动作模型，而是展示了如何高效地适配现有视频扩散模型使其与音乐输入对齐。我们提出一种新颖的逐层可适配性准则，该准则基于受引导启发的构造性影响函数来选择可适配层，从而在显著降低训练成本的同时保留丰富的先验知识，即使在有限的专业数据集上亦然。实验表明，MusicInfuser 有效弥合了音乐与视频之间的鸿沟，能够生成随音乐动态响应的新颖且多样的舞蹈动作。此外，我们的框架对未见过的音乐轨道、更长的视频序列以及非常规主体均具有良好的泛化能力，在一致性和同步性上优于基线模型。所有这一切均无需动作数据，训练在单块 GPU 上一天内即可完成。
英文原文摘要 1096 字符
We introduce MusicInfuser, an approach that aligns pre-trained text-to-video diffusion models to generate high-quality dance videos synchronized with specified music tracks. Rather than training a multimodal audio-video or audio-motion model from scratch, our method demonstrates how existing video diffusion models can be efficiently adapted to align with musical inputs. We propose a novel layer-wise adaptability criterion based on a guidance-inspired constructive influence function to select adaptable layers, significantly reducing training costs while preserving rich prior knowledge, even with limited, specialized datasets. Experiments show that MusicInfuser effectively bridges the gap between music and video, generating novel and diverse dance movements that respond dynamically to music. Furthermore, our framework generalizes well to unseen music tracks, longer video sequences, and unconventional subjects, outperforming baseline models in consistency and synchronization. All of this is achieved without requiring motion data, with training completed on a single GPU within a day.