利用合成运动数据学习生成高动态视频
Learning to Generate Highly Dynamic Videos using Synthetic Motion Data
用渲染光流作为与外观解耦的运动监督信号，训练视频扩散模型学习真实数据中稀缺的剧烈运动与精确相机运动控制，可直接迁移到任何需要细粒度运动条件的视频生成管线。
原文依据 rendered optical flow encodes only motion and is decoupled from appearance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管近期取得了进展，视频扩散模型在合成涉及高动态运动或需要细粒度运动可控性的真实感视频方面仍存在困难。一个核心限制在于常用训练数据集中此类样本的稀缺。为解决这一问题，我们提出 DynaVid，一个在训练中利用合成运动数据的视频合成框架，该运动数据以光流表示并通过计算机图形学管线渲染。这一方法具有两个关键优势。首先，合成运动提供了难以从真实数据中获得的多样运动模式与精确控制信号。其次，与具有人工外观的渲染视频不同，渲染光流仅编码运动并与外观解耦，从而避免模型复现合成视频的不自然观感。基于这一思路，DynaVid 采用两阶段生成框架：运动生成器首先合成运动，随后运动引导的视频生成器以该运动为条件生成视频帧。这种解耦形式使模型能够从合成数据中学习动态运动模式，同时保留真实世界视频的视觉真实感。我们在两个具有挑战性的场景上验证了该框架，即剧烈人体运动生成与极端相机运动控制，现有数据集在这些场景中尤为有限。大量实验表明，DynaVid 提升了动态运动生成与相机运动控制中的真实感与可控性。
英文原文摘要 1541 字符
Despite recent progress, video diffusion models still struggle to synthesize realistic videos involving highly dynamic motions or requiring fine-grained motion controllability. A central limitation lies in the scarcity of such examples in commonly used training datasets. To address this, we introduce DynaVid, a video synthesis framework that leverages synthetic motion data in training, which is represented as optical flow and rendered using computer graphics pipelines. This approach offers two key advantages. First, synthetic motion offers diverse motion patterns and precise control signals that are difficult to obtain from real data. Second, unlike rendered videos with artificial appearances, rendered optical flow encodes only motion and is decoupled from appearance, thereby preventing models from reproducing the unnatural look of synthetic videos. Building on this idea, DynaVid adopts a two-stage generation framework: a motion generator first synthesizes motion, and then a motion-guided video generator produces video frames conditioned on that motion. This decoupled formulation enables the model to learn dynamic motion patterns from synthetic data while preserving visual realism from real-world videos. We validate our framework on two challenging scenarios, vigorous human motion generation and extreme camera motion control, where existing datasets are particularly limited. Extensive experiments demonstrate that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.