通过自适应低通引导改善图像到视频模型中的运动
Improving Motion in Image-to-Video Models via Adaptive Low-Pass Guidance
对做图像到视频生成的人,这是一个无需训练、只改采样过程的插件:在去噪早期对条件图像施加自适应低通滤波,抑制高频细节导致的静态捷径,可直接套用到任意 I2V 模型的采样循环中以提升运动幅度。
原文依据 we propose adaptive low-pass guidance (ALG), a simple training-free fix to the I2V model sampling procedure
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的文本到视频(T2V)模型在生成高质量、富有动态感的视频方面展现出强大能力。为提升视觉可控性,近期工作考虑对预训练 T2V 模型进行微调以支持图像到视频(I2V)生成。然而,这种适配常常抑制生成输出的运动动态性,导致相比对应的 T2V 模型产生更静态的视频。在本工作中,我们分析了这一现象,并指出其根源在于输入图像中的高频细节被过早暴露,从而使采样过程偏向一条过拟合于参考图像静态外观的捷径轨迹。为解决该问题,我们提出自适应低通引导(ALG),一种对 I2V 模型采样过程的简单、免训练修正,可在不损害逐帧图像质量的前提下生成更具动态感的视频。具体而言,ALG 通过在去噪早期阶段施加低通滤波,自适应地调制条件图像的频率成分。大量实验表明,ALG 显著提升了生成视频的时序动态性,同时保持甚至改善了图像保真度与文本对齐。例如,在 VBench 测试套件上,ALG 在保持原始视频质量的同时,使各模型的动态程度平均提升 33%。更多可视化结果与源代码见项目主页。
英文原文摘要 1428 字符
Recent text-to-video (T2V) models have demonstrated strong capabilities in producing high-quality, dynamic videos. To improve the visual controllability, recent works have considered fine-tuning pre-trained T2V models to support image-to-video (I2V) generation. However, such adaptation frequently suppresses motion dynamics of generated outputs, resulting in more static videos compared to their T2V counterparts. In this work, we analyze this phenomenon and identify that it stems from the premature exposure to high-frequency details in the input image, which biases the sampling process toward a shortcut trajectory that overfits to the static appearance of the reference image. To address this, we propose adaptive low-pass guidance (ALG), a simple training-free fix to the I2V model sampling procedure to generate more dynamic videos without compromising per-frame image quality. Specifically, ALG adaptively modulates the frequency content of the conditioning image by applying a low-pass filter at the early stage of denoising. Extensive experiments show ALG significantly improves the temporal dynamics of generated videos, while preserving or even improving image fidelity and text alignment. For instance, on the VBench test suite, ALG achieves a 33% average improvement across models in dynamic degree while maintaining the original video quality. For additional visualizations and source code, see the project page.