图像到视频模型是优秀的零样本图像编辑器吗?
Are Image-to-Video Models Good Zero-Shot Image Editors?
将预训练的图像到视频扩散模型直接复用为零样本图像编辑器,通过思维链提示词增强、时序潜变量丢弃和自洽后细化三个无需训练的技巧,解决提示词错位、时序潜变量冗余和后期帧模糊问题,为指令驱动的图像编辑提供了新范式。
原文依据 IF-Edit (Image Edit by Generating Frames), a tuning-free framework that repurposes pre-trained image-to-video diffusion models for instruction-driven image editing.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模视频扩散模型展现出强大的世界模拟与时序推理能力,但其作为零样本图像编辑器的潜力仍未得到充分探索。我们提出 IF-Edit(通过生成帧进行图像编辑),一个无需调优的框架,将预训练的图像到视频扩散模型重新用于指令驱动的图像编辑。IF-Edit 通过以下方式解决三个核心障碍——提示词错位、冗余的时序潜变量和模糊的后期帧:(1)思维链提示词增强模块,将静态编辑指令重新表述为基于时序的推理提示词;(2)时序潜变量丢弃策略,在专家切换点之后压缩帧潜变量,在加速去噪的同时保留全局语义和时序一致性;(3)自洽后细化步骤,通过短暂的静止-视频轨迹对最清晰的后期帧进行细化,利用视频先验获得更清晰、更忠实的结果。在四个公开基准(涵盖非刚性变形、物理与时序推理以及通用指令编辑)上的大量实验表明,IF-Edit 在非刚性和以推理为中心的任务上取得了强劲性能,同时在通用编辑上保持竞争力。我们的研究提供了视频扩散模型作为图像编辑器的系统视角,揭示了其独特优势、局限性以及一个用于统一视频-图像生成推理的简单方案。
英文原文摘要 1498 字符
Large-scale video diffusion models exhibit strong world-simulation and temporal reasoning capabilities, yet their potential as zero-shot image editors remains underexplored. We present \ifedit IF-Edit (Image Edit by Generating Frames), a tuning-free framework that repurposes pre-trained image-to-video diffusion models for instruction-driven image editing. \ifedit IF-Edit addresses three core obstacles--prompt misalignment, redundant temporal latents, and blurry late-stage frames--via: (1) a Chain-of-Thought Prompt Enhancement module that reformulates static editing instructions into temporally grounded reasoning prompts; (2) a Temporal Latent Dropout strategy that compresses frame latents after the expert-switch point, accelerating denoising while preserving global semantics and temporal coherence; and (3) a Self-Consistent Post-Refinement step that refines the sharpest late-stage frame through a brief still-video trajectory, leveraging the video prior for sharper and more faithful results. Extensive experiments across four public benchmarks--covering non-rigid deformations, physical and temporal reasoning, and general instruction editing--show that \ifedit IF-Edit achieves strong performance on non-rigid and reasoning-centric tasks while remaining competitive on general-purpose edits. Our study offers a systematic view of video diffusion models as image editors, revealing their unique strengths, limitations, and a simple recipe for unified video-image generative reasoning.