RFDM：用于视频编辑的残差流扩散模型
RFDM: Residual Flow Diffusion Models for Video Editing
把 2D 图生图扩散模型改造成因果逐帧的视频编辑器，用 t-1 步预测作为条件、并让前向过程预测相邻帧残差，从而以图像级算力处理任意长度视频；这套「残差流 + 自回归条件」思路可直接迁移到其他需要变长、低算力的视频生成/编辑任务。
原文依据 we propose a new I2I diffusion forward process formulation that encourages the model to predict the residual between the target output and the previous prediction
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论指令式视频编辑仅使用文本提示对输入视频施加编辑，从而实现直观的自然语言控制。尽管进展迅速，多数方法仍要求固定长度输入且计算开销巨大。与此同时，自回归视频生成能够高效地合成变长视频，但在视频编辑中仍未得到充分探索。我们提出一种因果、高效的视频编辑模型，可逐帧编辑变长视频。为提升效率，我们从二维图像到图像（I2I）扩散模型出发，通过将第 t 步的编辑条件设为模型在第 t-1 步的预测，将其适配为视频到视频（V2V）编辑。为利用视频的时序冗余，我们提出一种新的 I2I 扩散前向过程形式，促使模型预测目标输出与前一预测之间的残差。我们将该模型称为残差流扩散模型（RFDM），它使去噪过程聚焦于相邻帧之间的变化。此外，我们提出一个新的基准，能够按视频编辑任务的忠实度更好地对最优方法进行排序。在用于全局/局部风格迁移与物体移除的成对视频数据上训练后，RFDM 超越了基于 I2I 的方法，并与完全时空（3D）V2V 模型相当，同时其计算量与图像模型持平，且可随输入视频长度独立扩展。更多内容见 https://smsd75.github.io/RFDM_page/ 项目主页。
英文原文摘要 1516 字符
Instructional video editing applies edits to an input video using only text prompts, enabling intuitive natural-language control. Despite the rapid progress, most methods still require fixed-length inputs and substantial compute. Meanwhile, autoregressive video generation enables efficient variable-length synthesis, yet remains under-explored for video editing. We introduce a causal, efficient video editing model that edits variable-length videos frame by frame. For efficiency, we start from a 2D image-to-image (I2I) diffusion model and adapt it to video-to-video (V2V) editing by conditioning the edit at time step t on the model's prediction at t-1. To leverage videos' temporal redundancy, we propose a new I2I diffusion forward process formulation that encourages the model to predict the residual between the target output and the previous prediction. We call this \underline R esidual \underline F low \underline D iffusion \underline M odel (\methodname), which focuses the denoising process on changes between consecutive frames. Moreover, we propose a new benchmark that better ranks state-of-the-art methods by faithfulness for video editing tasks. Trained on paired video data for global/local style transfer and object removal, \methodname surpasses I2I-based methods and competes with fully spatiotemporal (3D) V2V models, while matching the compute of image models and scaling independently of input video length. More content can be found in \href https://smsd75.github.io/RFDM_page/ RFDM page .