VIVA：基于 VLM 引导与奖励优化的指令驱动视频编辑
VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization
把 VLM 作为指令编码器，将文本指令、首帧和参考图编码成视觉接地的指令表示，再送入扩散 Transformer 主干；并用 Edit-GRPO 以相对奖励直接优化指令遵循、内容保持与美学质量，这套「VLM 编码 + 强化学习后训练」组合可迁移到其他条件生成任务。
原文依据 we propose a post-training stage, Edit-GRPO, which adapts Group Relative Policy Optimization to the domain of video editing
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论指令驱动的视频编辑旨在根据自然语言指令修改输入视频，同时保持内容保真度与时序连贯性。然而，现有的基于扩散的方法通常在简单编辑操作的配对数据上训练，这从根本上限制了它们泛化到多样且复杂的真实世界指令的能力。为弥补这一泛化差距，我们提出 VIVA，一个可扩展的指令驱动视频编辑框架，利用 VLM 引导的编码与奖励优化。首先，我们引入一个基于 VLM 的指令器，将文本指令、源视频首帧以及可选的参考图像编码为视觉接地的指令表示，为扩散 Transformer 主干提供细粒度的空间与语义上下文。其次，我们提出一个后训练阶段 Edit-GRPO，将组相对策略优化适配到视频编辑领域，利用相对奖励直接优化模型，使其产生指令忠实、内容保持且视觉美观的编辑结果。此外，我们提出一个数据构建流程，用于合成生成基础编辑操作的多样、高保真配对视频-指令数据。大量实验表明，VIVA 在指令遵循、泛化与编辑质量上均优于最优方法。
英文原文摘要 1420 字符
Instruction-based video editing aims to modify an input video according to a natural-language instruction while preserving content fidelity and temporal coherence. However, existing diffusion-based approaches are often trained on paired data of simple editing operations, which fundamentally limits their ability to generalize to diverse and complex, real-world instructions. To address this generalization gap, we propose VIVA, a scalable framework for instruction-based video editing that leverages VLM-guided encoding and reward optimization. First, we introduce a VLM-based instructor that encodes the textual instruction, the first frame of the source video, and an optional reference image into visually-grounded instruction representations, providing fine-grained spatial and semantic context for the diffusion transformer backbone. Second, we propose a post-training stage, Edit-GRPO, which adapts Group Relative Policy Optimization to the domain of video editing, directly optimizing the model for instruction-faithful, content-preserving, and aesthetically pleasing edits using relative rewards. Furthermore, we propose a data construction pipeline designed to synthetically generate diverse, high-fidelity paired video-instruction data of basic editing operations. Extensive experiments show that VIVA achieves superior instruction following, generalization, and editing quality over state-of-the-art methods.