CoT-Edit:让思维链引导指令式视频编辑
CoT-Edit: Let CoT Guide Instruction Video Editing
用 CoT 增强的 MLLM 作为规划器输出边界框序列与属性化编辑指令,再把全局文本检索转成框条件掩码生成,为指令式视频编辑提供可复用的「先规划空间先验、再局部精修」范式。
原文依据 a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论复杂场景下基于文本指令的视频编辑仍具挑战性:纯文本提示往往无法捕捉精确的空间关系与物理约束,导致目标歧义与物理上不合理的结果。为此,我们提出一种「规划—引导—编辑」框架,显式地连接语义意图与空间执行。在该框架中,一个思维链(CoT)增强的多模态大语言模型(MLLM)充当规划器,对视频与指令进行结构化推理,推导出精确的边界框序列与属性丰富的编辑指令。这些空间先验随后引导一个以框为条件的掩码生成器,将模糊的全局检索转化为局部化、上下文感知的精修,并生成能更准确捕捉物体尺度、接触关系与摆放位置的掩码。在这些空间与语义信号的基础上,一个基于扩散模型的编辑器整合掩码、丰富后的指令与帧特征,渲染出保持时间连贯且空间对齐良好的高保真编辑。我们的框架先以模块化方式训练、再进行联合训练,在降低数据需求的同时取得更优性能,能够在包含多个相似物体的场景中实现精确定位,并实现物理一致的物体添加;大量实验表明其在多个强基线方法之上达到最优性能。更多细节见:https://github.com/flying-sky999/CoT-Edit
英文原文摘要 1533 字符
Text-driven instruction-based video editing in complex scenes remains challenging: purely textual prompts often fail to capture precise spatial relationships and physical constraints, resulting in target ambiguity and physically implausible outcomes. To address this, we propose a plan-guide-edit framework that explicitly bridges semantic intent and spatial execution. In our framework, a Chain-of-Thought (CoT)-enhanced multimodal large language model (MLLM) serves as a planner, performing structured reasoning over the video and instructions to derive a precise sequence of bounding boxes and attribute-enriched editing directives. These spatial priors then guide a box-conditioned mask generator, transforming ambiguous global retrieval into localized, context-aware refinement and producing masks that more accurately capture object scale, contact relationships, and placement. Building on these spatial and semantic signals, a diffusion-based editor integrates the masks, enriched instructions, and frame features to render high-fidelity edits that remain temporally coherent and spatially well aligned. Trained first in a modular manner and then jointly, our framework achieves superior performance with reduced data requirements, delivering precise localization in scenes with multiple similar objects and physically consistent object additions, and extensive experiments demonstrate state-of-the-art performance over multiple strong baseline methods. More details are available at: https://github.com/flying-sky999/CoT-Edit