EasyOmnimatte：驯服预训练图像修复扩散模型实现端到端视频分层分解
EasyOmnimatte: Taming Pretrained Inpainting Diffusion Models for End-to-End Video Layered Decompositio
将预训练视频修复扩散模型通过LoRA微调为双专家（效果专家与质量专家），在采样不同阶段切换，实现端到端视频分层分解，为视频编辑中的前景与效果分离提供高效方案。
原文依据 we finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的视频全遮罩方法通常依赖缓慢、多阶段或推理时优化的流程，无法充分利用强大的生成先验，导致分解效果欠佳。我们的关键洞察是：如果视频修复模型能够被微调以去除与前景相关的效果，那么它必然具备感知这些效果的内在能力，因此也可以被微调用于互补任务：即带有相关效果的前景层分解。然而，尽管简单地对所有模块应用低秩适配来微调修复模型可以产生高质量的alpha遮罩，却无法捕捉相关效果。我们的系统分析表明，这是因为效果相关的线索主要编码在特定的DiT模块中，而当低秩适配应用于所有模块时会被抑制。为了解决这个问题，我们提出了EasyOmnimatte，首个统一的端到端视频全遮罩方法。具体而言，我们微调预训练的视频修复扩散模型以学习双互补专家，同时保持其原始权重不变：一个效果专家，仅对效果敏感的DiT模块应用低秩适配，以捕捉前景及相关效果的粗结构；另一个完全低秩适配微调的质量专家，学习细化alpha遮罩。在采样过程中，效果专家用于早期高噪声步骤的去噪，而质量专家在后期低噪声步骤接管。这种设计消除了两次完整扩散过程的需要，显著降低了计算成本，同时不损害输出质量。消融研究验证了双专家策略的有效性。实验表明，EasyOmnimatte在视频全遮罩任务上达到了新的最优性能，并支持多种下游任务，在质量和效率上均显著优于基线。
英文原文摘要 1874 字符
Existing video omnimatte methods typically rely on slow, multi-stage, or inference-time optimization pipelines that fail to fully exploit powerful generative priors, producing suboptimal decompositions. Our key insight is that, if a video inpainting model can be finetuned to remove the foreground-associated effects, then it must be inherently capable of perceiving these effects, and hence can also be finetuned for the complementary task: foreground layer decomposition with associated effects. However, although naively finetuning the inpainting model with LoRA applied to all blocks can produce high-quality alpha mattes, it fails to capture associated effects. Our systematic analysis reveals this arises because effect-related cues are primarily encoded in specific DiT blocks and become suppressed when LoRA is applied across all blocks. To address this, we introduce EasyOmnimatte, the first unified, end-to-end video omnimatte method. Concretely, we finetune a pretrained video inpainting diffusion model to learn dual complementary experts while keeping its original weights intact: an Effect Expert, where LoRA is applied only to effect-sensitive DiT blocks to capture the coarse structure of the foreground and associated effects, and a fully LoRA-finetuned Quality Expert learns to refine the alpha matte. During sampling, Effect Expert is used for denoising at early, high-noise steps, while Quality Expert takes over at later, low-noise steps. This design eliminates the need for two full diffusion passes, significantly reducing computational cost without compromising output quality. Ablation studies validate the effectiveness of this Dual-Expert strategy. Experiments demonstrate that EasyOmnimatte sets a new state-of-the-art for video omnimatte and enables various downstream tasks, significantly outperforming baselines in both quality and efficiency.