YOSE：为高效 DiT 视频物体移除仅选择必要词元
YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal
对做 DiT 视频生成/编辑的人，这篇给出了一个可复用的思路：用可微的动态索引按掩码只处理必要词元，并用扩散过程模拟器近似未掩码区域在自注意力中的影响，从而让推理时间随掩码面积线性增长，可直接迁移到 inpainting、视频编辑等掩码驱动的生成任务上做加速。
原文依据 YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期基于扩散 Transformer（DiT）的视频生成技术进展在视频物体移除上取得了令人印象深刻的结果。然而，这些方法仍然面临显著的推理延迟。例如，尽管 MiniMax Remover 达到了最优的视觉质量，其运行速度仅为约 10FPS，主要原因在于对整个时空词元空间进行稠密计算，即使实际只有一小块掩码区域需要处理。本文提出 YOSE（You Only Select Essential Tokens），一个高效的微调框架。YOSE 引入两个关键组件：批处理变长索引（Batch Variable-length Indexing，BVI）与扩散过程模拟器（Diffusion Process Simulator，DiffSim）模块。BVI 是一种可微的动态索引算子，它根据掩码信息自适应地选择必要词元，从而支持跨样本的变长词元处理。DiffSim 为未掩码词元提供了一种扩散过程近似机制，在 DiT 自注意力中模拟未掩码区域的影响，以维持掩码词元的语义一致性。借助这些设计，YOSE 实现了掩码感知的加速：推理时间随掩码区域近似线性增长，而全词元扩散方法的计算量无论掩码大小如何都保持不变。大量实验表明，YOSE 在 70% 的情况下实现了最高 2.5 倍的加速，同时保持与基线相当的视觉质量。代码已发布于：https://github.com/Wucy0519/YOSE-CVPR26。
英文原文摘要 1542 字符
Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26.