通过异构缓存加速基于扩散的视频编辑：超越在采样去噪时间步上的全量计算
Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep
对做视频扩散编辑的人，HetCache 提供了一种免训练、可插拔的 DiT 推理加速手段：按空间先验把时空 token 分为上下文 token 与生成 token，只缓存与生成 token 相关性最强的上下文 token，从而跳过冗余注意力计算，可直接套用到 MV2V 类编辑管线以降低延迟与浮点运算量。
原文依据 HetCache achieves a noticeable acceleration, including a 2.67 xlatency speedup and FLOPs reduction over commonly used foundation models
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于扩散的视频编辑已成为高质量、灵活内容生成的重要范式。然而，尽管扩散 Transformer（DiT）具有通用性和强大的建模能力，其迭代去噪过程仍带来高昂的计算开销，给实际部署带来挑战。现有的视频扩散加速方法主要利用去噪时间步级别的特征复用，缓解了去噪过程中的冗余，但忽略了 DiT 架构内部的冗余：许多作用于时空 token 的注意力操作被冗余执行，对模型输出的增量贡献很小甚至没有。本文提出 HetCache，一个免训练的扩散加速框架，旨在利用基于扩散的掩码视频到视频（MV2V）生成与编辑中固有的异构性。HetCache 不再均匀复用或随机采样 token，而是在指定的计算步骤中评估各类 token 之间的上下文相关性与交互强度。在空间先验的引导下，它将 DiT 模型中的时空 token 划分为上下文 token 与生成 token，并选择性地缓存那些与生成 token 相关性最强、语义最具代表性的上下文 token。该策略在保持编辑一致性与保真度的同时减少了冗余的注意力操作。实验表明，HetCache 实现了显著的加速，相比常用的基础模型取得 2.67 倍的延迟加速并降低了浮点运算量，而编辑质量下降可忽略不计。
英文原文摘要 1643 字符
Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model's output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67 xlatency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality.