TimeRipples：通过理解潜空间中的时空相关性加速 vDiT
TimeRipples: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space
针对 vDiT 自注意力推理延迟，利用潜空间中词元通道级的时空相关性，设计轻量自适应复用策略近似注意力计算，实现 85% 计算节省且视频质量几乎无损，为视频生成模型的高效推理提供可复用方案。
原文依据 we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频生成的激增表明，使用大型视觉模型进行高质量视频合成的需求日益增长。现有视频生成模型主要基于视频扩散 Transformer（vDiT），然而，由于自注意力机制，它们面临显著的推理延迟。尽管先前的研究专注于减少自注意力中的冗余计算，但它们往往忽略了视频流中固有的时空相关性，并直接利用来自大型语言模型的稀疏模式来减少注意力计算。在本工作中，我们采取一种原则性方法，通过利用潜空间中的时空相关性来加速 vDiT 中的自注意力。我们表明，vDiT 内的注意力模式主要源于词元通道级别上主导的空间和时间相关性。基于这一见解，我们提出了一种轻量级自适应复用策略，通过沿单个通道复用空间或时间相关词元的部分注意力分数来近似注意力计算。我们证明，与最优技术相比，我们的方法在 4 个 vDiT 上实现了显著更高的计算节省（85%），同时保持几乎相同的视频质量（VBench 上损失 <0.06%）。
英文原文摘要 1304 字符
The recent surge in video generation has shown the growing demand for high-quality video synthesis using large vision models. Existing video generation models are predominantly based on the video diffusion transformer (vDiT), however, they suffer from substantial inference delay due to self-attention. While prior studies have focused on reducing redundant computations in self-attention, they often overlook the inherent spatio-temporal correlations in video streams and directly leverage sparsity patterns from large language models to reduce attention computations. In this work, we take a principled approach to accelerate self-attention in vDiTs by leveraging the spatio-temporal correlations in the latent space. We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level. Based on this insight, we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels. We demonstrate that our method achieves significantly higher computational savings (85%) compared to state-of-the-art techniques over 4 vDiTs, while preserving almost identical video quality (<0.06% loss on VBench).