DeltaQuant：基于时空差分平滑的 4 比特视频扩散模型
DeltaQuant: 4-bit Video Diffusion Models with Spatiotemporal Delta Smoothing
DeltaQuant 将视频扩散模型中的激活值按 3D 时空立方体分组，仅对差分词元做 4-bit 量化，核心词元保留 FP8，为视频生成模型在消费级 GPU 上的低比特推理提供了可复用的量化范式。
原文依据 we focus on quantizing both weights and activations to 4 bits to accelerate these layers
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频扩散模型已取得显著的生成性能，但其巨大的计算与显存开销给部署带来了重大挑战，尤其是在消费级 GPU 上。随着注意力优化领域的最新进展缓解了此前的计算瓶颈，线性层如今在计算成本和推理显存中均占据主导地位。在本工作中，我们专注于将权重和激活值同时量化为 4 比特，以加速这些线性层。现有方法（如 SVDQuant）忽略了激活值在不同去噪时间步上的高度动态特性，即离群通道和幅度会发生剧烈变化。然而，视频数据在空间和时间上相邻的词元之间天然表现出强烈的激活相似性，我们将其称为时空激活相似性，这类似于视频编解码器利用帧内和帧间冗余的方式。利用这一特性，我们提出了 DeltaQuant，它将激活值划分为局部 3D 时空立方体，并使用每个立方体的均值词元作为核心词元，仅将小的差异（差分词元）量化为 4 比特，而核心词元则保持为 FP8。这种分解以极小的开销大幅降低了量化误差。对于权重量化，DeltaQuant 结合了 SVDQuant 的低秩分解来进一步降低量化误差。我们还实现了一个高效内核，将 DeltaQuant 的计算优势转化为实际的速度提升。在 Wan 2.2 I2V、Wan 2.2 T2V 和 LTX-Video T2V 上的大量实验表明，DeltaQuant 保持了高生成保真度。在 Wan 2.2 上，它将模型大小压缩了 2.9 倍，并将显存占用降低了 2.3 倍。DeltaQuant 与高效注意力机制和少步蒸馏兼容。当与这些技术集成时，它额外实现了 3.0 倍的加速，端到端总加速达到 111.8 倍。代码和模型将在论文发表后发布。
英文原文摘要 1928 字符
Video diffusion models have achieved remarkable generative performance, but their substantial computational and memory costs pose significant challenges for deployment, especially on consumer GPUs. As recent advances in attention optimization mitigate previous computational bottlenecks, linear layers now dominate both computational cost and inference memory. In this work, we focus on quantizing both weights and activations to 4 bits to accelerate these layers. Previous methods, such as SVDQuant, overlook the highly dynamic nature of activations across denoising timesteps, where outlier channels and magnitudes vary dramatically. However, video data inherently exhibits strong activation similarity among neighboring tokens in space and time, which we term spatiotemporal activation similarity, analogous to how video codecs exploit intra- and inter-frame redundancy. Leveraging this property, we introduce DeltaQuant, which partitions activations into local 3D spatiotemporal cubes and uses each cube's mean token as a \coretoken, quantizing only the small differences (delta tokens) to 4 bits while keeping core tokens in FP8. This decomposition substantially reduces quantization error with minimal overhead.For weight quantization, DeltaQuant incorporates SVDQuant's low-rank decomposition to further reduce quantization error.We also implement an efficient kernel that translates DeltaQuant's computational benefits into real-world speedups.Extensive experiments on Wan 2.2 I2V, Wan 2.2 T2V, and LTX-Video T2V demonstrate that DeltaQuant maintains high generation fidelity.On Wan 2.2, it compresses model size by 2.9x and reduces memory footprint by 2.3x. DeltaQuant is compatible with efficient attention mechanisms and few-step distillation. When integrated with these techniques, it achieves an additional 3.0x acceleration, for a total 111.8x end-to-end speedup. Code and models will be released upon publication.