VecAttention：面向长上下文推理加速的向量级稀疏注意力
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
对做长视频理解或视频生成的视觉研究者，这篇论文提供了一个可直接替换全注意力的向量级稀疏注意力方案：利用视频注意力图的垂直向量稀疏模式，配合轻量重要向量选择与定制稀疏核，在 VideoMME、LongVideoBench、VBench 等任务上以约 2.65 倍加速保持精度，可作为长上下文视频模型推理加速的通用组件。
原文依据 VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论长上下文视频理解与生成由于自注意力的二次复杂度，给基于 Transformer 的视频模型带来了显著的计算挑战。现有稀疏注意力方法采用粗粒度模式来提升效率，但通常会产生冗余计算并导致次优性能。为解决该问题，本文提出 VecAttention，一种新颖的向量级稀疏注意力框架，为视频模型实现更优的精度—效率权衡。我们观察到视频注意力图呈现出强烈的垂直向量稀疏模式，并进一步证明与现有粗粒度稀疏模式相比，该垂直向量模式能持续提供更好的精度—稀疏度权衡。基于这一观察，VecAttention 通过轻量级的重要向量选择（将内存访问开销降至最低）与优化的向量稀疏注意力核，动态地仅选择并处理信息量大的垂直向量。在视频理解（VideoMME、LongVideoBench 和 VCRBench）与生成（VBench）任务上的全面评估表明，VecAttention 相比全注意力实现 2.65 倍加速，相比最优稀疏注意力方法实现 1.83 倍加速，同时精度与全注意力相当。
英文原文摘要 1320 字符
Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose VecAttention, a novel vector-wise sparse attention framework that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized vector sparse attention kernel. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65xspeedup over full attention and a 1.83xspeedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention.