BinaryAttention：面向视觉与扩散Transformer的一比特QK注意力
BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
将注意力中的QK点积二值化为符号位运算，用位运算替代浮点乘法，在视觉Transformer和扩散Transformer上实现端到端加速，为低比特注意力提供可复用方案。
原文依据 we retain only the sign of queries and keys in computing the attention, and replace the floating dot products with bit-wise operations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer已取得广泛而显著的成功，但其注意力模块的计算复杂度仍是视觉任务的主要瓶颈。现有方法主要采用8比特或4比特量化来平衡效率与精度。本文通过理论论证指出，注意力的二值化能够保留本质的相似性关系，并提出BinaryAttention，一种用于快速且精确的一比特QK注意力的有效方法。具体而言，我们在计算注意力时仅保留查询和键的符号，并用位运算替代浮点点积，从而显著降低计算开销。我们通过引入可学习偏置来缓解一比特量化下的固有信息损失，并实现端到端加速。为保持注意力的精度，我们采用量化感知训练和自蒸馏技术，在减少量化误差的同时确保符号对齐的相似性。在A100 GPU上，BinaryAttention比FlashAttention2快2倍以上。在视觉Transformer和扩散Transformer基准上的大量实验表明，BinaryAttention达到甚至超越全精度注意力，验证了其有效性。我们的工作为全精度注意力提供了一种高效且有效的替代方案，推动了视觉任务低比特Transformer的前沿。代码和模型可在https://github.com/EdwardChasel/BinaryAttention获取。
英文原文摘要 1518 字符
Transformers have achieved widespread and remarkable success, while the computational complexity of their attention modules remains a major bottleneck for vision tasks. Existing methods mainly employ 8-bit or 4-bit quantization to balance efficiency and accuracy. In this paper, with theoretical justification, we indicate that binarization of attention preserves the essential similarity relationships, and propose BinaryAttention, an effective method for fast and accurate 1-bit qk-attention. Specifically, we retain only the sign of queries and keys in computing the attention, and replace the floating dot products with bit-wise operations, significantly reducing the computational cost. We mitigate the inherent information loss under 1-bit quantization by incorporating a learnable bias, and enable end-to-end acceleration. To maintain the accuracy of attention, we adopt quantization-aware training and self-distillation techniques, mitigating quantization errors while ensuring sign-aligned similarity. BinaryAttention is more than 2xfaster than FlashAttention2 on A100 GPUs. Extensive experiments on vision transformer and diffusion transformer benchmarks demonstrate that BinaryAttention matches or even exceeds full-precision attention, validating its effectiveness. Our work provides a highly efficient and effective alternative to full-precision attention, pushing the frontier of low-bit transformers for vision tasks. The codes and models can be found at https://github.com/EdwardChasel/BinaryAttention.