CAR-SAM：面向分割一切模型训练后量化的交叉注意力重建
CAR-SAM: Cross-Attention Reconstruction for Post-Training Quantization of the Segment Anything Model
针对 SAM 解码器中交叉注意力结构做训练后量化，提出把 MatMul 激活量化误差转移到前置线性权重的补偿机制，以及联合重建双向耦合注意力分支的策略，可直接迁移到其他含交叉注意力的视觉 Transformer 模型（如检测、分割解码器）的低比特部署。
原文依据 we propose CAR-SAM, a unified quantization framework tailored for SAMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论分割一切模型（SAM）被广泛用于计算机视觉中的通用图像分割，但其高昂的计算与内存需求使其难以部署在资源受限设备上。训练后量化（PTQ）是一种广泛使用的模型压缩与加速技术。然而，现有 PTQ 方法未考虑 SAM 解码器中的交叉注意力架构。这种性能退化主要源于 SAM 带来的独特挑战：（1）注意力耗散，即解码器中对于表示分割掩码至关重要的注意力信息在低比特量化下坍缩为弥散且无语义的形式；（2）重建振荡，即双向 Transformer 内部的双向耦合引入跨分支误差干扰并使收敛不稳定。为解决这些问题，我们提出 CAR-SAM，一个为 SAM 量身定制的统一量化框架。首先，为缓解注意力耗散，我们引入矩阵乘法感知补偿（MAC）机制，将激活引起的量化误差从矩阵乘法转移到前置线性权重上。其次，为缓解解码器优化中的振荡，我们提出联合交叉注意力重建（JCAR）策略，联合重建相互耦合的注意力分支，抑制振荡行为并促进稳定收敛。大量实验表明，CAR-SAM 能将 SAM 模型稳健地量化至 4 比特精度，在 SAM-B 和 SAM-L 上分别以 14.6% 和 6.6% 的平均精度均值超越现有方法。
英文原文摘要 1584 字符
Segment Anything Models (SAMs) are extensively used in computer vision for universal image segmentation, but deploying them on resource-constrained devices is challenging due to their high computational and memory demands. Post-Training Quantization (PTQ) is a widely used technique for model compression and acceleration. However, existing PTQ methods fail to consider the cross-attention architecture in the SAM decoder. This degradation primarily stems from the unique challenges posed by SAMs: (1) Attention dissipation, where the attention information in the decoder, which is crucial for representing segmentation masks, collapses into a diffuse and non-semantic form under low-bit quantization; and (2) Reconstruction oscillation, where bidirectional coupling within the two-way transformer introduces cross-branch error interference and destabilizes convergence. To tackle these issues, we propose CAR-SAM, a unified quantization framework tailored for SAMs. Firstly, to mitigate attention dissipation, we introduce MatMul-Aware Compensation (MAC) mechanism that transfers activation-induced quantization errors from MatMul to preceding linear weights. Secondly, to mitigate oscillation in decoder optimization, we develop a Joint Cross-Attention Reconstruction (JCAR) strategy that jointly reconstructs coupled attention branches, suppressing oscillatory behavior and promoting stable convergence. Extensive experiments show that CAR-SAM robustly quantizes SAM models down to 4-bit precision, surpassing existing methods by 14.6% and 6.6% mAP on SAM-B and SAM-L respectively.