LS-ViT：面向视觉 Transformer 低比特训练后量化的最小二乘 Hessian 块重建
LS-ViT: Least-Squares Hessian Based Block Reconstruction for Low-Bit Post-Training Quantization of Vision Transformers
对需要在边缘设备部署 ViT 的视觉研究者，这篇给出了一个即插即用的 4-bit 以下 PTQ 块重建方案：用最小二乘显式最小化跨样本的 Hessian 近似误差（补上被忽略的协方差项），可直接替换 APHQ-ViT/FIMA-Q 中的 Hessian 估计环节，且量化耗时更低。
原文依据 we propose LS-ViT, a block reconstruction framework that effectively estimates a representative Hessian by explicitly minimizing this approximation error across all samples
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉 Transformer（ViT）已在各类视觉任务上取得最优结果。为使 ViT 能够在现代硬件系统上实际部署，近年来训练后量化（PTQ）得到了积极研究。特别是基于 Hessian 的块重建方法在将 ViT 模型量化为超低比特（如 4-bit）方面展现出良好效果。然而，作为 APHQ-ViT 与 FIMA-Q 等近期方法中的基础步骤，如何找到一个具有代表性的近似 Hessian，在量化诱导误差与估计代价两方面仍缺乏充分探索。针对这些不足，我们首先揭示：近期工作中使用的样本独立性假设忽略了协方差项，会导致显著的近似误差，在低于 4-bit 时尤为明显。受最小二乘回归启发，我们提出 LS-ViT，一个块重建框架，通过在所有样本上显式最小化该近似误差来有效估计具有代表性的 Hessian。在不同视觉任务上针对多种 ViT 模型的大量实验表明，LS-ViT 取得了新的最优性能。此外，与先前工作相比，LS-ViT 降低了量化时间，从而为 ViT 提供了一种实用、即插即用的量化感知部署方案。源代码已公开。
英文原文摘要 1362 字符
Vision Transformers (ViTs) have achieved state-of-the-art results across various vision tasks. To enable a practical deployment of ViTs on modern hardware systems, post-training quantization (PTQ) has been actively studied in recent years. In particular, Hessian-based block reconstruction approaches have demonstrated promising results in quantizing ViT models to ultra-low bitwidths (e.g., 4-bit). However, finding a representative approximate Hessian, a fundamental step in recent approaches such as APHQ-ViT and FIMA-Q, remains underexplored in terms of the quantization-induced error and estimation cost. To address these shortcomings, we first reveal that the sample independence assumption used in recent works, which ignores the covariance term, can lead to a significant approximation error, especially for sub-four-bits. Inspired by least-squares regression, we propose LS-ViT, a block reconstruction framework that effectively estimates a representative Hessian by explicitly minimizing this approximation error across all samples. Extensive experiments with various ViT models across different vision tasks demonstrate that LS-ViT achieves new state-of-the-art performance. In addition, LS-ViT reduces quantization time compared to prior work, enabling a practical, plug-and-play, quantization-aware deployment for ViTs. The source code is available.