S2D：选择性谱衰减，用于神经激活的量化友好调理
S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
对做视觉大模型部署的人，S2D 给出一个在微调阶段抑制激活离群值的正则手段：只对权重最大奇异值对应的分量做谱衰减，就能让 CLIP/SigLIP 类骨干在 W4A4 PTQ 下少掉点，可直接接到自己的量化流水线前。
原文依据 we propose Selective Spectral Decay (S2D), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模 Transformer 模型中的激活离群值对模型量化构成根本性挑战，它造成过大的数值范围，导致量化时精度严重下降。我们通过实验观察到，离群值的严重程度随预训练规模增大而加剧（例如从 CLIP 到训练更充分的 SigLIP 和 SigLIP2 逐步恶化）。通过理论分析以及经验相关性研究，我们建立了这些激活离群值与权重主导奇异值之间的直接联系。基于这一发现，我们提出选择性谱衰减（S2D），一种具有几何原理依据的调理方法，它在微调期间仅对与最大奇异值对应的权重分量进行精准正则化。通过大量实验，我们证明 S2D 显著减少了激活离群值，并产生本身即量化友好的良态表示。在 W4A4 量化下，用 S2D 训练的模型在 ImageNet 上的 PTQ 精度最高提升 7%，与 QAT 结合时提升 4%。这些改进还可泛化到下游任务和视觉语言模型，使得在不牺牲部署效率的前提下扩展规模日益增大、训练日益严苛的模型成为可能。
英文原文摘要 1296 字符
Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization, creating excessively large ranges that cause severe accuracy drops during quantization. We empirically observe that outlier severity intensifies with pre-training scale (e.g., progressing from CLIP to the more extensively trained SigLIP and SigLIP2). Through theoretical analysis as well as empirical correlation studies, we establish the direct link between these activation outliers and dominant singular values of the weights. Building on this insight, we propose Selective Spectral Decay (S2D), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values during fine-tuning. Through extensive experiments, we demonstrate that S2D significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly. Models trained with S2D achieve up to 7% improved PTQ accuracy on ImageNet under W4A4 quantization and 4% gains when combined with QAT. These improvements also generalize across downstream tasks and vision-language models, enabling the scaling of increasingly large and rigorously trained models without sacrificing deployment efficiency.