面向视觉语言模型提示词微调的校准
Towards Calibrating Prompt Tuning of Vision- Language Models
为 CLIP 提示词微调提供了一套即插即用的校准正则项：均值-方差间隔惩罚稳定类间 logit 间隔，文本矩匹配损失约束微调后文本嵌入的一二阶矩贴近冻结 CLIP 嵌入，可直接加到现有 prompt tuning 训练目标上降低 ECE。
原文依据 a mean-variance margin penalty that stabilizes inter-class logit margins by maximizing their average while minimizing dispersion
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论CLIP 等大规模视觉语言模型的提示词微调无需更新模型权重即可实现高效的任务适配。然而，它往往导致置信度校准不佳与预测不确定性不可靠。我们通过提出一个校准框架来解决该问题，在提升预测可靠性的同时保持预训练 CLIP 嵌入空间的几何结构，而这对稳健泛化是必需的。我们的方法在标准交叉熵损失基础上扩展了两个互补的正则项：(1) 均值-方差间隔惩罚，通过最大化类间 logit 间隔的平均值并最小化其离散度来稳定间隔，从而缓解置信不足与过度自信的尖峰；(2) 文本矩匹配损失，使微调后文本嵌入的一阶与二阶矩与冻结的 CLIP 对应嵌入对齐，从而保持对泛化至关重要的语义离散度。通过在 7 种提示词微调方法和 11 个多样数据集上的大量实验，我们证明：与有竞争力的校准技术相比，我们的方法在基类和新类上均显著降低了期望校准误差（ECE）。
英文原文摘要 1156 字符
Prompt tuning of large-scale vision-language models such as CLIP enables efficienttask adaptation without updating model weights. However, it often leads to poorconfidence calibration and unreliable predictive uncertainty. We address thisproblem by proposing a calibration framework that enhances predictive reliabilitywhile preserving the geometry of the pretrained CLIP embedding space, which isrequired for robust generalization. Our approach extends the standard cross-entropyloss with two complementary regularizers: (1) a mean-variance margin penalty thatstabilizes inter-class logit margins by maximizing their average while minimizingdispersion, mitigating underconfidence and overconfidence spikes; and (2) a textmoment-matching loss that aligns the first and second moments of tuned textembeddings with their frozen CLIP counterparts, preserving semantic dispersioncrucial for generalization. Through extensive experiments across 7 prompt-tuningmethods and 11 diverse datasets, we demonstrate that our approach significantlyreduces the Expected Calibration Error (ECE) compared to competitive calibrationtechniques on both base and novel classes.