重新审视多标签图像识别微调中的 BCE 损失
Rethinking BCE Loss for Multi-Label Image Recognition with Fine-Tuning
针对 CLIP 提示词微调做多标签识别时 BCE 监督导致的置信度失准问题，提出用类别协方差正则把预测的类别置信度协方差结构对齐到预训练文本嵌入的语义相关性上，可作为即插即用的正则项用于任何基于 CLIP 的多标签/多类别微调流程，改善校准而不损精度。
原文依据 we propose Class-wise Covariance Regularization, which aligns the predicted covariance structure of class confidences with the semantic correlations encoded in pretrained text embeddings
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论微调 CLIP 等视觉语言模型已成为多标签图像识别的主流范式，而提示词微调因其轻量的参数开销和强大的迁移能力被广泛采用。然而我们发现，当这些方法使用二元交叉熵作为监督损失时，模型的置信度结构会被系统性地扭曲，导致明显的失准。现有的校准技术，如温度缩放或基于正则化的方法，在多标签设定下大多失效，因为它们既无法捕捉类别之间固有的语义依赖，也无法纠正微调过程中引入的全局结构偏移。为解决这一问题，我们提出类别协方差正则化（Class-wise Covariance Regularization, CCR），它将类别置信度的预测协方差结构与预训练文本嵌入所编码的语义相关性对齐。这种对齐在整个微调过程中保持了类别空间的几何一致性，从而在各类别上得到更稳定、更可解释的置信度分布。在多标签基准上的实验表明，CCR 在显著降低校准误差的同时，保持甚至提升了识别性能。
英文原文摘要 1249 字符
Fine-tuning vision-language models such as CLIP has become the mainstream paradigm for multi-label image recognition, and prompt tuning is widely adopted due to its lightweight parameter cost and strong transferability. However, we find that when these methods use Binary Cross-entropy as the supervision loss, the model's confidence structure becomes systematically distorted, leading to pronounced miscalibration. Existing calibration techniques, such as temperature scaling or regularization-based methods, largely fail in multi-label settings because they cannot capture inherent semantic dependencies between classes, nor can they correct the global structural shifts introduced during fine-tuning. To address this issue, we propose Class-wise Covariance Regularization, which aligns the predicted covariance structure of class confidences with the semantic correlations encoded in pretrained text embeddings. This alignment preserves the geometric consistency of the class space throughout fine-tuning, resulting in more stable and interpretable confidence distributions across categories. Experiments on multi-label benchmarks show that CCR significantly reduces calibration errors while maintaining or even improving recognition performance.