通过无数据平坦性感知提示词预训练改进视觉语言模型测试时提示词调优中的校准
Improving Calibration in Test-Time Prompt Tuning for Vision-Language Models via Data-Free Flatness-Aware Prompt Pretraining
提出一种无数据预训练方法，在测试时提示词调优前将提示词初始化到损失景观的平坦区域，从而同时提升视觉语言模型的校准性和性能，且不增加测试时计算开销。
原文依据 we introduce Flatness-aware Prompt Pretraining (FPP), a simple yet effective pretraining framework for TPT that initializes prompts within flatter regions of the loss landscape prior to adaptation.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论测试时提示词调优（TPT）已成为一种有前景的技术，通过使用无标签测试数据优化文本提示词来增强视觉语言模型的适应性。然而，先前的研究观察到TPT通常会产生校准不佳的模型，引发对其预测可靠性的担忧。近期工作通过引入额外的正则化项来约束模型输出以解决此问题，这些方法改善了校准但往往降低了性能。在本工作中，我们揭示这些正则化策略隐式地鼓励优化朝向更平坦的极小值，并且适应后提示词周围损失景观的锐度是控制校准质量的关键因素。受此观察启发，我们引入了平坦性感知提示词预训练（FPP），一个简单而有效的TPT预训练框架，在适应之前将提示词初始化在损失景观的更平坦区域。我们表明，仅替换现有TPT流程中的初始化——而不修改任何其他组件——就足以同时改善校准和性能。值得注意的是，FPP不需要标注数据，并且在测试时调优期间不产生额外计算成本，使其在实际部署中高度实用。代码可在以下网址获取：https://github.com/YonseiML/fpp。
英文原文摘要 1354 字符
Test-time prompt tuning (TPT) has emerged as a promising technique for enhancing the adaptability of vision-language models by optimizing textual prompts using unlabeled test data. However, prior studies have observed that TPT often produces poorly calibrated models, raising concerns about the reliability of their predictions. Recent works address this issue by incorporating additional regularization terms that constrain model outputs, which improve calibration but often degrade performance. In this work, we reveal that these regularization strategies implicitly encourage optimization toward flatter minima, and that the sharpness of the loss landscape around adapted prompts is a key factor governing calibration quality. Motivated by this observation, we introduce Flatness-aware Prompt Pretraining (FPP), a simple yet effective pretraining framework for TPT that initializes prompts within flatter regions of the loss landscape prior to adaptation. We show that simply replacing the initialization in existing TPT pipelines--without modifying any other components--is sufficient to improve both calibration and performance. Notably, FPP requires no labeled data and incurs no additional computational costs during test-time tuning, making it highly practical for real-world deployment. The code is available at: https://github.com/YonseiML/fpp.