STAR：测试时自适应可增强视觉语言模型的通用提示词学习
STAR: Test-Time Adaptation Can Enhance Universal Prompt Learning for Vision-Language Models
提出基于Fisher分数的可分离性感知门控与共轭优化，结合动态记忆库的原型检索来校准OOD检测，为视觉语言模型在无标签且含OOD样本的目标域上进行测试时提示词学习提供了可靠伪标签生成与偏差缓解的通用技术方案。
原文依据 we propose a novel framework named Separability-aware Conjugate Optimization with Prototypical Retrieval (STAR) for universal test-time prompt learning of VLMs.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文研究视觉语言模型（VLM）的通用测试时提示词学习问题，旨在通过包含分布外（OOD）样本的无标签目标数据增强预训练VLM的提示词学习。然而，现有的测试时自适应方法往往忽略目标域中的类别特定多样性，并且由于不确定性估计不足而依赖不可靠的伪标签，这可能在测试时导致额外的自适应偏差。为此，我们提出了一种名为可分离性感知共轭优化与原型检索（STAR）的新框架，用于VLM的通用测试时提示词学习。STAR的核心是将可分离性感知门控机制融入共轭优化，以利用OOD样本进行可靠的伪学习。具体而言，我们首先计算Fisher分数以量化分布内（ID）和OOD样本之间的可分离性，该分数指导我们的软门控机制进行划分训练。然后，我们采用共轭优化为无标签数据推导可靠的伪标签以进行测试时自适应。为了进一步减轻OOD检测中的偏差，我们维护一个动态记忆库，存储高置信度样本以构建类别原型，这些原型将作为原型检索的查询以校准OOD检测。在多个基准上的大量实验表明，STAR始终优于竞争的基线方法。
英文原文摘要 1474 字符
This paper studies the problem of universal test-time prompt learning for vision-language models (VLMs) which aims to enhance prompt learning for a pre-trained VLM via unlabeled target data containing out-of-distribution (OOD) samples. However, existing test-time adaptation approaches often overlook class-specific diversity in the target domain and rely on unreliable pseudo-labels due to inadequate uncertainty estimation, which may result in additional adaptation bias during test time. Towards this end, we propose a novel framework named Separability-aware Conjugate Optimization with Prototypical Retrieval (STAR) for universal test-time prompt learning of VLMs. The core of our STAR is to incorporate a separability-aware gating mechanism into conjugate optimization for reliable pseudo-learning with OOD samples. In particular, we first compute the Fisher score to quantify the separability between in-distribution (ID) and OOD samples, which guides our soft gating mechanism for divided training. Then, we employ conjugate optimization to derive reliable pseudo-labels of unlabeled data for test-time adaptation. To further mitigate biases in OOD detection, we maintain a dynamic memory bank which stores high-confidence samples to build class-wise prototypes, which would serve as queries for prototypical retrieval to calibrate OOD detection. Extensive experiments on multiple benchmarks demonstrate that STAR consistently outperforms competing baseline methods.