CAPT：面向减少视觉-语言错位的混淆感知提示词微调
CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment
把 CLIP 类模型在相似类别间的系统性混淆当作可学习的监督信号：用混淆库加语义/样本双挖掘器与多粒度专家融合来修正提示词，可直接迁移到任何需要细粒度判别或开放词表分类的视觉-语言下游任务。
原文依据 we propose CAPT, a Confusion-Aware Prompt Tuning framework that enables models to learn from their own misalignment
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论CLIP 等视觉-语言模型在跨模态表征学习上取得了显著进展，但在视觉与语义相似的类别之间仍会出现系统性误分类。我们观察到，这类混淆模式并非随机，而是持续发生在特定类别对之间，反映出模型的内在偏置与有限的细粒度判别能力。为此，我们提出 CAPT，一种混淆感知提示词微调框架，使模型能够从自身的错位中学习。具体而言，我们构建混淆库以显式建模类别之间以及误分类样本之间稳定的混淆关系。在此基础上，我们引入语义混淆挖掘器（SEM），通过语义差异提示词与语义共性提示词捕获全局的类间混淆；并引入样本混淆挖掘器（SAM），从混淆库中检索具有代表性的误分类实例，并通过融合全局与局部上下文的 Diff-Manner 适配器捕获样本级线索。为进一步统一不同粒度上的混淆信息，我们设计了多粒度差异专家（MGDE）模块，联合利用语义级与样本级专家进行更稳健的混淆感知推理。在 11 个基准数据集上的大量实验表明，本方法显著减少了由混淆引发的错误，同时提升了基类与新类的判别性与泛化能力，成功化解了 50.72% 的易混淆样本对。代码将发布于 https://github.com/greatest-gourmet/CAPT。
英文原文摘要 1607 字符
Vision-language models like CLIP have achieved remarkable progress in cross-modal representation learning, yet suffer from systematic misclassifications among visually and semantically similar categories. We observe that such confusion patterns are not random but persistently occur between specific category pairs, revealing the model's intrinsic bias and limited fine-grained discriminative ability. To address this, we propose CAPT, a Confusion-Aware Prompt Tuning framework that enables models to learn from their own misalignment. Specifically, we construct a Confusion Bank to explicitly model stable confusion relationships across categories and misclassified samples. On this basis, we introduce a Semantic Confusion Miner (SEM) to capture global inter-class confusion through semantic difference and commonality prompts, and a Sample Confusion Miner (SAM) to retrieve representative misclassified instances from the bank and capture sample-level cues through a Diff-Manner Adapter that integrates global and local contexts. To further unify confusion information across different granularities, a Multi-Granularity Difference Expert (MGDE) module is designed to jointly leverage semantic- and sample-level experts for more robust confusion-aware reasoning. Extensive experiments on 11 benchmark datasets demonstrate that our method significantly reduces confusion-induced errors while enhancing the discriminability and generalization of both base and novel classes, successfully resolving 50.72 percent of confusable sample pairs. Code will be released at https://github.com/greatest-gourmet/CAPT.