面向视觉语言模型长尾泛化的聚类感知神经坍缩提示词微调
Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
把神经坍缩的 ETF 几何约束引入提示词微调，用聚类级局部约束替代全局约束，为在类不平衡数据上微调 VLM 时保住尾类判别力提供了一个可复用的损失设计思路。
原文依据 we propose cluster-aware neural collapse prompt tuning (CPT), which enhances the discriminability of tail classes in prompt-tuned VLMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论提示词学习已成为微调预训练视觉语言模型（VLM）的一种高效替代方案。尽管前景可观，现有方法在适配类不平衡数据集时仍难以维持尾类判别力。本文提出聚类感知神经坍缩提示词微调（CPT），在不牺牲整体泛化能力的前提下增强提示词微调 VLM 中尾类的判别力。首先，我们通过从预训练 VLM 中挖掘语义指派并将其映射到提示词微调后的特征，设计了一个聚类不变空间。该空间计算聚类级边界并将约束限制在局部邻域内，从而减少对预训练 VLM 全局语义结构的干扰。其次，我们引入神经坍缩驱动的判别力优化，包含三个损失：文本等角紧框架（ETF）分离损失、类级收敛损失与旋转稳定损失。这些损失协同塑造簇内几何结构，以获得更好的类间分离与类内对齐。在 11 个多样化数据集上的大量实验表明，CPT 优于当前最优方法，在长尾类上表现更强，并对未见类具有良好的泛化能力。
英文原文摘要 1275 字符
Prompt learning has emerged as an efficient alternative to fine-tuning pre-trained vision-language models (VLMs).Despite its promise, current methods still struggle to maintain tail-class discriminability when adapting to class-imbalanced datasets. In this work, we propose cluster-aware neural collapse prompt tuning (CPT), which enhances the discriminability of tail classes in prompt-tuned VLMs without sacrificing their overall generalization.First, we design a cluster-invariant space by mining semantic assignments from the pre-trained VLM and mapping them to prompt-tuned features.This computes cluster-level boundaries and restricts the constraints to local neighborhoods, which reduces interference with the global semantic structure of the pre-trained VLM.Second, we introduce neural-collapse-driven discriminability optimization with three losses: textual Equiangular Tight Frame (ETF) separation loss, class-wise convergence loss, and rotation stabilization loss.These losses work together to shape intra-cluster geometry for better inter-class separation and intra-class alignment.Extensive experiments on 11 diverse datasets demonstrate that CPT outperforms SOTA methods, with stronger performance on long-tail classes and good generalization to unseen classes.