从量化残差到连续提示词:面向视觉语言模型少样本类增量学习
Quantized Residuals to Continuous Prompts for Few-Shot Class Incremental Learning in Vision-Language Models
把 VLM 图文特征的残差当作补充细粒度线索的载体,并用离散码本量化残差子空间、再由分层提示词编码器解码成连续类自适应提示词,这套「残差量化→连续提示词」的接口可直接迁移到其他需要稳定+可塑的 VLM 增量适配场景。
原文依据 leverages the visual-textual feature residual of VLMs to recover discriminative fine-grained cues missing from the contrastive space
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论少样本类增量学习(FSCIL)要求在防止灾难性遗忘的同时,从极为有限的数据中学习新类别。现有方法主要依赖视觉特征,容易过拟合;近期的视觉语言模型(VLM)具有更好的迁移性,但由于对比式特征去相关,会抑制细粒度信息。此外,当前的 FSCIL 方法通常使用静态或完全可优化的提示词,使其要么过于刚性,要么在增量会话中易受语义漂移影响。我们提出 QR-Prompt,一个由残差驱动的框架,利用 VLM 的视觉-文本特征残差来恢复对比空间中缺失的判别性细粒度线索。为保证稳定性,我们提出判别子空间量化(DSQ),构建残差子空间的离散记忆。为获得可塑性,分层提示词编码器(HPE)与提示词合成器(PC)将这些离散编码转换为面向新类别的连续、类自适应提示词。我们推导了将 DSQ 码本大小与泛化和分类间隔相联系的界,并在 CUB200、CIFAR100 和 miniImageNet 上相较最优 FSCIL 方法取得了一致提升。结果表明,基于残差的量化结合分层提示词合成,可为 FSCIL 带来稳定且富有表现力的 VLM 适配。
英文原文摘要 1378 字符
Few-shot Class-Incremental Learning (FSCIL) requires learning new classes from very limited data while preventing catastrophic forgetting. Existing methods rely mainly on visual features and are prone to overfitting, while recent vision-language models (VLMs) offer better transferability but suppress fine-grained information due to contrastive feature decorrelation. Moreover, current FSCIL approaches often use static or fully optimizable prompts, making them either rigid or susceptible to semantic drift in incremental sessions. We introduce QR-Prompt, a residual-driven framework that leverages the visual-textual feature residual of VLMs to recover discriminative fine-grained cues missing from the contrastive space. To ensure stability, we propose Discriminative Subspace Quantization (DSQ), which builds a discrete memory of residual subspaces. To enable plasticity, a Hierarchical Prompt Encoder (HPE) and Prompt Composer (PC) transform these discrete codes into continuous, class-adaptive prompts for novel classes. We derive bounds relating DSQ codebook size to generalization and classification margin, and achieve consistent improvements over state-of-the-art FSCIL methods on CUB200, CIFAR100, and miniImageNet. Our results show that residual-based quantization combined with hierarchical prompt composition yields stable and expressive VLM adaptation for FSCIL.