通过动态前缀加权增强视觉语言模型的持续学习
Enhancing Continual Learning of Vision-Language Models via Dynamic Prefix Weighting
对做 VLM 下游适配的人，DPW 给出了一种按 token 重要性动态加权 prefix 的门控机制，并让 adapter 只作为 prefix 权重的残差补充，可直接复用到域类增量学习或多任务适配场景中，避免统一归一化带来的过度调整。
原文依据 we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们研究了近期提出的视觉语言模型（VLM）域类增量学习场景。近期工作使用参数高效方法（如前缀微调或适配器）来应对这一挑战，这些方法通过加性向量将任务特定信息注入输入 token，从而促进模型对下游任务的适配。然而，先前方法往往对这些向量的权重进行归一化，忽略了不同输入 token 需要不同程度的调整这一事实。为解决该问题，我们提出动态前缀加权（DPW）框架，动态地为前缀分配权重，并以适配器作为补充。DPW 包含：1）一个门控模块，根据对应输入 token 的重要性调整每个前缀的权重；2）一种加权机制，将适配器输出权重推导为前缀微调权重的残差，确保适配器仅在必要时被使用。实验结果表明，我们的方法在 VLM 域类增量学习场景中取得了最优性能。代码见：https://github.com/YonseiML/dpw。
英文原文摘要 1175 字符
We investigate recently introduced domain-class incremental learning scenarios for vision-language models (VLMs). Recent works address this challenge using parameter-efficient methods, such as prefix-tuning or adapters, which facilitate model adaptation to downstream tasks by incorporating task-specific information into input tokens through additive vectors. However, previous approaches often normalize the weights of these vectors, disregarding the fact that different input tokens require different degrees of adjustment. To overcome this issue, we propose Dynamic Prefix Weighting (DPW), a framework that dynamically assigns weights to prefixes, complemented by adapters. DPW consists of 1) a gating module that adjusts the weights of each prefix based on the importance of the corresponding input token, and 2) a weighting mechanism that derives adapter output weights as a residual of prefix-tuning weights, ensuring that adapters are utilized only when necessary. Experimental results demonstrate that our method achieves state-of-the-art performance in domain-class incremental learning scenarios for VLMs. The code is available at: https://github.com/YonseiML/dpw.