面向视觉语言模型的协同多模式剪枝
Collaborative Multi-Mode Pruning for Vision-Language Models
对需要在边缘设备部署 VLM 的视觉研究者，这篇给出了参数剪枝与词元剪枝联合进行的框架：CIM 把词元显著性纳入参数重要性打分、并反向抑制已剪参数对词元打分的影响，MPS 用剪枝代价驱动的多阶段调度自适应切换剪枝模式，可直接迁移到 BLIP、LLaVA 类多模态模型的压缩流程中。
原文依据 we propose Collaborative Multi-Mode Pruning (CoMP), a novel framework tailored for VLMs by performing joint parameter and token pruning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在统一的 Transformer 架构内快速发展，然而由于计算复杂度高，其在资源受限设备上的部署仍然困难。剪枝虽已成为压缩 VLM 的有效技术，但现有方法大多只关注单一模式，即仅剪枝参数或仅剪枝词元，未能充分挖掘各模式内部的固有冗余，导致在高剪枝率下性能显著下降。为解决上述局限，我们提出协同多模式剪枝（CoMP），这是一个专为 VLM 设计、联合执行参数剪枝与词元剪枝的新框架。具体而言，我们首先设计协同重要性指标（CIM），用于考察相互耦合的参数与词元之间的相互干扰：它将词元的不同显著性纳入参数重要性分数的计算，同时减轻被剪枝参数对词元重要性分数的影响。此外，我们提出多模式剪枝策略（MPS），将整体剪枝过程分解为一系列剪枝阶段，在每个阶段中依据剪枝代价估计不同剪枝模式的优先级，并自适应地切换到最优模式。MPS 还融合了历史代价与随机探索，以实现稳定的剪枝过程并避免陷入局部最优。在多种视觉语言任务与模型上的大量实验表明，与最优方法相比，我们的方法在高剪枝率下有效提升了性能。源代码见 https://github.com/Wuzimeng/CoMP.git。
英文原文摘要 1735 字符
Vision-Language Models (VLMs) have advanced rapidly within the unified Transformer architecture, yet their deployment on resource-constrained devices remains challenging due to high computational complexity. While pruning has emerged as an effective technique for compressing VLMs, existing approaches predominantly focus on a single mode by pruning either parameters or tokens, neglecting fully exploring the inherent redundancy in each mode, which leads to substantial performance degradation at high pruning ratios. To address the above limitations, we propose Collaborative Multi-Mode Pruning (CoMP), a novel framework tailored for VLMs by performing joint parameter and token pruning. Specifically, we first design a Collaborative Importance Metric (CIM) that investigates the mutual interference between the coupled parameters and tokens. It incorporates distinct significance of tokens into the computation of parameter importance scores, while simultaneously mitigating the affect of pruned parameters on token importance scores. Moreover, we develop a Multi-Mode Pruning Strategy (MPS) that decomposes the overall pruning process into a sequence of pruning stages, while in each stage we estimate the priory of different pruning modes based on their pruning cost and adaptively shift to the optimal one. Additionally, MPS integrates the historical cost and random exploration, in order to achieve a stable pruning process and avoid local optimum. Extensive experiments across various vision-language tasks and models demonstrate that our method effectively promotes the performance under high pruning ratios by comparing to the state-of-the-art approaches. The source code is available at https://github.com/Wuzimeng/CoMP.git.