以文本为主、视觉为辅：面向大型视觉语言模型的非对称文本-视觉剪枝
Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models
针对多模态大模型提出非对称剪枝策略，视觉研究者可借鉴其按模态分别校准重要性的思路，用于压缩视觉-语言模型或设计模态感知的稀疏化方法。
原文依据 we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论网络剪枝是使大型视觉语言模型（LVLMs）轻量化的一项有效技术，其重要性度量主要同时纳入权重和激活值。然而，现有工作通常以统一方式处理来自不同模态的校准数据，忽略了模态特有的行为。这引出一个关键挑战：如何应对文本与视觉令牌的差异化行为，以实现LVLMs的准确剪枝。为此，我们通过解耦视觉与文本令牌对应的权重，系统研究了它们对剪枝操作的敏感性，揭示出：(i) 文本通路应通过文本令牌进行校准，因为其敏感性高于视觉通路；(ii) 视觉通路具有高度冗余性，甚至可允许50%的稀疏度。受这些发现启发，我们提出一种简单而有效的面向LVLMs的非对称文本-视觉权重剪枝方法，称为ATV-Pruning，该方法通过从文本和视觉通路中选取信息量大的令牌来建立重要性度量，以实现准确的权重剪枝。具体而言，ATV-Pruning融合了两项主要创新：首先，通过利用所有文本令牌和一部分视觉令牌自适应地构建校准池；其次，我们设计了一种层自适应选择策略来生成重要的视觉令牌。最后，在标准多模态基准上的大量实验验证了ATV-Pruning优于最优方法。
英文原文摘要 1538 字符
Network pruning is an effective technique for enabling lightweight Large Vision-Language Models (LVLMs), which primarily incorporates both weights and activations into the importance metric. However, existing efforts typically process calibration data from different modalities in a unified manner, overlooking modality-specific behaviors. This raises a critical challenge: how to address the divergent behaviors of textual and visual tokens for accurate pruning of LVLMs. To this end, we systematically investigate the sensitivity of visual and textual tokens to the pruning operation by decoupling their corresponding weights, revealing that: (i) the textual pathway should be calibrated via text tokens, since it exhibits higher sensitivity than the visual pathway; (ii) the visual pathway exhibits high redundancy, permitting even 50% sparsity. Motivated by these insights, we propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for LVLMs, dubbed ATV-Pruning, which establishes the importance metric for accurate weight pruning by selecting the informative tokens from both textual and visual pathways. Specifically, ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens; second, we devise a layer-adaptive selection strategy to yield important visual tokens. Finally, extensive experiments across standard multimodal benchmarks verify the superiority of our ATV-Pruning over state-of-the-art methods.