QuietPrune：面向视觉语言模型的查询引导早期词元剪枝
QuietPrune: Query-Guided Early Token Pruning for Vision-Language Models
把剪枝时机提前到 ViT 内部，并用投影器逆变换构造的 [Q-CLS] 词元为视觉词元提供文本相关性打分，这一「查询引导的早期剪枝」思路可直接迁移到任何 ViT-LLM 架构的 VLM 推理加速中，且 2x2 分组剪枝与主流视觉词元合并操作兼容。
原文依据 we recognize that implementing early pruning within the vision transformer (ViT) can achieve benefits in both latency reduction and accuracy maintenance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在多模态任务中展现出强大能力，但数量庞大的视觉词元带来了显著的计算开销。本文提出 QuietPrune，一种查询引导的早期词元剪枝方法，用于移除 VLM 中冗余的视觉词元，从而提升计算效率。与以往的后期剪枝方法不同，我们发现：在视觉 Transformer（ViT）内部实施早期剪枝，能够在降低延迟与保持精度两方面同时获益。为解决早期剪枝中的语义丢失问题，我们设计了一个轻量级适配器，其做法是对 VLM 中的投影器执行逆变换。该适配器将上下文查询转换为视觉域中的 [Q-CLS]（Query [CLS]）词元，为 ViT 剪枝提供文本引导。在剪枝过程中，我们进一步引入基于视觉-文本相关性的半结构化剪枝方案。具体而言，我们将空间上相邻的 2×2 词元分为一组，以适配主流 VLM 中普遍采用的视觉词元合并操作。我们使用 [Q-CLS] 词元与视觉词元之间的平均注意力分数作为每组的相关性指标，从而避免额外计算。随后依据相关性分数在组级别执行剪枝，保持位置连续性。剪枝之后，我们将冗余词元聚合为单个词元，以保留上下文线索。在近期的 Qwen3-VL 与 InternVL3 系列模型上，与现有后期剪枝方法相比，我们的方法最高可将预填充延迟降低 19.0%，同时精度提升 4.2%。
英文原文摘要 1610 字符
Vision-language models (VLMs) demonstrate powerful capabilities in multimodal tasks. However, the large number of visual tokens imposes a significant computational cost. In this paper, we propose QuietPrune, a QUery-guIded Early Token Pruning method to remove redundant visual tokens within VLMs, thereby enhancing computational efficiency. Unlike previous late pruning methods, we recognize that implementing early pruning within the vision transformer (ViT) can achieve benefits in both latency reduction and accuracy maintenance. To address the semantic loss problem in early pruning, we design a lightweight adapter by performing an inverse transformation of the projector in VLMs. The proposed adapter converts the contextual query into a visual domain [Q-CLS] (Query [CLS]) token, providing textual guidance for ViT pruning. During pruning, we further introduce a semi-structured pruning scheme based on visual-textual relevance. Specifically, we group spatially adjacent 2 x2 tokens to accommodate the visual token merging operation prevalent in mainstream VLMs. We use the mean attention scores between the [Q-CLS] token and the visual tokens as the relevance metric for each group, avoiding additional computation. Pruning is then applied at the group level based on the relevance score, preserving positional continuity. After pruning, we aggregate the redundant tokens into a single token to maintain context cues. Our method achieves up to 19.0% reduction in prefill latency while outperforming 4.2% in accuracy on the recent Qwen3-VL and InternVL3 series compared to existing late pruning methods.