HAWK：多模态模型中头重要性感知的视觉词元剪枝
HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
对做多模态视觉理解的人，这篇给出了一个免训练、即插即用的视觉词元剪枝方案：用注意力头重要性权重加权文本引导的注意力来打分并保留任务相关词元，可直接套到任意 MLLM 的视觉编码器到 LLM 的投影环节上，在剪掉 80% 视觉词元时仍保住 96% 精度。
原文依据 HAWK retains 96.0% of the original accuracy after pruning 80.2% of the visual tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在多模态大语言模型中，视觉词元数量的激增显著增加了推理时间与计算开销，使其难以用于实时或资源受限的应用。视觉词元剪枝通过去除冗余视觉词元来降低 MLLM 推理成本，是一种有前景的策略。已有研究通常假设所有注意力头对视觉解读的贡献相同。然而，本研究发现不同注意力头可能捕获不同的视觉语义，并在视觉处理中天然扮演不同角色。基于这一观察，我们提出 HAWK，一种头重要性感知的视觉词元剪枝方法，它感知注意力头在视觉任务中的不同重要性，从而最大化关键词元的保留。HAWK 利用头重要性权重与文本引导的注意力来评估视觉词元的重要性，在去除冗余词元的同时有效保留任务相关的视觉词元。所提出的 HAWK 完全免训练，可无缝应用于各类 MLLM。在多个主流视觉-语言基准上的大量实验表明，HAWK 取得了最优的准确率。应用于 Qwen2.5-VL 时，HAWK 在剪除 80.2% 的视觉词元后仍保留原始准确率的 96.0%。此外，它将端到端延迟降至原始的 74.4%，并在所测试的模型上进一步降低了 GPU 显存占用。
英文原文摘要 1457 字符
In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead, making them impractical for real-time or resource-constrained applications.Visual token pruning is a promising strategy for reducing the cost of MLLM inference by removing redundant visual tokens.Existing researches usually assume that all attention heads contribute equally to the visual interpretation.However, our study reveals that different heads may capture distinct visual semantics and inherently play distinct roles in visual processing.In light of this observation, we propose HAWK, a head importance-aware visual token pruning method that perceives the varying importance of attention heads in visual tasks to maximize the retention of crucial tokens.By leveraging head importance weights and text-guided attention to assess visual token significance, HAWK effectively retains task-relevant visual tokens while removing redundant ones.The proposed HAWK is entirely training-free and can be seamlessly applied to various MLLMs.Extensive experiments on multiple mainstream vision-language benchmarks demonstrate that HAWK achieves state-of-the-art accuracy. When applied to Qwen2.5-VL, HAWK retains 96.0% of the original accuracy after pruning 80.2% of the visual tokens. Additionally, it reduces end-to-end latency to 74.4% of the original and further decreases GPU memory usage across the tested models.