当词元剪枝不如随机剪枝：理解视觉语言模型中的视觉词元信息
When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
为视觉语言模型中的视觉词元剪枝提供了理论依据：深层视觉词元信息消失，随机剪枝即可，且可增强现有方法（如DivPrune），在OCR等视觉密集任务中需保留更深层词元。
原文依据 in deeper layers (e.g., beyond the 20th), existing training-free pruning methods perform no better than random pruning
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉大语言模型（VLLM）因依赖数百个视觉词元来表示图像而带来高昂的计算成本。词元剪枝为加速推理提供了一种有前景的解决方案，然而本文发现了一个关键现象：在较深层（例如第20层之后），现有的免训练剪枝方法的表现并不优于随机剪枝。我们假设这种性能退化是由“词元信息消失”导致的，即视觉词元随着网络深度增加而逐渐失去其显著性。为验证这一假设，我们通过测量移除某个词元时模型输出概率的变化来量化该词元的信息量。利用所提出的度量，我们对视觉词元信息跨层分析揭示了三个关键发现：（1）随着层数加深，视觉词元的信息逐渐趋于均匀，并最终在某个中间层消失，我们将该层称为“信息地平线”，超过该层后视觉词元变得冗余；（2）该地平线的位置并非固定不变；对于视觉密集型任务（如光学字符识别，OCR），它比更通用的任务（如视觉问答，VQA）延伸得更深；（3）该地平线还与模型容量强相关，因为更强的VLLM（如Qwen2.5-VL）比更弱的模型（如LLaVA-1.5）使用更深的视觉词元。基于这些发现，我们表明在深层进行简单的随机剪枝能有效平衡性能与效率。此外，将随机剪枝集成到现有方法中能持续提升其性能。使用DivPrune结合随机剪枝达到了最优结果，在剪枝50%视觉词元的同时保持了Qwen-2.5-VL-7B性能的96.9%。代码见 https://github.com/YahongWang1/Information-Horizon。
英文原文摘要 1844 字符
Vision Large Language Models (VLLMs) incur high computational costs due to their reliance on hundreds of visual tokens to represent images. While token pruning offers a promising solution for accelerating inference, this paper, however, identifies a key observation: in deeper layers (e.g., beyond the 20th), existing training-free pruning methods perform no better than random pruning. We hypothesize that this degradation is caused by "vanishing token information", where visual tokens progressively lose their salience with increasing network depth. To validate this hypothesis, we quantify a token's information content by measuring the change in the model output probabilities upon its removal. Using this proposed metric, our analysis of the information of visual tokens across layers reveals three key findings: (1) As layers deepen, the information of visual tokens gradually becomes uniform and eventually vanishes at an intermediate layer, which we term as "information horizon", beyond which the visual tokens become redundant;(2) The position of this horizon is not static; it extends deeper for visually intensive tasks, such as Optical Character Recognition (OCR), compared to more general tasks like Visual Question Answering (VQA);(3) This horizon is also strongly correlated with model capacity, as stronger VLLMs (e.g., Qwen2.5-VL) employ deeper visual tokens than weaker models (e.g., LLaVA-1.5).Based on our findings, we show that simple random pruning in deep layers efficiently balances performance and efficiency. Moreover, integrating random pruning consistently enhances existing methods. Using DivPrune with random pruning achieves state-of-the-art results, maintaining 96.9% of Qwen-2.5-VL-7B performance while pruning 50% of visual tokens. The code is available at https://github.com/YahongWang1/Information-Horizon.