IF-Prune：信息流引导的词元剪枝用于高效视觉语言模型
IF-Prune: Information-Flow Guided Token Pruning for Efficient Vision-Language Models
提出基于变分信息瓶颈的视觉词元剪枝方法，利用小型VLM估计词元熵作为剪枝指导，可复用于动态分辨率视觉编码器的高效推理。
原文依据 By adding a variational information bottleneck in the small VLM, we can approximate the entropy of each visual token as pruning guidance.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论采用动态分辨率视觉编码器的视觉语言模型（VLM）性能强大，但由于输入序列较长而面临显著的效率挑战。一种常见方法是评估词元的重要性并剪除信息量较少的词元。近期利用小型VLM提供视觉词元重要性图的方法在高剪枝率下优于现有的基于规则和相似性驱动的剪枝方法。然而，直接使用小型VLM仍不可靠，因为它使用聚合的视觉注意力权重作为重要性分数，当生成的词元不正确时可能导致噪声指导。为解决此问题，我们反转思路，让小型VLM根据用户输入查询检测非信息性视觉词元。通过在小型VLM中加入变分信息瓶颈，我们可以近似每个视觉词元的熵作为剪枝指导。这种后验引导的剪枝方法使大型VLM在提高效率的同时保持其推理能力。在八个基准上的大量实验证明了我们方法的有效性。仅保留5%的视觉词元时，大型VLM仍保持其原始性能的95%，比当前最优方法高出8%。
英文原文摘要 1290 字符
Vision-language models (VLMs) with dynamic resolution vision encoders achieve strong performance, but face significant efficiency challenges due to long input sequences. A common approach is to assess the importance of tokens and prune those that are less informative. Recent methods utilizing a small VLM to provide the importance map of visual tokens have outperformed existing rule-based and similarity-driven pruning approaches, particularly under high pruning ratios. However, directly using the small VLM remains unreliable, as it utilizes the aggregated visual attention weights as importance score, which can lead to noisy guidance if the generated tokens are incorrect.To address this, we invert the approach by having it detect non-informative visual tokens according to the user's input query. By adding a variational information bottleneck in the small VLM, we can approximate the entropy of each visual token as pruning guidance. Such a posteriori-guided pruning method allows the large VLM to retain its reasoning capacity with improved efficiency. Extensive experiments on eight benchmarks demonstrate the effectiveness of our approach. With only 5% of visual tokens retained, the large VLM preserves 95% of its original performance, outperforming the state of the art by 8%.