变化感知的视觉词元丢弃：加速大型视觉语言模型
Variation-aware Vision Token Dropping for Faster Large Vision-Language Models
对做 LVLM 推理加速的人，这篇给出了一个可直接插入 LLM 内部的视觉词元丢弃准则：按词元在层间的变化幅度（variation）而非注意力位置来排序剪枝，绕开了位置偏置与 FlashAttention 等高效算子的兼容问题，可复用到任意 LVLM 的 prefill 阶段。
原文依据 progressively removes visual tokens with minimal variation during LVLM inference
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）在多模态理解任务中已展现出卓越能力。然而，对高分辨率图像与长视频理解日益增长的需求带来了庞大的词元数量，进而导致推理效率下降。词元压缩通过减少待处理的词元数量提供了直接的解决方案，从而在不改动架构的前提下提升计算效率。通过大量分析，我们识别出现有 LLM 内部词元压缩方法的两个关键局限：位置偏置，以及与高效算子的不兼容性，这两点严重阻碍了它们在 LVLM 加速中的实际部署。本文首次从动态词元变化的角度切入，揭示出 LLM 内部视觉词元的变化具有任务无关的性质。我们提出变化感知的视觉词元丢弃方法（即 V2Drop），它在 LVLM 推理过程中逐步移除变化极小的视觉词元，从而提升计算效率。在多个模型与基准上的大量实验一致表明，V2Drop 在图像与视频理解任务上分别保持了原始性能的 94.0% 与 98.6%，同时将 LLM 生成延迟分别降低 31.5% 与 74.2%。
英文原文摘要 1335 字符
Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding tasks. However, the increasing demand for high-resolution image and long-video understanding results in substantial token counts, consequently leading to reduced inference efficiency. Token compression offers a direct solution by reducing the number of tokens to be processed, thereby improving computational efficiency without architectural changes. Through extensive analysis, we identify two critical limitations in existing inner-LLM token compression methods: positional bias and incompatibility with efficient operators, which critically hinder their practical deployment for LVLM acceleration. This paper presents the first approach from a dynamic token variation perspective, revealing that visual token variations within LLMs exhibit task-agnostic properties. We propose Variation-aware Vision Token Dropping (i.e., V2Drop), which progressively removes visual tokens with minimal variation during LVLM inference, thereby enhancing computational efficiency. Extensive experiments across multiple models and benchmarks consistently demonstrate that V2Drop maintains 94.0% and 98.6% of the original performance for image and video understanding tasks respectively, while reducing LLM generation latency by 31.5% and 74.2%.