一层的垃圾是另一层的宝藏：LVLM 中的自适应逐层视觉词元选择
One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs
对做多模态大模型视觉侧推理加速的人，这篇给出了「按层动态选择视觉词元 + 被跳过词元在后续层重新汇入」的免训练方案，并用重要性一致性约束的低秩近似来逼近全注意力，可直接替换现有静态剪枝模块以缓解早期层信息丢失。
原文依据 once tokens are pruned at a specific layer, they become inaccessible to all subsequent layers, leading to premature information loss
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）在多种多模态任务上取得了显著成功，但其实际部署仍受制于冗长视觉词元带来的计算负担。尽管视觉词元剪枝已成为一种有前景的解决方案，现有方法却存在一个根本性局限：词元一旦在某一层被剪除，便无法被后续所有层访问，从而导致过早的信息丢失，可能损害模型性能。通过实证研究，我们观察到不同层关注不同的视觉区域，这表明各层的最优词元子集是变化的。受此启发，我们提出自适应逐层视觉词元选择（ALVTS），这一新框架摆脱了传统的静态词元剪枝范式。ALVTS 引入一个轻量级词元选择器，识别并路由重要词元进入后续处理，同时允许较不重要的词元跳过该层，从而将计算冗余降至最低。这两条词元流在送入后续层之前被无缝重新整合，实现整个模型上的自适应压缩。基于我们提出的重要性一致性约束低秩近似，所提出的词元选择模块紧密模拟全注意力机制，在无需重新训练模型的情况下有效捕捉其本质模式。在 LLaVA-1.5、LLaVA-NeXT 和 Qwen2.5-VL 上的大量实验验证了本方法的有效性。在 89% 的词元压缩率下，ALVTS 保留了原模型 96.7% 的准确率，为 LVLM 推理实现了更优的效率—精度权衡。
英文原文摘要 1712 字符
Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens. While visual token pruning has emerged as a promising solution, existing methods suffer from a fundamental limitation: once tokens are pruned at a specific layer, they become inaccessible to all subsequent layers, leading to premature information loss that can compromise model performance. Through empirical studies, we observe that different layers exhibit distinct visual region focus, indicating a varying optimal token subset across layers. Motivated by this insight, we propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm. ALVTS incorporates a lightweight token selector to identify and route important tokens for further processing, while allowing less important tokens to skip the layer, thus minimizing computational redundancy. These two streams of tokens are seamlessly reintegrated before being fed into subsequent layers, facilitating adaptive compression across the entire model. Grounded in our importance consistency constrained low-rank approximation, the proposed token selection module closely emulates the full attention mechanism, effectively capturing its essential patterns without requiring model retraining. Extensive experiments on LLaVA-1.5, LLaVA-NeXT, and Qwen2.5-VL validate the effectiveness of our method. With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.