ZOO-Prune：视觉语言模型中基于零阶梯度估计的免训练词元剪枝
ZOO-Prune: Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models
用零阶扰动在投影层估计每个视觉词元的敏感度，无需反向传播即可决定剪枝对象，可直接移植到任意 VLM 的视觉编码器—投影层之间做免训练加速。
原文依据 we estimate token sensitivity using zeroth-order perturbations at the lightweight projection layer
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（VLM）具备强大的多模态推理能力，但冗余的视觉词元带来了沉重的推理开销。词元剪枝可缓解该问题，然而现有方法仍存在局限。基于注意力的方法依赖原始注意力分数，而这些分数在不同层与不同注意力头之间往往不稳定，容易导致冗余选择。基于多样性的方法通过在特征空间中选择相距较远的词元来提升鲁棒性，但可能丢弃准确预测所需的区域。我们提出 ZOO-Prune，一个免训练框架，其出发点是：高敏感度的词元对模型输出的影响更强，并且捕获的是互补的视觉线索而非冗余信息。为此，我们在轻量投影层上利用零阶扰动估计词元敏感度，即度量小幅随机扰动如何影响投影后的特征，从而无需反向传播即可高效近似每个词元的影响力。在多个 VLM 与基准上的大量实验表明，ZOO-Prune 一致优于先前方法，在剪枝高达 94.4% 的词元时仍不损失准确率。该方法还提升了效率，相比基线端到端推理最高加速 2.30 倍。代码见 https://aim-skku.github.io/ZOO-Prune。
英文原文摘要 1348 字符
Large Vision-Language Models (VLMs) enable strong multimodal reasoning but incur heavy inference costs from redundant visual tokens. Token pruning alleviates this issue, yet existing approaches face limitations. Attention-based methods rely on raw attention scores, which are often unstable across layers and heads and can lead to redundant selections. Diversity-based methods improve robustness by selecting tokens far apart in feature space, but risk dropping regions needed for accurate prediction. We propose ZOO-Prune, a training-free framework built on the intuition that highly sensitive tokens have a stronger influence on the model's output and capture complementary visual cues rather than redundant ones. To achieve this, we estimate token sensitivity using zeroth-order perturbations at the lightweight projection layer. This measures how small random perturbations affect the projected features and enables efficient approximation of each token's influence without backpropagation. Extensive experiments across multiple VLMs and benchmarks show that ZOO-Prune consistently outperforms prior methods while pruning up to 94.4% of tokens without sacrificing accuracy. Our method also improves efficiency, reaching up to 2.30x faster end-to-end inference compared to the baseline. Code is available at https://aim-skku.github.io/ZOO-Prune.