面向视觉 Transformer 的显著性驱动词元合并
Saliency-Driven Token Merging for Vision Transformers
提供了一种免训练的词元合并策略：利用输入图块的潜在视觉显著性统计量判断哪些词元与全局梯度方向不一致，从而安全合并，可直接插入任意 ViT 主干以降低浮点运算量，无需微调。
原文依据 we propose a novel training-free SAliency-Driven Token Merging (SAD-TM) approach
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉 Transformer（ViT）在多种视觉场景中表现出稳健的性能。然而，其效率受制于过多的词元数量。词元合并为实现高效 ViT 提供了一种可行方案。现有方法仅依据注意力机制内部的特定特征来合并词元，而这些特征在不同层之间变化显著。本文提出一种新颖的免训练显著性驱动词元合并方法（SAD-TM），它不仅利用注意力空间中的语义相关性，还利用输入图块的潜在视觉显著性。SAD-TM 的灵感来源于一项发现：基于显著性的统计量能够直接刻画模型输入与输出之间的因果关系，且与层无关。基于这一观察，我们开发了一种在数学上形式化的方法，用于合并具有高显著性离群值的词元。我们的合并原理是：具有高显著性离群值的词元通常意味着其与全局梯度方向不一致，因而可以被安全地合并。此外，我们的系统分析表明，类别注意力在早期模块中表现出相当大的变化，因此引入了延迟合并策略以优化合并率的选择。SAD-TM 以免训练的方式在多种 ViT 架构上展现出优越性能。特别是在 DeiT-Tiny 上实现 23.08% 的浮点运算量压缩时，SAD-TM 在 ImageNet 数据集上取得了与预训练基线相当的 Top-1 准确率。代码即将发布。
英文原文摘要 1551 字符
Vision Transformers (ViTs) exhibit robust performance across diverse visual scenarios. However, their efficiency is constrained by excessive token counts. Token merging offers a viable solution for achieving efficient ViTs. Existing methods merge tokens based solely on specific characteristics within the attention mechanism, which changes significantly across layers. In this paper, we propose a novel training-free SAliency-Driven Token Merging (SAD-TM) approach by leveraging not only the semantic relevance in the attention space but also the latent visual saliency of input patches. Our SAD-TM is inspired by the discovery that saliency-based statistics can directly capture the causal relationship between model input and output, regardless of the layers. Based on the observation, we develop a method that is mathematically formulated to merge tokens with high saliency outliers. The principle behind our merging is that tokens with high saliency outliers usually imply inconsistencies with the global gradient direction, and thus can be merged safely. Besides, our systematic analysis indicates that class attention shows considerable variation across early blocks, so a deferred merging strategy is introduced to optimize the selection of merging rates. In a training-free manner, SAD-TM demonstrates superior performance across various ViT architectures. Especially, with a FLOPs compression of 23.08% on DeiT-Tiny, SAD-TM achieves a Top-1 Accuracy comparable to the pretrained baseline on ImageNet dataset. The code will be available soon.