视觉 Transformer 需要的不仅是寄存器
Vision Transformers Need More Than Registers
论文指出 ViT 的 CLS 令牌会通过全局注意力把语义无关的背景图像块当作捷径来聚合全局语义，并给出一种选择性聚合图像块特征的方案；做视觉表征学习或下游微调时，可以直接替换 CLS 令牌的聚合方式，以抑制背景主导的伪影。
原文依据 ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉 Transformer（ViT）在大规模数据上预训练后，可为多种下游任务提供通用表征。然而，ViT 中的伪影在不同监督范式和下游任务中被广泛观察到。通过对 ViT 伪影的系统分析，我们发现其基本机制尚未得到充分阐明。本文通过系统分析得出结论：这些伪影源于一种惰性聚合行为——在全局注意力和粗粒度语义监督的驱动下，ViT 将语义无关的背景图像块用作捷径来表示全局语义。我们的解决方案是将图像块特征选择性地整合进 CLS 令牌，从而降低以背景为主导的捷径的影响，并在标签监督、文本监督和自监督三种设定下的 12 个基准上持续提升性能。我们希望这项工作能为理解 ViT 的行为提供新的视角。所有代码与权重将公开。
英文原文摘要 1009 字符
Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior. All the code and weights will be made publicly available.