在自监督Transformer中发现分布式目标中心属性
Finding Distributed Object-Centric Properties in Self-Supervised Transformers
提出一种无需训练的方法，通过聚类所有层的注意力头并利用patch级query/key/value相似度，提取分布式目标中心信息，可直接用于提升无监督目标发现和缓解多模态大语言模型的物体幻觉。
原文依据 Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论自监督视觉Transformer（ViT）如DINO展现出发现目标的涌现能力，通常通过最后一层的[CLS]词元注意力图观察到。然而，这些图常包含虚假激活，导致目标定位不佳。这是因为在图像级目标上训练的[CLS]词元概括了整个图像而非聚焦于目标。这种聚合稀释了存在于局部块级交互中的目标中心信息。我们通过计算所有层中块级注意力组件（查询、键和值）的块间相似度来分析此问题。我们发现：（1）目标中心属性编码在由所有三个组件（q、k、v）导出的相似度图中，而先前工作仅使用键特征或[CLS]词元。（2）这种目标中心信息分布在整个网络中，不仅限于最后一层。基于这些见解，我们提出Object-DINO，一种无需训练的方法，用于提取这种分布式目标中心信息。Object-DINO基于所有层中注意力头的块相似度对其进行聚类，并自动识别对应于所有目标的目标中心聚类。我们在两个应用上展示了Object-DINO的有效性：增强无监督目标发现（CorLoc提升+3.6至+12.4）以及通过提供视觉定位来缓解多模态大语言模型中的物体幻觉。我们的结果表明，使用这种分布式目标中心信息可在无需额外训练的情况下改善下游任务。
英文原文摘要 1644 字符
Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in \texttt [CLS] token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the \texttt [CLS] token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components (q, k, v), unlike prior work that uses only key features or the \texttt [CLS] token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object hallucination in Multimodal Large Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.