抑制掩码图像建模表征中的非语义噪声
Suppressing Non-Semantic Noise in Masked Image Modeling Representations
对做 MIM 预训练的人，这篇给出了一个免训练、可插拔的线性头，把图像块表征里与语义无关的分量投影掉，从而直接提升下游零样本性能；其基于 PCA 的语义不变性打分也可作为评估表征语义纯度的通用诊断工具。
原文依据 MIM objectives cause the learned representations to retain non-semantic information, which ultimately hurts performance during inference
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论掩码图像建模（MIM）已成为一种广泛使用的自监督视觉范式。本文表明，MIM 目标会使学习到的表征保留非语义信息，这最终会损害推理阶段的性能。我们利用主成分分析（PCA）在真实与合成的非语义图像上引入了一个与模型无关的语义不变性评分。基于该评分，我们提出了一种简单方法——语义正交伪影投影（SOAP），直接在图像块表征中抑制非语义信息，从而在多种基于 MIM 的模型上带来一致的零样本性能提升。SOAP 是一种事后抑制方法，无需任何训练，可作为单个线性头附加到任意模型上。
英文原文摘要 780 字符
Masked Image Modeling (MIM) has become a ubiquitous self-supervised vision paradigm. In this work, we show that MIM objectives cause the learned representations to retain non-semantic information, which ultimately hurts performance during inference. We introduce a model-agnostic score for semantic invariance using Principal Component Analysis (PCA) on real and synthetic non-semantic images. Based on this score, we propose a simple method, Semantically Orthogonal Artifact Projection (SOAP), to directly suppress non-semantic information in patch representations, leading to consistent improvements in zero-shot performance across various MIM-based models. SOAP is a post-hoc suppression method, requires zero training, and can be attached to any model as a single linear head.