面向免训练开放词表语义分割的免 logits 优化直接分割方法
Direct Segmentation without Logits Optimization for Training-Free Open-Vocabulary Semantic Segmentation
把视觉-语言 logits 与真值的分布差异直接取解析解当作语义图，省掉迭代训练和模型特定的注意力调制，可直接套用到任意 CLIP 类骨干的免训练开放词表分割流程中。
原文依据 we directly utilize the analytic solution of this distribution discrepancy as the semantic maps
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论开放词表语义分割（OVSS）旨在利用开放词表提示对图像中的任意类别区域进行分割，这要求现有方法具备像素级的视觉-语言对齐能力。通常，该能力通过计算视觉特征与语言特征之间的余弦相似度（即 logits），并最小化 logits 与真值（GT）之间的分布差异来实现，从而生成最优 logits，随后用于构建分割图；然而，这一过程依赖于耗时的迭代训练或模型特定的注意力调制。在本工作中，我们提出一种更直接的方法，绕过 logits 优化过程，直接推导分割图的解析解。我们提出一个关键假设：分布差异本身编码了语义信息；具体而言，该差异在同类别图像块之间表现出一致性，而在不同类别之间表现出不一致性。基于该假设，我们直接将该分布差异的解析解用作语义图。换言之，我们将分布差异的优化重新表述为求取其解析解，从而消除了耗时的迭代训练，摆脱了对模型特定注意力调制的依赖，并在八个基准数据集上取得了最优性能。
英文原文摘要 1420 字符
Open-vocabulary semantic segmentation (OVSS) aims to segment arbitrary category regions in images using open-vocabulary prompts, necessitating that existing methods possess pixel-level vision-language alignment capability. Typically, this capability involves computing the cosine similarity, ie, logits, between visual and linguistic features, and minimizing the distribution discrepancy between the logits and the ground truth (GT) to generate optimal logits that are subsequently used to construct segmentation maps, yet it depends on time-consuming iterative training or model-specific attention modulation. In this work, we propose a more direct approach that eschews the logits-optimization process by directly deriving an analytic solution for the segmentation map. We posit a key hypothesis: the distribution discrepancy encodes semantic information; specifically, this discrepancy exhibits consistency across patches belonging to the same category but inconsistency across different categories. Based on this hypothesis, we directly utilize the analytic solution of this distribution discrepancy as the semantic maps. In other words, we reformulate the optimization of the distribution discrepancy as deriving its analytic solution, thereby eliminating time-consuming iterative training, freeing us from model-specific attention modulation, and achieving state-of-the-art performance on eight benchmark datasets.