利用 CLIP 中的类别分布进行弱监督语义分割
Leveraging Class Distributions in CLIP for Weakly Supervised Semantic Segmentation
利用 CLIP 的 patch 级类别分布（JS 散度）生成更完整的 CAM，并用 DINO 的结构知识做边界超类原型监督，可直接用于弱监督分割的伪标签质量提升。
原文依据 The CDA module captures richer semantic relationships by modeling patch-wise distributions across all classes using Jensen-Shannon divergence
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论图像级弱监督语义分割（WSSS）通常利用类激活图（CAM）进行像素级定位。然而，现有的基于 CLIP 的方法往往产生激活不足的 CAM，主要原因是基于亲和力的细化中语义关系不准确。在本工作中，我们提出了一个新框架 CD-CLIP（基于类别分布的 CLIP），通过引入类别分布感知（CDA）模块来解决这一问题。CDA 模块利用 Jensen-Shannon 散度对所有类别进行 patch 级分布建模，从而捕获更丰富的语义关系，进而增强 CAM 的完整性。尽管这显著提高了前景类别的覆盖范围，但由于全面整合了目标间类别的关系，类别边界处也可能存在过度激活。为了减轻这种对分割监督的不利影响，我们引入了一个超类边界探索（SBE）模块，该模块利用 DINO 的结构知识生成边界感知的超类原型 CAM。通过采用边界增强损失，我们的 SBE 模块有效地为最终分割提供了准确的边界监督。我们提出的 CD-CLIP 框架在 PASCAL VOC 和 MS COCO 基准上均达到了最优性能。代码将公开。
英文原文摘要 1357 字符
Image-level Weakly Supervised Semantic Segmentation (WSSS) typically leverages Class Activation Maps (CAMs) for pixel-wise localization. However, existing CLIP-based methods often yield under-activated CAMs, primarily due to the inaccurate semantic relationships in the affinity-based refinement. In this work, we propose a novel framework, CD-CLIP (Class Distribution based CLIP), which addresses this issue by introducing a Class Distribution Aware (CDA) module. The CDA module captures richer semantic relationships by modeling patch-wise distributions across all classes using Jensen-Shannon divergence, thereby enhancing the completeness of CAMs. While this significantly improves the coverage of the foreground class, the over-activation at class boundaries might also exist due to the comprehensive integration of relationships between inter target classes. To mitigate this adverse effect on segmentation supervision, we introduce a Super-class Boundary Exploration (SBE) module, which leverages structural knowledge of DINO to generate boundary-aware super-class prototype CAMs. By employing the boundary-enhanced loss, our SBE module effectively provides accurate boundary supervision for the final segmentation. Our proposed CD-CLIP framework achieves state-of-the-art performance on both PASCAL VOC and MS COCO benchmarks. Code will be released.