为开放词表稠密感知重构 CLIP
Reconstructing CLIP for Open-Vocabulary Dense Perception
对做开放词表稠密感知的人，这篇给出了如何从 CLIP 全局 cls token 的多层 value embedding 构造稠密特征、并用轻量门控模块按 head 异质性重加权以缓解空间聚合带来的语义错位的具体做法，可直接迁移到开放词表检测与分割的稠密-全局对齐流程中。
原文依据 we design a lightweight Head-Selective Gating (HSG) module that adaptively reweights feature heads according to their intrinsic heterogeneity
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模视觉语言模型（VLM）如 CLIP 在零样本图像分类上表现优异，却难以达到开放词表稠密感知（OVDP）所需的稠密跨模态对齐。近期自蒸馏方法通过将稠密特征与具有泛化性的全局语义对齐来缓解该问题，但一个关键问题仍然存在：应如何构造此类稠密特征才能实现最优对齐？为此，我们提出 DenseRC，一个基于两条关键洞见、为 OVDP 重构 CLIP 的原则性稠密表征构造框架。首先，通过分析全局 cls token 中编码的内部语义，我们发现多层 value embedding 可作为稠密特征的信息性基。其次，我们揭示空间聚合往往会放大语义错位。受此启发，我们设计了一个轻量的头选择门控（HSG）模块，依据特征头内在的异质性对其自适应重加权，从而构造出具有判别性且利于对齐的稠密表征。大量实验表明，DenseRC 在包括目标检测与语义分割在内的 OVDP 任务上带来一致且显著的增益，在多个基准上刷新了最优性能。
英文原文摘要 1316 字符
Large-scale vision-language models (VLMs) such as CLIP have excelled in zero-shot image classification, yet they struggle to achieve the dense cross-modal alignment required by open-vocabulary dense perception (OVDP). While recent self-distillation methods address this by aligning dense features with the generalizable global semantics, a key question remains: how should such dense features be constructed to achieve optimal alignment? To address this, we propose DenseRC, a principled Dense Representations Construction framework that reconstructs CLIP for OVDP based on two key insights.First, by analyzing the internal semantics encoded in the global cls token, we identify that multi-layer value embeddings serve as an informative basis for dense features. Second, we reveal that spatial aggregation tends to amplify semantic misalignment. Motivated by this, we design a lightweight Head-Selective Gating (HSG) module that adaptively reweights feature heads according to their intrinsic heterogeneity, enabling discriminative and alignment-friendly dense representations construction. Extensive experiments demonstrate that DenseRC delivers consistent and substantial gains across OVDP tasks including object detection and semantic segmentation, setting new state-of-the-art performance on multiple benchmarks.