超越窗口：全局-局部对齐CLIP用于免训练开放词表语义分割
Looking Beyond the Window: Global-Local Aligned CLIP for Training-free Open-Vocabulary Semantic Segmentation
提出全局-局部对齐的CLIP框架，通过跨窗口键值令牌交换和代理锚点解决滑窗推理中的语义不一致问题，可直接用于提升免训练开放词表分割方法的高分辨率处理能力。
原文依据 A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期免训练开放词表语义分割方法常采用滑窗推理策略，以克服CLIP在处理高分辨率图像时的局限性。然而，该方法引入新挑战：每个窗口独立处理，导致窗口间语义不一致。为解决此问题，我们提出全局-局部对齐CLIP（GLA-CLIP），一个促进跨窗口全面信息交换的框架。GLA-CLIP不将注意力局限于单个窗口内的令牌，而是扩展键值令牌以纳入所有窗口的上下文线索。然而，我们观察到窗口偏差：外部窗口令牌较少被关注，因为查询特征通过内部窗口块内的交互产生，从而缺乏局部上下文之外的语义定位。为缓解此问题，我们引入代理锚点，通过聚合所有窗口中与给定查询高度相似的令牌构建，为衡量内部和外部窗口块的相似性提供统一语义参考。此外，我们提出动态归一化方案，通过动态缩放和阈值化注意力图，根据对象尺度调整注意力强度，以应对小对象场景。此外，GLA-CLIP可装备于现有方法并扩大其感受野。大量实验验证了GLA-CLIP在提升免训练开放词表语义分割性能方面的有效性。代码见github.com/2btlFe/GLA-CLIP。
英文原文摘要 1615 字符
A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP (GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Codes are available at github.com/2btlFe/GLA-CLIP