图像分割 · CVPR2026 ·2026.06.03

ReAttnCLIP:通过重新定义CLIP中的注意力实现免训练开放词表遥感图像分割

ReAttnCLIP: Training-Free Open-Vocabulary Remote Sensing Image Segmentation via Re-defined Attention in CLIP

Xin Niu、Manqi Zhao、Dongsheng Jiang、Yingying Wu、Bing Su

与你有关的部分

通过重新定义CLIP的注意力矩阵(替换patch-to-patch子矩阵、优先中间层注意力、禁用[CLS]自注意力)来提升开放词表分割的patch级表征质量,该思路可迁移到其他基于CLIP的密集预测任务。

原文依据 optimizing CLIP's 197x197 attention matrix through three key modifications
可迁移到 开放词表分割遥感图像分割建筑物提取道路提取

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

遥感图像分割对于自然灾害监测和精准农业等一系列应用至关重要。开放词表分割通过消除固定类别约束增强了灵活性,能够实现更细粒度、自适应的场景理解。与CLIP原始预训练目标强调全局图像-文本对齐不同,分割任务需要准确且具有判别性的块级表征来支持精确的像素级预测。因此,注意力图的质量——尤其是最终Transformer层生成的注意力图——在引导区域间交互方面起着关键作用。然而,现有方法在捕捉遥感中复杂的空间层次结构时生成的表征欠佳。我们通过三项关键修改优化CLIP的197x197注意力矩阵来填补这一空白:(1) 用中间层特征相似度替换196x196的块对块子矩阵以保留空间结构;(2) 优先使用中间层注意力进行全局到局部(类别到块)的词元对齐以减少分类干扰;(3) 禁用[CLS]词元的自注意力以减轻偏差。在八个遥感基准和两个建筑物/道路提取数据集上的大量实验表明,我们的方法在现有免训练方法中达到了最优性能。

英文原文摘要 1447 字符

Remote sensing image segmentation is critical for a range of applications, including natural disaster monitoring and precision agriculture. Open-vocabulary segmentation enhances flexibility by removing fixed category constraints, enabling more fine-grained and adaptive scene understanding. Unlike CLIP's original pretraining objective, which emphasizes global image-text alignment, segmentation tasks require accurate and discriminative patch-level representations to support precise pixel-wise predictions. As a result, the quality of attention maps--particularly those generated in the final transformer layers--plays a pivotal role in guiding inter-region interactions. However, current methods generate suboptimal representations when capturing the complex spatial hierarchies in remote sensing. We address this gap by optimizing CLIP's 197x197 attention matrix through three key modifications: (1) substituting the 196x196 patch-to-patch submatrix with intermediate-layer feature similarities to preserve spatial structures; (2) prioritizing intermediate-layer attention for global-to-local (class-to-patch) token alignment to reduce classification interference; (3) disabling the \texttt [CLS] token's self-attention to mitigate bias. Extensive experiments on eight remote sensing benchmarks and two building/road extraction datasets demonstrate that our method achieves state-of-the-art performance among existing training-free approaches.

标签

会议 CVPR2026
页码 24980-24989
Bibkey
来源 CVF Open Access