ReAttnCLIP:通过重新定义CLIP中的注意力实现免训练开放词表遥感图像分割
ReAttnCLIP: Training-Free Open-Vocabulary Remote Sensing Image Segmentation via Re-defined Attention in CLIP
通过重新定义CLIP的注意力矩阵(替换patch-to-patch子矩阵、优先中间层注意力、禁用[CLS]自注意力)来提升开放词表分割的patch级表征质量,该思路可迁移到其他基于CLIP的密集预测任务。
原文依据 optimizing CLIP's 197x197 attention matrix through three key modifications
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论遥感图像分割对于自然灾害监测和精准农业等一系列应用至关重要。开放词表分割通过消除固定类别约束增强了灵活性,能够实现更细粒度、自适应的场景理解。与CLIP原始预训练目标强调全局图像-文本对齐不同,分割任务需要准确且具有判别性的块级表征来支持精确的像素级预测。因此,注意力图的质量——尤其是最终Transformer层生成的注意力图——在引导区域间交互方面起着关键作用。然而,现有方法在捕捉遥感中复杂的空间层次结构时生成的表征欠佳。我们通过三项关键修改优化CLIP的197x197注意力矩阵来填补这一空白:(1) 用中间层特征相似度替换196x196的块对块子矩阵以保留空间结构;(2) 优先使用中间层注意力进行全局到局部(类别到块)的词元对齐以减少分类干扰;(3) 禁用[CLS]词元的自注意力以减轻偏差。在八个遥感基准和两个建筑物/道路提取数据集上的大量实验表明,我们的方法在现有免训练方法中达到了最优性能。
英文原文摘要 1447 字符
Remote sensing image segmentation is critical for a range of applications, including natural disaster monitoring and precision agriculture. Open-vocabulary segmentation enhances flexibility by removing fixed category constraints, enabling more fine-grained and adaptive scene understanding. Unlike CLIP's original pretraining objective, which emphasizes global image-text alignment, segmentation tasks require accurate and discriminative patch-level representations to support precise pixel-wise predictions. As a result, the quality of attention maps--particularly those generated in the final transformer layers--plays a pivotal role in guiding inter-region interactions. However, current methods generate suboptimal representations when capturing the complex spatial hierarchies in remote sensing. We address this gap by optimizing CLIP's 197x197 attention matrix through three key modifications: (1) substituting the 196x196 patch-to-patch submatrix with intermediate-layer feature similarities to preserve spatial structures; (2) prioritizing intermediate-layer attention for global-to-local (class-to-patch) token alignment to reduce classification interference; (3) disabling the \texttt [CLS] token's self-attention to mitigate bias. Extensive experiments on eight remote sensing benchmarks and two building/road extraction datasets demonstrate that our method achieves state-of-the-art performance among existing training-free approaches.