CF-IPT：面向多源遥感数据分类的视觉语言预训练模型跨模态融合交互提示微调
CF-IPT: Cross-Modal Fusion Interactive Prompt Tuning of Vision-Language Pre-Trained Model for Multisource Remote Sensing Data Classification
把 CLIP 的提示学习从自然图像迁移到多源遥感：用光谱-空间提示矩阵引导视觉与文本分支的交互，仅训练 0.76% 参数即可完成多源图像分类，可作为跨模态提示微调在非自然图像域的可复用范式。
原文依据 leverages spectral-spatial prompt matrices to guide visual-textual prompt interaction and inject RS-specific information into both branches of CLIP
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在大规模自然图像-文本对数据集上训练的视觉语言模型（VLM），经微调后在多种下游任务上展现出令人印象深刻的性能。然而，将其微调用于遥感（RS）任务时面临双重障碍：（1）数据层面障碍，源于自然影像与遥感数据之间的根本性模态差异；（2）任务层面障碍，源于对多源交互建模能力的需求。本文提出一种跨模态融合交互提示微调（CF-IPT）方法，用于微调 CLIP 以完成多源遥感图像分类任务。该方法旨在借助提示学习框架，将文本分支的对齐目标从自然图像迁移至多源遥感图像。具体而言，我们设计了一个多源交互融合引导的光谱-空间提示生成（MFPG）模块，该模块实现跨模态特征交互，生成在保留原始光谱与空间信息的同时进行自适应多尺度融合的提示矩阵，以解决多源图像适配问题。随后，提出一种光谱-空间提示引导的视觉-文本提示交互（V-TPI）策略，利用光谱-空间提示矩阵引导视觉-文本提示交互，并将遥感特有信息注入 CLIP 的两个分支，最终实现多源遥感图像-文本表征对齐。所提方法仅使用 CLIP 参数的 0.76% 即可完成多源遥感图像分类的下游任务。在多个广泛使用的数据集上的评估结果验证了所提方法的有效性。
英文原文摘要 1685 字符
Fine-tuning Vision-Language Models (VLMs) trained on large-scale datasets of natural image-text pairs has demonstrated impressive performance for various downstream tasks. However, their fine-tuning for remote sensing (RS) tasks faces dual barriers: (1) Data-level barrier caused by the fundamental modality gap between natural imagery and RS data, and (2) Task-level barrier stemming from the requirement for multi-source interaction modeling capabilities. This paper proposes a Cross-modal Fusion Interactive Prompt Tuning (CF-IPT) method to fine-tune CLIP for multi-source RS image classification tasks. It aims to leverage the prompt learning framework to transfer the alignment target of the text branch shifts from natural images to multi-source RS images. Specifically, we design a Multi-source Interactive Fusion-guided Spectral-Spatial Prompt Generation (MFPG) module, which enables cross-modal feature interaction to generate a prompt matrix that preserves the original spectral and spatial information while performing adaptive multi-scale fusion to address the multi-source image adaptation problem. Subsequently, a Spectral-Spatial Prompt-guided Visual-Text Prompt Interaction (V-TPI) Strategy is proposed, which leverages spectral-spatial prompt matrices to guide visual-textual prompt interaction and inject RS-specific information into both branches of CLIP, ultimately enabling multi-source RS image-text representation alignment. The proposed approach performs the downstream task of multi-source RS image classification with merely 0.76% of CLIP's parameters. It is evaluated on several widely used datasets, demonstrating the effectiveness of the proposed approach.