SkySense-VITA：面向多模态遥感影像的通用上下文分割
SkySense-VITA: Towards Universal In-context Segmentation of Multi-modal Remote Sensing Imagery
论文提出的提示与预测解耦策略（VITA-Former 负责融合多模态提示，VITA-Decoder 负责预测）以及语义粒度退火（SGA）的由粗到细课程，可直接迁移到其他需要同时支持视觉提示、文本提示与融合提示的通用分割框架中。
原文依据 a novel prompt-and-prediction decoupling strategy, we propose the VITA-Former and VITA-Decoder to decouple multi-modal prompt fusion and prediction process
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管近期面向遥感分割的基础模型已取得显著进展，但它们在处理多样化的多模态输入、协同利用互补的提示词类型以及利用语义层级方面仍存在不足。为解决这些局限，我们提出 SkySense-VITA，一个统一的上下文分割模型，它借助视觉、文本或融合提示词，协同处理光学与合成孔径雷达（SAR）影像。基于一种新颖的提示与预测解耦策略，我们提出 VITA-Former 与 VITA-Decoder，将多模态提示词融合与预测过程解耦，使模型能够灵活支持纯视觉、纯文本以及融合提示词三种模式。我们采用渐进式两阶段策略训练 SkySense-VITA：第一阶段为图像级对齐预训练，包含光学-SAR 对齐；第二阶段为像素级上下文预训练，使用语义粒度退火（SGA），这是一种由粗到细的课程，可实现稳健的层级学习。为支撑该训练，我们引入了新的大规模多模态数据集 Sky-VT-300k。大量实验表明，SkySense-VITA 在 18 个数据集上建立了新的最优（SOTA）性能，平均性能领先超过 10% 的平均交并比（mIoU）。
英文原文摘要 1308 字符
While recent foundation models for remote sensing segmentation have shown notable progress, they still fall short in processing diverse multi-modal inputs, synergizing complementary prompt types, and leveraging semantic hierarchies. To address these limitations, we introduce SkySense-VITA, a unified in-context segmentation model, which synergistically processes both optical and Synthetic Aperture Radar (SAR) imagery using VIsual, TextuAl, or fused prompts. Based on a novel prompt-and-prediction decoupling strategy, we propose the VITA-Former and VITA-Decoder to decouple multi-modal prompt fusion and prediction process, allowing the model to flexibly support visual-only, textual-only, and fused prompt modes. We train SkySense-VITA with a progressive two-stage strategy: a first stage of Image-Level Alignment Pretraining featuring optical-SAR alignment, and a second stage of Pixel-Level In-context Pretraining using Semantic Granularity Annealing (SGA), a coarse-to-fine curriculum that enables robust hierarchical learning. To support this training, we introduce our new large-scale, multi-modal Sky-VT-300k dataset. Extensive experiments show SkySense-VITA establishes a new state-of-the-art (SOTA) on 18 datasets, with an average performance lead of over 10% mean Intersection over Union (mIoU).