面向开放词表遥感图像分割的测试时多提示词自适应
Test-Time Multi-Prompt Adaptation for Open-Vocabulary Remote Sensing Image Segmentation
提出一种即插即用的测试时多提示词自适应方法，利用大语言模型生成上下文感知的类别描述，并通过视觉引导的测试时自适应策略和熵最小化像素级损失优化提示词表示，可灵活集成到现有开放词表分割方法中提升性能。
原文依据 we propose a plug-and-play yet effective Test-time Multi-Prompt Adaptation (TMPA) method to mitigate textual ambiguity in OVRSIS
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）的兴起推动了开放词表遥感图像语义分割（OVRSIS）的初步探索，使得在复杂对地观测场景中识别未见类别成为可能。然而，现有方法主要关注增强领域特定遥感图像的视觉表示，而忽略了文本信息的影响。本文认为，OVRSIS任务中存在一个关键的文本歧义问题，限制了最终的分割性能。因此，我们提出了一种即插即用且有效的测试时多提示词自适应（TMPA）方法，以缓解OVRSIS中的文本歧义。具体而言，TMPA首先通过执行大语言模型并输入任务驱动的提示词，为每个类别生成多样化的、上下文感知的描述，而非简单的类别名称，这能有效避免某些文本歧义，即背景类别在不同任务中具有不同含义。此外，TMPA针对生成的多提示词开发了一种视觉引导的测试时自适应策略，该策略利用高置信度的视觉特征，对具有高熵的不确定预测自适应地细化每个类别的提示词表示，使TMPA更好地适用于不同场景。特别地，提出了一种带有熵最小化的像素级损失，在推理过程中利用偏置优化文本提示词，其中提示词偏置基于高置信度视觉特征的加权组合构建。我们的TMPA可以灵活地集成到现有方法中，以提升其性能。在17个遥感数据集上进行了大量实验，结果表明我们的TMPA能显著改进其对应方法，同时达到最优性能。
英文原文摘要 1799 字符
The rise of vision-language models (VLMs) has driven the initial exploration of open-vocabulary remote sensing image semantic segmentation (OVRSIS), enabling recognition of unseen categories in complex Earth observation scenes. However, existing methods primarily focus on enhancing visual representations of domain-specific remote sensing images, while overlooking the effect of textual information. In this paper, we argue that there exists a crucial issue of textual ambiguity in OVRSIS task, limiting final segmentation performance. Therefore, we propose a plug-and-play yet effective Test-time Multi-Prompt Adaptation (TMPA) method to mitigate textual ambiguity in OVRSIS. Specifically, TMPA first generates diverse, context-aware descriptions for each category instead of the naive class name by executing a large language model with a task-driven prompt, which can effectively avoid some textual ambiguity, i.e., background class has different meanings in various tasks. Furthermore, TMPA develops a visual-guided test-time adaptation strategy for the generated multi-prompts, which adaptively refines the prompt representations of each category with high-confidence visual features for the uncertain predictions with high entropy, making TMPA better applicable to different scenarios. Particularly, a pixel-level loss with entropy minimization is proposed to optimize the text prompt with a bias during inference, where prompt bias is constructed based on a weighted combination of high-confidence visual features. Our TMPA can be flexibly integrated into existing methods for boosting their performance. Extensive experiments are conducted on 17 remote sensing datasets, and the results show our TMPA can significantly improve its counterparts, while achieving state-of-the-art performance.