先验的力量：基于LLaVA的免训练开放词表语义分割
The Power of Prior: Training-Free Open-Vocabulary Semantic Segmentation with LLaVA
利用LLaVA内部层产生的局部化特征作为免训练开放词表分割的像素级激活，并生成视觉提示引导SAM，为无需微调即可复用MLLM空间先验提供了可操作路径。
原文依据 certain layers within the LLM part of LLaVA can generate localized features corresponding to given object classes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论像LLaVA这样的多模态大语言模型（MLLMs）在多模态理解与生成方面已展现出显著能力。这一成功促使我们探究此类MLLMs中嵌入的固有先验知识是否具备足够的空间感知能力以用于密集预测任务，而无需任何任务特定的微调。因此，本文探索利用LLaVA进行免训练开放词表语义分割。我们发现LLaVA中LLM部分的某些层能够生成与给定目标类别相对应的局部化特征。基于这一内在能力，我们设计了三个模块：一个问答流程用于识别图像中的目标类别，一个文本-视觉响应模块用于提取目标类别初始可靠的像素级激活，以及一个视觉生成模块用于产生可靠的细化提示，这些提示进一步作为引导SAM生成预测的依据。我们基于LLaVA的方法在“Thing”类别数据集（如PASCAL VOC 2012和COCO-object）上取得了新的最优性能。此外，我们的方法不需要显式的背景类别名称，展示了其在处理开放世界场景方面的卓越潜力。代码见https://github.com/zbf1991/FSeg-LLaVA。
英文原文摘要 1319 字符
Multimodal Large Language Models (MLLMs) like LLaVA have demonstrated remarkable capabilities in multi-modal understanding and generation. This success motivates us to investigate whether the inherent prior knowledge embedded within such MLLMs contains sufficient spatial awareness for dense prediction tasks, without requiring any task-specific fine-tuning. Thus, in this paper, we explore the utilization of LLaVA for training-free open-vocabulary semantic segmentation. We discover that certain layers within the LLM part of LLaVA can generate localized features corresponding to given object classes. Building on this intrinsic capability, we design three modules: A question-answer pipeline to identify target classes in the image, a text-visual response module to extract initial reliable pixel-level activations for the target class, and a visual generation module to produce reliable refined prompts, which further serve as guidance for SAM to generate the predictions. Our LLaVA-based approach achieves new state-of-the-art performance on "Thing" category datasets, e.g., PASCAL VOC 2012 and COCO-object. Moreover, our method does not require explicit background class names, demonstrating its exceptional potential for handling open-world scenarios. Code is available at https://github.com/zbf1991/FSeg-LLaVA.