重新审视 MLLM 自身作为分割器：仅用一个分割嵌入
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
对做分割的研究者，这篇给出了一个可复用的思路：把 MLLM 的压缩图像特征与原始分辨率残差特征融合，并用像素反洗牌放大细节，再通过双通路注意力掩码让单个分割 token 直接输出掩码，从而省掉 specialist mask decoder。
原文依据 unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期利用多模态大语言模型（MLLM）的分割方法已展现出可靠的目标级分割能力与增强的空间感知。然而，此前几乎所有方法都主要依赖专门的掩码解码器，从生成的分割相关嵌入与视觉特征中解读掩码，或引入多个额外 token 加以辅助。本文旨在探究能否以及如何仅用一个分割嵌入（SELF1E）从 MLLM 自身解锁分割能力，同时取得有竞争力的结果，从而无需外部解码器。为此，我们的方法针对 MLLM 像素洗牌图像特征中分辨率降低这一根本局限。首先，我们将图像特征保留在其原始未压缩分辨率，并用从 MLLM 处理后的压缩特征中提取的残差特征对其进行回填，从而提升特征精度。随后，我们分别对经 LLM 处理与未经 LLM 处理的图像特征施加像素反洗牌操作，以释放压缩特征的细节并放大未压缩分辨率下的残差特征，进一步提升回填特征的分辨率。此外，我们重新设计了注意力掩码，引入双感知通路，即图像到图像与图像到分割，使像素与分割 token 之间能够进行丰富的特征交互。在多个分割任务上的综合实验验证，SELF1E 取得了与基于专门掩码解码器的方法相竞争的性能，证明了在 MLLM 中实现无解码器分割的可行性。项目主页：https://github.com/ANDYZAQ/SELF1E。
英文原文摘要 1695 字符
Recent segmentation methods leveraging Multi-modal Large Language Models (MLLMs) have shown reliable object-level segmentation and enhanced spatial perception. However, almost all previous methods predominantly rely on specialist mask decoders to interpret masks from generated segmentation-related embeddings and visual features, or incorporate multiple additional tokens to assist. This paper aims to investigate whether and how we can unlock segmentation from MLLM itSELF with 1 segmentation Embedding (SELF1E) while achieving competitive results, which eliminates the need for external decoders. To this end, our approach targets the fundamental limitation of resolution reduction in pixel-shuffled image features from MLLMs. First, we retain image features at their original uncompressed resolution, and refill them with residual features extracted from MLLM-processed compressed features, thereby improving feature precision. Subsequently, we integrate pixel-unshuffle operations on image features with and without LLM processing, respectively, to unleash the details of compressed features and amplify the residual features under uncompressed resolution, which further enhances the resolution of refilled features. Moreover, we redesign the attention mask with dual perception pathways, i.e., image-to-image and image-to-segmentation, enabling rich feature interaction between pixels and the segmentation token. Comprehensive experiments across multiple segmentation tasks validate that SELF1E achieves performance competitive with specialist mask decoder-based methods, demonstrating the feasibility of decoder-free segmentation in MLLMs. Project page: https://github.com/ANDYZAQ/SELF1E.