更好、更强、更快：以同步文本全掩码预测破解基于多模态大模型的分割三难困境
Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction
把分割掩码当作图像 patch 上的并行填空任务，在 MLLM 单次前向中一次性输出整张掩码，从而在不破坏对话能力的前提下兼顾分割质量与推理速度；做指代分割或多任务 VLM 的工作可直接复用这一解耦式输出头设计。
原文依据 STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论将分割能力集成到多模态大语言模型（MLLM）中面临一个核心三难困境：同时保持对话能力、取得高分割性能并确保快速推理。主流范式被迫做出妥协。嵌入预测方法引入了相互冲突的像素级目标，会削弱 MLLM 的通用对话能力。另一种方案即下一词元预测，把分割重新表述为自回归任务，虽能保持对话能力，却被迫在两种代价之间权衡：输出稀疏时分割性能差，输出丰富时推理速度又难以承受。我们以全掩码预测破解这一三难困境，这是一种将自回归对话生成与非自回归掩码预测解耦的新范式。我们提出 STAMP：同步文本全掩码预测，一个体现该范式的 MLLM。在生成文本回复之后，STAMP 将分割掩码视为图像块上的并行“填空”任务，在单次前向传播中预测出完整的分割掩码。该设计通过避免目标冲突保持了 MLLM 的对话能力，通过为所有掩码词元利用丰富的双向空间上下文实现了高分割性能，并达到了出色的速度。大量实验表明，STAMP 在多个分割基准上显著优于最优方法，提供了一个在对话、分割与速度上均不妥协的解决方案。
英文原文摘要 1514 字符
Integrating segmentation into Multimodal Large Language Models (MLLMs) presents a core trilemma: simultaneously preserving dialogue ability, achieving high segmentation performance, and ensuring fast inference. Prevailing paradigms are forced into a compromise. Embedding prediction methods introduce a conflicting pixel-level objective that degrades the MLLM's general dialogue abilities. The alternative, next-token prediction, reframes segmentation as an autoregressive task, which preserves dialogue but forces a trade-off between poor segmentation performance with sparse outputs or prohibitive inference speeds with rich ones. We resolve this trilemma with all-mask prediction, a novel paradigm that decouples autoregressive dialogue generation from non-autoregressive mask prediction. We present STAMP: Simultaneous Textual All-Mask Prediction, an MLLM that embodies this paradigm. After generating a textual response, STAMP predicts an entire segmentation mask in a single forward pass by treating it as a parallel "fill-in-the-blank" task over image patches. This design maintains the MLLM's dialogue ability by avoiding conflicting objectives, enables high segmentation performance by leveraging rich, bidirectional spatial context for all mask tokens, and achieves exceptional speed. Extensive experiments show that STAMP significantly outperforms state-of-the-art methods across multiple segmentation benchmarks, providing a solution that excels in dialogue, segmentation, and speed without compromise.