SAMTok：用两个词表示任意掩码
SAMTok: Representing Any Mask with Two Words
把任意区域掩码离散化为两个 token，使基础 MLLM 无需改架构、无需专用分割解码器即可用标准下一词预测和 RL 获得像素级理解与生成能力；做区域级 VLM、grounding 或交互式分割的工作可直接复用该掩码分词方案与文本答案匹配奖励。
原文依据 a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask from these tokens with high fidelity
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论像素级能力是构建交互式智能系统的关键。然而，像素级多模态大语言模型（MLLM）仍难以规模化，原因在于区域级编码器复杂、分割解码器专用、训练目标不兼容。为应对这些挑战，我们提出 SAMTok，一种离散掩码分词器，可将任意区域掩码转换为两个特殊 token，并能从这些 token 高保真地重建掩码。通过将掩码视为一种新的语言，SAMTok 使基础 MLLM（如 QwenVL 系列）能够借助标准的下一词预测和简单的强化学习习得像素级能力，而无需修改架构，也无需设计专用损失。SAMTok 基于 SAM2 构建，使用掩码编码器和残差向量量化器在 2.09 亿个多样掩码上训练，以产生离散、紧凑且信息丰富的 token。借助 500 万条 SAMTok 格式的掩码理解与生成数据样本，QwenVL-SAMTok 在区域描述、区域视觉问答、基于对话的 grounding、指代分割、场景图解析和多轮交互式分割上取得最优或可比的结果。我们进一步引入文本答案匹配奖励，使掩码生成的高效强化学习成为可能，在 GRES 和 GCG 基准上带来显著提升。我们的结果展示了一种可扩展且简洁的范式，用于为 MLLM 赋予强大的像素级能力。
英文原文摘要 1473 字符
Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask from these tokens with high fidelity. By treating masks as a new language, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities.