为多模态大语言模型将一切定位到 token 中
Grounding Everything in Tokens for Multimodal Large Language Models
GETok 将二维空间定位转化为可学习的网格 token 与偏移 token 的序列预测，为 MLLM 在原生自回归框架下实现精确的 2D 视觉定位提供了可复用的 token 化方案，可直接用于指代检测、视觉定位等任务。
原文依据 GETok first uses grid tokens to partition the image plane into structured spatial anchors, and then exploits offset tokens to enable precise and iterative refinement of localization predictions.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在视觉理解与推理方面已取得显著进展。然而，MLLMs 所采用的自回归 Transformer 架构需要对输入图像进行 token 化，这限制了它们在二维图像空间中准确定位目标的能力。这引出一个重要问题：如何改进序列化的语言 token，使 MLLMs 能更好地在二维空间中对目标进行定位？为此，我们提出了一种用于目标定位的空间表征方法 GETok，它将一个专门的可学习 token 词汇表集成到 MLLMs 中。GETok 首先使用网格 token 将图像平面划分为结构化的空间锚点，然后利用偏移 token 实现对定位预测的精确且迭代的细化。通过将空间关系直接嵌入到 token 中，GETok 在不修改自回归架构的情况下显著提升了 MLLMs 在原生二维空间中的推理能力。大量实验表明，在有监督微调和强化学习设置下，GETok 在各种指代任务上均取得了优于最优方法的性能。
英文原文摘要 1186 字符
Multimodal large language models (MLLMs) have made significant advancements in vision understanding and reasoning. However, the autoregressive Transformer architecture used by MLLMs requires tokenization on input images, which limits their ability to accurately ground objects within the 2D image space. This raises an important question: how can sequential language tokens be improved to better ground objects in 2D spatial space for MLLMs? To address this, we present a spatial representation method for grounding objects, namely GETok, that integrates a specialized vocabulary of learnable tokens into MLLMs. GETok first uses grid tokens to partition the image plane into structured spatial anchors, and then exploits offset tokens to enable precise and iterative refinement of localization predictions. By embedding spatial relationships directly into tokens, GETok significantly advances MLLMs in native 2D space reasoning without modifying the autoregressive architecture. Extensive experiments demonstrate that GETok achieves superior performance over the state-of-the-art methods across various referring tasks in both supervised fine-tuning and reinforcement learning settings.