为任意开源多模态大语言模型增强部件级点定位能力
Enhancing Part-Level Point Grounding for Any Open-Source MLLMs
提出一种冻结主干、仅靠中间层注意力合成文本条件查询并解码为点热图的插件式方案，可让任意开源 MLLM 获得部件级点定位能力，为细粒度机器人操作等任务提供可直接复用的适配模块。
原文依据 By synthesizing text-conditioned, grounding-aware queries within intermediate layers via the proposed Q-Synth Module
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉定位旨在将自由形式的文本查询与图像中的特定区域关联起来。尽管近期的多模态大语言模型（MLLMs）在该领域已展现出可观的能力，但它们主要在物体级定位上表现优异，而在部件级定位上往往力不从心——而部件级定位是机器人操作等细粒度任务的一项基本需求。在本工作中，我们提出一种通用方法，可为任意开源 MLLM 赋予准确的二维部件级点定位能力，为传统定位表示提供一种更直接的替代方案。我们的方法利用 MLLM 中固有的注意力机制。通过所提出的 Q-Synth 模块在中间层合成文本条件化、定位感知的查询，我们捕获与目标相关的注意力模式，并用一个轻量的注意力到点解码器对其进行精化，该解码器将这些模式转换为以点为中心的热图以用于最终预测。值得注意的是，所有原始 MLLM 参数均被冻结，从而确保其预训练能力得到完整保留。实验表明，我们的设计在不同数据集上一致地提升了部件级定位精度，并且可以无缝集成到任意开源 MLLM 中。
英文原文摘要 1237 字符
Visual grounding aims to associate free-form textual queries with specific regions in an image. While recent Multimodal Large Language Models (MLLMs) have demonstrated promising capabilities in this domain, they primarily excel at object-level grounding and often struggle with part-level grounding--an essential requirement for fine-grained tasks such as robotic manipulation. In this work, we introduce a general approach that equips any open-source MLLMs with accurate 2D part-level point grounding, offering a more direct alternative to conventional grounding representations. Our method leverages the attention mechanisms inherently present in MLLMs. By synthesizing text-conditioned, grounding-aware queries within intermediate layers via the proposed Q-Synth Module, we capture target-relevant attention patterns and refine them with a lightweight Attention-to-Point Decoder, which converts these patterns into a point-centric heatmap for final prediction. Notably, all original MLLM parameters are frozen, ensuring full preservation of their pre-trained capabilities. Experiments show that our design consistently improves part-level grounding accuracy across datasets and can be seamlessly integrated into any open-source MLLMs.