GroundingME：通过多维度评测揭示多模态大语言模型中的视觉定位差距
GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
对做视觉定位/VLM 的研究者，这篇提供了四维诊断基准（判别、空间、受限、拒绝），可直接用来暴露模型在相似目标区分、复杂关系描述和不可定位查询上的失效模式；其测试时扩展（按思维轨迹选最优回答）与数据混合训练两种改进策略也可迁移到其他 grounding 模型的评测与微调流程。
原文依据 the best model achieves only 45.1% accuracy, while most score 0% on rejection tasks
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉定位，即根据自然语言描述定位目标，是连接语言理解与视觉理解的关键桥梁。尽管多模态大语言模型（MLLMs）在现有基准上取得了令人瞩目的成绩，但一个根本性问题仍然存在：MLLMs 能否真正以接近人类的精细程度完成视觉定位，还是仅仅在简化数据集上进行模式匹配？现有基准无法刻画真实世界的复杂性——在真实场景中，人类可以轻松应对错综复杂的指代，并能识别出何时定位无法完成。为严格评估 MLLMs 的真实能力，我们提出 GroundingME，一个从四个关键维度系统挑战模型的基准：（1）判别性：区分高度相似的目标；（2）空间性：理解复杂的关系描述；（3）受限性：处理遮挡或微小目标；（4）拒绝：识别不可定位的查询。通过将自动生成与人工校验相结合的精心构建，我们创建了 1,005 个反映真实世界复杂性的挑战性样例。对 25 个最优 MLLMs 的评估揭示出显著的能力差距：最佳模型仅达到 45.1% 的准确率，而大多数模型在拒绝任务上得分为 0%。我们探索了两种改进策略：（1）测试时扩展通过思维轨迹选择最优回答，将整体性能最多提升 4.5%；（2）数据混合训练将拒绝准确率从 0% 提升至 27.9%。因此，GroundingME 既是一个揭示 MLLMs 当前局限性的诊断工具，也是迈向人类水平视觉定位的路线图。项目主页：https://groundingme.github.io
英文原文摘要 1658 字符
Visual grounding, localizing objects from natural language descriptions, represents a critical bridge between language and vision understanding. While multimodal large language models (MLLMs) achieve impressive scores on existing benchmarks, a fundamental question remains: can MLLMs truly visually ground with human-like sophistication, or are they merely pattern-matching on simplified datasets? Current benchmarks fail to capture real-world complexity where humans effortlessly navigate intricate references and recognize when grounding is impossible. To rigorously assess MLLMs' true capabilities, we introduce GroundingME, a benchmark that systematically challenges models across four critical dimensions: (1) Discriminative: distinguishing highly similar objects, (2) Spatial: understanding complex relational descriptions, (3) Limited: handling occlusions or tiny objects, and (4) Rejection: recognizing ungroundable queries. Through careful curation combining automated generation with human verification, we create 1,005 challenging examples mirroring real-world complexity. Evaluating 25 state-of-the-art MLLMs reveals a profound capability gap: the best model achieves only 45.1% accuracy, while most score 0% on rejection tasks. We explore two strategies for improvements: (1) test-time scaling selects optimal response by thinking trajectory to improve overall performance by up to 4.5%, and (2) data-mixture training boosts rejection accuracy from 0% to 27.9%. GroundingME thus serves as both a diagnostic tool revealing current limitations in MLLMs and a roadmap toward human-level visual grounding. Project page: https://groundingme.github.io