视觉语言模型达标了吗？用 MeasureBench 对视觉测量读数进行基准评测
Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
提供了一个可程序化合成仪表图像（指针、刻度、字体、光照、杂乱可控）的流水线，以及一个诊断 VLM 细粒度空间定位能力的基准；做 VLM 数值感知或空间 grounding 的研究可直接用它构造训练/评测数据，并验证 RFT 在合成数据上的迁移效果。
原文依据 Our pipeline procedurally generates a specified type of gauge with controllable visual appearance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论读取测量仪器对人类而言毫不费力，且所需领域专业知识相对较少，但正如我们在初步评估中发现的，这对当前的视觉语言模型（VLM）而言仍然出人意料地困难。在本工作中，我们提出了 MeasureBench，一个覆盖真实世界图像与合成图像、涵盖多种测量类型的视觉测量读数基准，并附带一条可扩展的数据合成流水线。我们的流水线以程序化方式生成指定类型的仪表，其视觉外观可控，从而能够对指针、刻度、字体、光照和杂乱背景等关键细节进行可扩展的变化。对主流专有模型与开放权重 VLM 的评估表明，即便是最强的前沿 VLM 在测量读数上也普遍表现吃力。我们还利用合成数据进行了强化学习微调（RFT）的初步实验，发现其在域内合成子集和真实世界图像上均有显著提升。我们的分析凸显了当前 VLM 在细粒度空间定位方面的一项根本性局限。我们希望这一资源与我们的代码发布能够推动未来在 VLM 视觉接地数值推理与精确空间感知方面的进展，弥合识别数字与度量世界之间的差距。
英文原文摘要 1293 字符
Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle with measurement reading in general. We have also conducted preliminary experiments with reinforcement finetuning (RFT) over synthetic data, and find a significant improvement on both in-domain synthetic subset and real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource and our code releases can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world.