QuantiPhy：评估视觉语言模型物理推理能力的定量基准
QUANTIPHY: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models
该基准把 VLM 的物理理解从定性 VQA 推进到对尺寸、速度、加速度的定量数值估计，并揭示模型依赖预训练世界知识而非忠实使用视觉输入；做视频理解或多模态推理的研究者可用它诊断模型是否真正从像素中推断运动学量，也可借鉴其标准化提示与评分协议来构建数值化评测。
原文依据 state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论理解物理世界是通用人工智能体的必备能力。然而，目前尚不清楚最优的视觉感知模型（例如大型视觉语言模型）能否完成定量的物理推理任务。现有评测大多基于视觉问答且为定性评测，难以揭示这些模型能否从视频中推断运动物体的运动学量。为此，我们提出 QuantiPhy，这是首个旨在定量衡量视觉语言模型物理推理能力的基准。QuantiPhy 包含超过 3.3K 个带数值真值的视频-文本实例，评估视觉语言模型在给定某一属性作为输入先验的条件下，估计某一时刻物体尺寸、速度和加速度的表现。该基准对提示与评分进行标准化，以评估数值精度，从而实现模型间的公平比较。我们在最优视觉语言模型上的实验表明，其定性合理性与实际数值正确性之间存在一致的差距。我们进一步深入分析了背景噪声、反事实先验和策略性提示等关键因素，发现最优视觉语言模型在推理物体运动学属性时严重依赖预训练的世界知识，而非忠实利用所提供的视觉与文本输入。QuantiPhy 提供了首个严谨、可扩展的测试平台，推动视觉语言模型超越单纯的语言合理性，走向以数值为基础的物理理解。
英文原文摘要 1512 字符
Understanding the physical world is essential for generalist AI agents. However, it remains unclear whether state-of-the-art vision perception models (e.g., large VLMs) can perform quantitative physical reasoning tasks. Existing evaluations are predominantly VQA-based and qualitative, offering limited insight into whether these models can infer the kinematic quantities of moving objects from videos. To address this, we present QuantiPhy, the first benchmark designed to quantitatively measure a VLM's physical reasoning ability. Comprising more than 3.3K video-text instances with numerical ground truth, QuantiPhy evaluates a VLM's performance on estimating an object's size, velocity, and acceleration at a given timestamp, using one of these properties as an input prior. The benchmark standardizes prompts and scoring to assess numerical accuracy, enabling fair comparisons across models. Our experiments on state-of-the-art VLMs reveal a consistent gap between their qualitative plausibility and actual numerical correctness. We further provide an in-depth analysis of key factors like background noise, counterfactual priors, and strategic prompting and find that state-of-the-art VLMs lean heavily on pre-trained world knowledge rather than faithfully using the provided visual and textual inputs when reasoning about objects' kinematic properties. QuantiPhy offers the first rigorous, scalable testbed to move VLMs beyond mere verbal plausibility toward a numerically grounded physical understanding.