在动态中思考：多模态大语言模型如何感知、跟踪与推理物理4D世界中的动态
Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
论文提出的 Dyn-Bench 基准和 ST-TCM 结构化融合方法，为视觉研究者提供了评估和提升 MLLMs 在动态 4D 场景中时空推理与动态目标定位能力的可复用工具与思路。
原文依据 we introduce Dyn-Bench, a large-scale benchmark designed to assess spatio-temporal reasoning and localized dynamics perception.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论人类栖居于一个物理4D世界中，空间几何与语义内容随时间演化，构成动态现实。当前多模态大语言模型（MLLMs）在理解静态视觉输入方面展现出强大能力，但其能否有效“在动态中思考”——即感知、跟踪并推理复杂场景中的时空演化——仍不清楚。为系统评估这些能力，我们提出Dyn-Bench，一个大规模基准，旨在评估时空推理与局部动态感知。通过对海量2D与4D数据源进行多阶段过滤构建，Dyn-Bench提供了高质量且多样的动态场景集合，包含1千段视频、7千个视觉问答（VQA）对以及3千个动态目标定位样本。我们全面研究了通用、空间感知与区域级MLLMs，从语言与视觉两个角度理解它们如何“在动态中思考”。结果表明，现有模型难以在时空推理与动态目标定位上同时表现出色，往往对运动与交互产生不一致的解释。传统提示策略（即思维链或基于描述的提示）仅带来有限提升。相比之下，结构化集成方法，包括掩码引导融合与时空文本认知地图（ST-TCM），显著增强了MLLMs在演化4D世界中的动态感知与时空推理能力。这些发现强调了显式时空结构线索对于弥合MLLMs静态感知与动态推理之间差距的重要性。
英文原文摘要 1733 字符
Humans inhabit a physical 4D world, where spatial geometry and semantic content evolve over time, forming a dynamic reality. While current Multimodal Large Language Models (MLLMs) demonstrate strong capabilities in understanding static visual inputs, it remains unclear whether they can effectively "think in dynamics," i.e., perceive, track, and reason about spatio-temporal evolution in complex scenes.To systematically evaluate these abilities, we introduce \texttt Dyn-Bench , a large-scale benchmark designed to assess spatio-temporal reasoning and localized dynamics perception. Constructed through multi-stage filtering over massive 2D and 4D data sources, \texttt Dyn-Bench provides a high-quality collection of diverse dynamic scenes, consisting of 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding samples.We comprehensively study general-purpose, spatial-aware, and region-level MLLMs to understand how they "think in dynamics" from both linguistic and visual perspectives. Our results reveal that existing models struggle to jointly excel in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Conventional prompting strategies i.e., chain-of-thought or caption-based hints) provide only limited improvements.In contrast, structured integration approaches, including Mask-Guided Fusion and the Spatio-Temporal Textual Cognitive Map (ST-TCM), substantially enhance MLLMs' dynamic perception and spatio-temporal reasoning in an evolving 4D world. These findings underscore the importance of explicit spatio-temporal structural cues to bridge the gap between static perception and dynamic reasoning in MLLMs.