在 4D 中学习推理：面向视觉语言模型的动态空间理解
Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models
把相机位姿、局部点云、物体掩码与 3D 轨迹等 4D 重建先验压缩成少量 geometry tokens 注入 VLM，为在视觉语言模型中引入几何先验提供了一条轻量、可复用的接口思路。
原文依据 a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）在通用理解上表现优异，但在动态空间推理（DSR）方面仍然薄弱，即推理物体几何与关系在三维空间中随时间的演变，这在很大程度上源于可扩展的 4D 感知训练资源稀缺。为在数据集、基准与模型三个层面弥合这一差距，我们提出 DSR Suite。首先，我们提出一条自动化流水线，从真实场景视频中为 DSR 生成多项选择问答对。该流水线借助现代视觉基础模型提取丰富的几何与运动信息，包括相机位姿、局部点云、物体掩码、朝向以及 3D 轨迹。这些几何线索使得构建用于学习的 DSR-Train 以及经人工精修的 DSR-Bench 评测基准成为可能。与已有工作相比，我们的数据强调：（i）真实场景视频来源；（ii）物体级与场景级的三维要求；（iii）视角变换；（iv）多物体交互；（v）细粒度、过程化的答案。在数据之外，我们提出轻量的几何选择模块（GSM），将几何先验无缝集成到 VLM 中，该模块凝练问题语义，并从预训练的 4D 重建先验中提取与问题相关的知识，压缩为一组紧凑的几何 token。这种有针对性的提取避免了无关知识对模型造成过载。实验表明，将 DSR-Train 与 GSM 集成到 Qwen2.5-VL-7B 中，显著增强了其动态空间推理能力，同时保持在通用视频理解基准上的准确率。
英文原文摘要 1616 字符
Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.