VLM-3R：以指令对齐三维重建增强的视觉语言模型
VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
把单目视频帧经几何编码器转成隐式 3D 空间 token 与相机运动 view token，再与语言模型对齐，为视觉研究者提供了一条无需外部深度传感器或预建 3D 地图即可让 VLM 获得空间与时序推理能力的可复用路径。
原文依据 a geometry encoder that derives implicit 3D tokens representing scene context (spatial tokens) and camera motion (view tokens)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论面向二维图像与视频的大型多模态模型（LMM）的快速发展，激发了将这些模型扩展到三维场景的兴趣，目标是实现类人的视觉-空间智能。然而，无论对模型设计还是数据获取而言，达到可与人类能力相比的深层空间理解仍然具有挑战性。现有方法往往依赖外部深度传感器来捕获几何信息，或依赖现成算法预先构建三维地图，这限制了其可扩展性。在本工作中，我们提出 VLM-3R，这是一个面向视觉语言模型的框架，将三维重建指令微调与可扩展的训练数据构建以及一个新的时序推理基准相结合。具体而言，VLM-3R 处理单目视频帧，借助几何编码器导出隐式三维 token，分别表示场景上下文（空间 token）与相机运动（视角 token）。与此同时，我们构建了一条可扩展的数据创建流程，产出超过 20 万条三维重建指令微调问答对。为评估时序推理，我们进一步提出视觉-空间-时序智能基准（VSTI-Bench），其中包含超过 13.86 万条问答对，覆盖五个聚焦于演化中的空间关系的不同任务。大量实验表明，VLM-3R 支持稳健的视觉-空间推理，并提升了对时序三维上下文变化的理解，从而支持单目三维空间辅助与具身推理。
英文原文摘要 1473 字符
The rapid advancement of Large Multimodal Models (LMMs) for 2D images and videos has sparked interest in extending these models to 3D scenes, with the goal of human-like visual-spatial intelligence. However, achieving deep spatial understanding comparable to human capabilities remains challenging for both model design and data acquisition. Existing methods often rely on external depth sensors for geometry capture or off-the-shelf algorithms for pre-constructing 3D maps, which limits their scalability.In this work, we introduce VLM-3R, a framework for Vision-Language Models that couples 3D reconstructive instruction tuning with scalable training data curation and a new benchmark for temporal reasoning. Specifically, VLM-3R processes monocular video frames with a geometry encoder that derives implicit 3D tokens representing scene context (spatial tokens) and camera motion (view tokens). In parallel, we build a scalable data creation pipeline with over 200K 3D reconstructive instruction-tuning question-answer pairs. To evaluate temporal reasoning, we further introduce the Vision-Spatial-Temporal Intelligence benchmark (VSTI-Bench), which contains over 138.6K question-answer pairs across five distinct tasks focused on evolving spatial relationships. Extensive experiments show that VLM-3R supports robust visual-spatial reasoning and improves the understanding of temporal 3D context changes, enabling monocular 3D spatial assistance and embodied reasoning.