通过宽基线匹配激发多模态大语言模型中的复杂空间推理
Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching
把宽基线匹配当作可验证奖励信号,用强化学习(DCRL)在无 CoT 监督下提升 MLLM 的几何与视角变化推理;其从 RGB-D 视频与 SfM 重建自动挖掘宽基线视图对的数据流水线,可直接复用于构建空间对应训练数据。
原文依据 we propose Dynamic Correspondence Reinforcement Learning (DCRL), which combines Image-Level Viewpoint Progression and Point-Level Correspondence Curriculum
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论宽基线匹配(WBM)要求综合几何理解、视角变化、细粒度感知与遮挡推理,因而成为评估部署于物理环境中的多模态大语言模型(MLLM)空间推理能力的严苛试验平台。然而,当前 MLLM 缺乏针对这些能力的系统性评测与训练框架。我们提出 ReasonMatch-Bench,这是一个按视角位移与匹配粒度分层、覆盖室内、室外与以物体为中心场景的基准,并表明当前 MLLM 在细粒度宽基线对应上仍存在困难:在一个包含 90 个样本的困难子集上,人类标注者达到 84.0 F1,而现有最佳基线仅为 37.2。为弥合这一差距,我们构建了一条可扩展的数据生成流水线,从大规模视频-三维语料(包括 RGB-D 视频与 SfM 重建)中自动提取宽基线视图对,从而获得多样且可验证的监督信号。我们进一步提出动态对应强化学习(DCRL),它结合图像级视角递进与点级对应课程,在无需显式思维链(CoT)监督的情况下,通过可验证奖励改进 WBM 训练。大量实验表明,DCRL 显著提升 ReasonMatch-Bench 上的表现,并可迁移至相关空间基准,同时在多个基准上保持通用视觉理解性能并取得小幅提升。
英文原文摘要 1457 字符
Wide-baseline matching (WBM) requires integrating geometric understanding, viewpoint changes, fine-grained perception, and occlusion reasoning, making it a challenging testbed for spatial reasoning in multimodal large language models (MLLMs) deployed in physical environments. However, current MLLMs lack systematic evaluation and training frameworks for these capabilities. We introduce ReasonMatch-Bench, a benchmark stratified by viewpoint displacement and matching granularity across indoor, outdoor, and object-centric scenarios, and show that current MLLMs still struggle with fine-grained wide-baseline correspondence: on a difficult 90-sample subset, human annotators achieve 84.0 F1, while the best existing baseline reaches 37.2. To bridge this gap, we build a scalable data-generation pipeline that automatically extracts wide-baseline view pairs from large-scale video-3D corpora, including RGB-D videos and SfM reconstructions, yielding diverse and verifiable supervision. We further propose Dynamic Correspondence Reinforcement Learning (DCRL), which combines Image-Level Viewpoint Progression and Point-Level Correspondence Curriculum to improve WBM training through verifiable rewards without explicit CoT supervision. Extensive experiments show that DCRL substantially improves ReasonMatch-Bench and transfers to related spatial benchmarks, while maintaining general visual understanding performance with modest gains on several benchmarks.