VGGDrive：为自动驾驶赋予视觉语言模型跨视角几何定位能力
VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving
将冻结的3D基础模型的跨视角几何特征通过分层自适应注入机制融入VLM的2D视觉特征，为视觉语言模型提供可复用的3D几何感知增强方案。
原文依据 we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论跨视角3D几何建模能力对于自动驾驶的重要性不言而喻，然而现有的视觉语言模型（VLM）本质上缺乏这一能力，导致其性能表现平平。尽管一些有前景的方法试图通过构建问答数据辅助训练来缓解这一问题，但它们仍未能从根本上赋予VLM全面处理多样化评估协议的能力。因此，我们开辟了一条新路径，倡导将成熟3D基础模型的跨视角几何定位能力注入VLM，以弥合自动驾驶中这一关键的能力差距。在此理念下，我们提出了一种新颖的架构VGGDrive，它为视觉语言模型赋予了面向自动驾驶的跨视角几何定位能力。具体而言，为了将来自冻结视觉3D模型的跨视角3D几何特征与VLM的2D视觉特征相桥接，我们引入了一个即插即用的跨视角3D几何赋能器（CVGE）。CVGE解耦了基础VLM架构，并通过分层自适应注入机制有效地为VLM赋予了3D特征。大量实验表明，VGGDrive在五个自动驾驶基准上提升了基础VLM的性能，涵盖跨视角风险感知、运动预测和轨迹规划等任务。我们相信，成熟的3D基础模型能够通过有效集成来赋能自动驾驶任务，并希望我们的初步探索能向自动驾驶社区展示这一范式的潜力。
英文原文摘要 1582 字符
The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.