DrivePI：面向统一自动驾驶理解、感知、预测与规划的空间感知 4D 多模态大语言模型
DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning
DrivePI 将点云、多视角图像与语言指令统一到单个 MLLM 中，并行输出 3D 占用、占用流与规划动作，为视觉研究者提供了一个可复用的端到端 4D 空间理解与预测框架，尤其适合探索小参数量 MLLM 在自动驾驶感知-预测-规划联合建模中的潜力。
原文依据 DrivePI, a novel spatial-aware 4D MLLM that serves as a unified Vision-Language-Action (VLA) framework
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLMs）在多个领域展现出强大能力，但其在自动驾驶中生成细粒度 3D 感知与预测输出方面的应用仍未被充分探索。本文提出 DrivePI，一种新颖的空间感知 4D MLLM，它作为统一的视觉-语言-动作（VLA）框架，同时也兼容视觉-动作（VA）模型。我们的方法通过端到端优化并行联合执行空间理解、3D 感知（即 3D 占用）、预测（即占用流）与规划（即动作输出）。为同时获得精确的几何信息与丰富的视觉外观，我们的方法在统一的 MLLM 架构中整合了点云、多视角图像与语言指令。我们进一步开发了一个数据引擎，用于生成面向 4D 空间理解的文本-占用与文本-流问答对。值得注意的是，仅以 0.5B 的 Qwen2.5 模型作为 MLLM 主干，DrivePI 作为单一统一模型即可匹配或超越现有的 VLA 模型与专用 VA 模型。具体而言，与 VLA 模型相比，DrivePI 在 nuScenes-QA 上以 2.5% 的平均准确率优势超越 OpenDriveVLA-7B，并在 nuScenes 上将碰撞率较 ORION 降低 70%（从 0.37% 降至 0.11%）。与专用 VA 模型相比，DrivePI 在 OpenOcc 上的 3D 占用任务中以 10.3 RayIoU 超越 FB-OCC，在 OpenOcc 上的占用流任务中将 mAVE 从 0.591 降至 0.509，并在 nuScenes 的规划任务中实现较 VAD 低 32% 的 L2 误差（从 0.72m 降至 0.49m）。代码将发布于 https://github.com/happinesslz/DrivePI。
英文原文摘要 1605 字符
Although multi-modal large language models (MLLMs) have shown strong capabilities across diverse domains, their application in generating fine-grained 3D perception and prediction outputs in autonomous driving remains underexplored. In this paper, we propose DrivePI, a novel spatial-aware 4D MLLM that serves as a unified Vision-Language-Action (VLA) framework that is also compatible with vision-action (VA) models. Our method jointly performs spatial understanding, 3D perception (i.e., 3D occupancy), prediction (i.e., occupancy flow), and planning (i.e., action outputs) in parallel through end-to-end optimization. To obtain both precise geometric information and rich visual appearance, our approach integrates point clouds, multi-view images, and language instructions within a unified MLLM architecture. We further develop a data engine to generate text-occupancy and text-flow QA pairs for 4D spatial understanding. Remarkably, with only a 0.5B Qwen2.5 model as MLLM backbone, DrivePI as a single unified model matches or exceeds both existing VLA models and specialized VA models. Specifically, compared to VLA models, DrivePI outperforms OpenDriveVLA-7B by 2.5% mean accuracy on nuScenes-QA and reduces collision rate by 70% over ORION (from 0.37% to 0.11%) on nuScenes. Against specialized VA models, DrivePI surpasses FB-OCC by 10.3 RayIoU for 3D occupancy on OpenOcc, reduces the mAVE from 0.591 to 0.509 for occupancy flow on OpenOcc, and achieves 32% lower L2 error than VAD (from 0.72m to 0.49m) for planning on nuScenes. Code will be available at https://github.com/happinesslz/DrivePI.