GaussianDWM：面向统一场景理解与多模态生成的 3D 高斯驾驶世界模型
GaussianDWM: 3D Gaussian Driving World Model for Unified Scene Understanding and Multi-Modal Generation
把语言特征直接嵌入每个 3D 高斯原语实现早期模态对齐，并用任务感知的语言引导采样把冗余高斯裁剪成紧凑 3D token 送入 LLM——这套「高斯原语承载语义 + 可学习 token 剪枝」的做法可直接迁移到任何需要 3D 场景问答或 3D 条件生成的任务。
原文依据 we propose a novel unified DWM framework based on 3D Gaussian scene representation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着生成模型的进步，驾驶世界模型（DWM）正快速发展。然而，现有 DWM 缺乏 3D 场景理解能力，只能以输入数据为条件生成内容，无法解释或推理驾驶环境。此外，当前方法用点云或鸟瞰图特征表示 3D 空间信息，无法将文本信息与底层 3D 场景准确对齐。为解决这些局限，我们提出一种基于 3D 高斯场景表示的新型统一 DWM 框架，它同时支持 3D 场景理解与多模态场景生成，并为理解与生成任务提供上下文增强。我们的方法通过将丰富的语言特征嵌入每个高斯原语，直接把文本信息与 3D 场景对齐，从而实现早期模态对齐。此外，我们设计了一种新颖的任务感知语言引导采样策略，去除冗余的 3D 高斯，并向 LLM 注入准确而紧凑的 3D token。进一步地，我们设计了一个双条件多模态生成模型，将视觉语言模型捕获的信息作为高层语言条件，与低层图像条件相结合，共同引导多模态生成过程。我们在 nuScenes 和 NuInteract 数据集上开展了全面研究，以验证框架的有效性。我们的方法取得了最优性能。我们将于 GitHub 公开代码。
英文原文摘要 1572 字符
Driving World Models (DWMs) have been developing rapidly with the advances of generative models. However, existing DWMs lack 3D scene understanding capabilities and can only generate content conditioned on input data, without the ability to interpret or reason about the driving environment. Moreover, current approaches represent 3D spatial information with point cloud or BEV features do not accurately align textual information with the underlying 3D scene. To address these limitations, we propose a novel unified DWM framework based on 3D Gaussian scene representation, which enables both 3D scene understanding and multi-modal scene generation, while also enabling contextual enrichment for understanding and generation tasks. Our approach directly aligns textual information with the 3D scene by embedding rich linguistic features into each Gaussian primitive, thereby achieving early modality alignment. In addition, we design a novel task-aware language-guided sampling strategy that removes redundant 3D Gaussians and injects accurate and compact 3D tokens into LLM. Furthermore, we design a dual-condition multi-modal generation model, where the information captured by our vision-language model is leveraged as a high-level language condition in combination with a low-level image condition, jointly guiding the multi-modal generation process. We conduct comprehensive studies on the nuScenes, and NuInteract datasets to validate the effectiveness of our framework. Our method achieves state-of-the-art performance. We will release the code publicly on GitHub.