面向真实感文本到激光雷达场景生成的自条件表征引导扩散模型
A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation
对做 3D 感知的研究者，这篇提供了用扩散模型从文本生成激光雷达点云场景的完整方案：训练时用真实表征做软监督、推理时解耦的自条件引导，可缓解文本-点云配对稀缺导致的过度平滑；其条件编码器设计还能把稀疏到稠密、语义到激光雷达等任务统一到同一去噪网络中，可作为 3D 数据增广与跨模态条件生成的参考。
原文依据 SCRG, by aligning to the real representations, provides the soft supervision with reconstruction details for the Denoising Network (DN) in training, while decoupled in inference.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论文本到激光雷达生成能够为下游任务定制具有丰富结构和多样场景的三维数据。然而，文本-激光雷达配对数据的稀缺往往导致训练先验不足，生成的三维场景过于平滑。此外，低质量的文本描述可能降低生成质量与可控性。本文提出一种用于场景生成的文本到激光雷达扩散模型，命名为 T2LDM，并引入自条件表征引导（SCRG）。具体而言，SCRG 通过对齐真实表征，在训练中为去噪网络（DN）提供带有重建细节的软监督，而在推理阶段则将其解耦。通过这种方式，T2LDM 能够从数据分布中感知丰富的几何结构，在场景中生成细节丰富的物体。同时，我们构建了一个内容可组合的文本-激光雷达基准 T2nuScenes，并配套提出一个可控性指标。在此基础上，我们分析了不同文本提示词对激光雷达生成质量与可控性的影响，给出了实用的提示词范式与洞见。此外，我们设计了一种方向位置先验以缓解街道畸变，进一步提升场景保真度。另外，通过利用冻结的去噪网络学习条件编码器，T2LDM 可支持多种条件任务，包括稀疏到稠密、稠密到稀疏以及语义到激光雷达生成。在无条件与条件生成上的大量实验表明，T2LDM 优于现有方法，实现了最优的场景生成。
英文原文摘要 1535 字符
Text-to-LiDAR generation can customize 3D data with rich structures and diverse scenes for downstream tasks. However, the scarcity of Text-LiDAR pairs often causes insufficient training priors, generating overly smooth 3D scenes. Moreover, low-quality text descriptions may degrade generation quality and controllability. In this paper, we propose a Text-to-LiDAR Diffusion Model for scene generation, named T2LDM, with a Self-Conditioned Representation Guidance (SCRG). Specifically, SCRG, by aligning to the real representations, provides the soft supervision with reconstruction details for the Denoising Network (DN) in training, while decoupled in inference. In this way, T2LDM can perceive rich geometric structures from data distribution, generating detailed objects in scenes. Meanwhile, we construct a content-composable Text-LiDAR benchmark, T2nuScenes, along with a controllability metric. Based on this, we analyze the effects of different text prompts for LiDAR generation quality and controllability, providing practical prompt paradigms and insights. Furthermore, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, by learning a conditional encoder via frozen DN, T2LDM can support multiple conditional tasks, including Sparse-to-Dense, Dense-to-Sparse, and Semantic-to-LiDAR generation. Extensive experiments in unconditional and conditional generation demonstrate that T2LDM outperforms existing methods, achieving state-of-the-art scene generation.