ScenDi：用于城市生成的三维到二维场景扩散级联
ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation
把 3D 潜扩散生成的高斯场景渲染成低分辨率图像，再作为条件喂给 2D 视频扩散做细节增强，这条「3D 粗生成 + 2D 精修」级联思路可直接迁移到需要相机可控性的街景/驾驶场景合成与仿真数据扩增。
原文依据 train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期基于扩散模型的三维物体生成取得了显著成功，但生成逼真的三维城市场景仍然具有挑战性。仅依赖三维扩散模型的现有方法往往出现外观细节退化，而仅使用二维扩散模型的方法通常会牺牲相机可控性。为克服这一局限，我们提出 ScenDi，一种融合三维与二维扩散模型的城市场景生成方法。我们首先训练一个三维潜变量扩散模型来生成三维高斯，从而能够以相对较低的分辨率渲染图像。为实现可控合成，该三维高斯生成过程可选择性地以三维边界框、道路地图或文本提示等输入作为条件。随后，我们训练一个二维视频扩散模型，以三维高斯渲染出的图像为条件来增强外观细节。通过利用粗糙的三维场景引导二维视频扩散，ScenDi 能够根据输入条件生成所需场景，并成功遵循准确的相机轨迹。在两个具有挑战性的真实世界数据集 Waymo 和 KITTI-360 上的实验证明了该方法的有效性。
英文原文摘要 1229 字符
Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach.