Pano3DComposer：从单张全景图像前馈式组合生成三维场景
Pano3DComposer: Feed-Forward Compositional 3D Scene Generation from Single Panoramic Image
把现成的 image-to-3D 模型生成的局部坐标物体，通过改造 VGGT 的前馈变换预测器直接放到世界坐标，绕开迭代布局优化；做单图/全景场景合成或需要把生成物体摆进场景的工作可以复用这个「物体-世界变换预测 + 渲染反馈由粗到细对齐」的范式。
原文依据 we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论当前的组合式图像到三维场景生成方法，要么通过耗时的迭代布局优化来构建三维场景，要么采用缺乏灵活性的物体与布局联合生成方式。此外，大多数方法依赖视场受限的透视图像，难以构建完整的 360° 环境。为解决这些局限，我们设计了 Pano3DComposer，一个面向全景图像的高效前馈框架。为将物体生成与布局估计解耦，我们提出即插即用的物体-世界变换预测器。该模块将现成图像到三维模型生成的三维物体从局部坐标转换到世界坐标。为此，我们通过使用目标物体裁剪图、多视角物体渲染图和相机参数来预测该变换，将 VGGT 架构改造为 Alignment-VGGT。该预测器使用伪几何监督进行训练，以应对生成物体与真值物体之间的形状差异。对于来自未见领域的输入图像，我们进一步为 Pano3DComposer 引入由粗到细（C2F）对齐机制，借助场景渲染的反馈迭代精化几何一致性。在合成与真实世界数据集上，我们的方法在图像/文本到三维任务中取得了更优的几何精度。它可以在 RTX 4090 GPU 上约 20 秒内生成高保真三维场景。若论文被接收，代码将公开。
英文原文摘要 1413 字符
Current compositional image-to-3D scene generation approaches construct 3D scenes by time-consuming iterative layout optimization or inflexible joint object-layout generation. Moreover, most methods rely on limited field-of-view perspective images, hindering the creation of complete 360^\circ environments. To address these limitations, we design Pano3DComposer, an efficient feed-forward framework for panoramic images. To decouple object generation from layout estimation, we propose a plug-and-play Object-World Transformation Predictor. This module converts the 3D objects generated by off-the-shelf image-to-3D models from local to world coordinates. To achieve this, we adapt the VGGT architecture to Alignment-VGGT by using target object crop, multi-view object renderings and camera parameters to predict the transformation. The predictor is trained using pseudo-geometric supervision to address the shape discrepancy between generated and ground-truth objects. For input images from unseen domains, we further introduce a Coarse-to-Fine (C2F) alignment mechanism for Pano3DComposer that iteratively refines geometric consistency with feedback of scene rendering. Our method achieves superior geometric accuracy for image/text-to-3D tasks on synthetic and real-world datasets. It can generate a high-fidelity 3D scene in approximately 20 seconds on an RTX 4090 GPU. The code will be released if accepted.