3D-Fixer:单图像三维场景的由粗到细原位补全
3D-Fixer: Coarse-to-Fine In-place Completion for 3D Scenes from a Single Image
把几何估计得到的碎片化点云当作空间锚框,在原始位置做 in-place 补全,从而绕开显式 6DoF 位姿优化;这种「以碎片几何为条件、粗到细生成」的思路可直接迁移到单视图场景重建与资产补全任务。
原文依据 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从单视图进行组合式三维场景生成,需要同时恢复场景布局与三维资产。现有方法主要分为两类:前馈生成方法与逐实例生成方法。前者通过高效的网络推理直接预测带有显式六自由度位姿的三维资产,但对复杂场景的泛化能力较差;后者通过分而治之的策略提升泛化能力,却受制于耗时的位姿优化。为弥合这一差距,本文提出 3D-Fixer,一种新颖的原位补全范式。具体而言,3D-Fixer 扩展三维物体生成先验,以原始位置处部分可见的点云为条件生成完整的三维资产,这些点云从几何估计方法得到的碎片化几何中裁剪而来。与需要显式位姿对齐的先前工作不同,3D-Fixer 将碎片化几何用作空间锚框,以保持布局保真度。其核心在于,我们提出一种由粗到细的生成方案以解决遮挡下的边界歧义,并辅以双分支条件网络与遮挡鲁棒特征对齐(ORFA)策略以实现稳定训练。此外,为缓解数据稀缺瓶颈,我们提出 ARSG-110K,这是迄今最大的场景级数据集,包含超过 11 万个多样化场景与 300 万张带高保真三维真值标注的图像。大量实验表明,3D-Fixer 取得了最优的几何精度,显著优于 MIDI 与 Gen3DSR 等基线方法,同时保持了扩散过程的效率。代码与数据将在 https://zx-yin.github.io/3dfixer 公开。
英文原文摘要 1728 字符
Compositional 3D scene generation from a single view requires the simultaneous recovery of scene layout and 3D assets. Existing approaches mainly fall into two categories: feed-forward generation methods and per-instance generation methods. The former directly predict 3D assets with explicit 6DoF poses through efficient network inference, but they generalize poorly to complex scenes. The latter improve generalization through a divide-and-conquer strategy, but suffer from time-consuming pose optimization. To bridge this gap, we introduce 3D-Fixer, a novel in-place completion paradigm. Specifically, 3D-Fixer extends 3D object generative priors to generate complete 3D assets conditioned on the partially visible point cloud at the original locations, which are cropped from the fragmented geometry obtained from the geometry estimation methods. Unlike prior works that require explicit pose alignment, 3D-Fixer uses fragmented geometry as a spatial anchor to preserve layout fidelity. At its core, we propose a coarse-to-fine generation scheme to resolve boundary ambiguity under occlusion, supported by a dual-branch conditioning network and an Occlusion-Robust Feature Alignment (ORFA) strategy for stable training. Furthermore, to address the data scarcity bottleneck, we present ARSG-110K, the largest scene-level dataset to date, comprising over 110K diverse scenes and 3M annotated images with high-fidelity 3D ground truth. Extensive experiments show that 3D-Fixer achieves state-of-the-art geometric accuracy, which significantly outperforms baselines such as MIDI and Gen3DSR, while maintaining the efficiency of the diffusion process. Code and data will be publicly available at https://zx-yin.github.io/3dfixer.