Realiz3D：通过域感知学习实现照片级真实的3D生成
Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning
通过引入协变量与小型残差适配器，将视觉域（真实/合成）从控制信号中解耦，使在合成渲染上微调的扩散生成器在施加几何、材质、视角控制时仍能输出照片级真实图像；其分层与去噪步角色分析可直接迁移到任何需要跨域控制迁移的扩散微调流程。
原文依据 we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们通常希望生成既照片级真实又具有3D一致性的图像，并遵循精确的几何、材质与视角控制。通常的做法是：使用合成3D资产的渲染图对在数十亿真实图像上预训练的图像生成器进行微调，因为渲染图带有控制信号的标注。虽然这种方式可以学到所需的控制能力，但由于照片与渲染图之间的域差距，往往会损害图像的真实感。我们观察到，这一问题很大程度上源于模型无意间学到了控制信号的存在与图像合成外观之间的关联。为解决该问题，我们提出Realiz3D，一个用于训练扩散模型的轻量级框架，它将控制与视觉域解耦。其核心思想是：通过引入一个协变量并将其输入小型残差适配器来切换域，从而将视觉域（真实或合成）与其他控制信号显式地分开学习。这样，生成器便可以在不拟合特定视觉域的情况下获得可控性。通过这种方式，即使施加控制，模型也能被引导生成真实感图像。我们利用关于扩散生成器中不同层与去噪步作用的洞见，增强控制向真实域的迁移能力，并据此提出新的训练与推理策略，进一步缩小域差距。我们在文本到多视图生成以及基于3D输入的纹理生成任务上展示了Realiz3D的优势，其输出兼具3D一致性与照片级真实感。
英文原文摘要 1609 字符
We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.