GeoRelight：用灵活多模态扩散 Transformer 联合学习几何重光照与重建
GeoRelight: Learning Joint Geometrical Relighting and Reconstruction with Flexible Multi-Modal Diffusion Transformers
把重光照与单图三维几何估计放进同一个多模态 DiT 联合求解，并用与潜变量扩散兼容的无畸变 NDC-正交深度表示 iNOD 作为几何接口，可直接复用到需要几何一致性的图像重光照、逆渲染与单图重建任务。
原文依据 we propose a unified Multi-Modal Diffusion Transformer (DiT) that jointly solves for both: GeoRelight
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从单张照片对人物进行重光照是一个有吸引力但病态的问题，因为二维图像将三维几何、本征外观与光照模糊地纠缠在一起。现有方法要么使用会遭受误差累积的串行流水线，要么在重光照过程中没有显式利用三维几何，从而限制了物理一致性。由于重光照与三维几何估计是相互促进的任务，我们提出一个统一的多模态扩散 Transformer（DiT），同时求解二者：GeoRelight。我们通过两项关键技术贡献使这成为可能：各向同性 NDC-正交深度（iNOD），一种与潜变量扩散模型兼容的无畸变三维表示；以及一种策略性的混合数据训练方法，将合成数据与自动标注的真实数据相结合。通过联合求解几何与重光照，GeoRelight 在具有物理一致阴影的逼真重光照方面取得了最优结果，同时也能从单张图像实现高保真三维重建与本征估计。项目主页：https://yuxuan-xue.com/georelight
英文原文摘要 1099 字符
Relighting a person from a single photo is an attractive but ill-posed task, as a 2D image ambiguously entangles 3D geometry, intrinsic appearance, and illumination. Current methods either use sequential pipelines that suffer from error accumulation, or they do not explicitly leverage 3D geometry during relighting, which limits physical consistency. Since relighting and estimation of 3D geometry are mutually beneficial tasks, we propose a unified Multi-Modal Diffusion Transformer (DiT) that jointly solves for both: GeoRelight. We make this possible through two key technical contributions: isotropic NDC-Orthographic Depth (iNOD), a distortion-free 3D representation compatible with latent diffusion models; and a strategic mixed-data training method that combines synthetic and auto-labeled real data. By solving geometry and relighting jointly, GeoRelight achieves state-of-the-art results in photorealistic relighting with physically-consistent shadows, as well as high-fidelity 3D reconstruction and intrinsic estimation from a single image. Project page: https://yuxuan-xue.com/georelight