SeeThrough3D：文本到图像生成中的遮挡感知三维控制
SeeThrough3D: Occlusion Aware 3D Control in Text-to-Image Generation
将三维布局中的遮挡关系显式编码为半透明三维框渲染，并通过掩码自注意力绑定物体与文本，为布局可控生成提供可复用的遮挡感知条件注入方式。
原文依据 We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们指出遮挡推理是三维布局条件生成中一个基础但被忽视的方面。它对于合成具有深度一致几何与尺度的部分遮挡物体至关重要。现有方法虽能生成遵循输入布局的逼真场景，却往往无法建模精确的物体间遮挡。我们提出 SeeThrough3D，一个显式建模遮挡的三维布局条件生成模型。我们引入遮挡感知三维场景表示（OSCR），其中物体被描绘为置于虚拟环境中的半透明三维框，并从期望相机视角渲染。透明度编码了隐藏的物体区域，使模型能够推理遮挡，而渲染视角则在生成过程中提供显式的相机控制。我们通过引入一组源自渲染三维表示的视觉 token，对预训练的基于流的文本到图像生成模型进行条件化。此外，我们应用掩码自注意力，将每个物体边界框准确绑定到其对应的文本描述，从而在生成多个物体时避免属性混淆。为训练模型，我们构建了一个包含强物体间遮挡的多样多物体场景合成数据集。SeeThrough3D 能有效泛化到未见过的物体类别，并实现具有逼真遮挡和一致相机控制的精确三维布局控制。
英文原文摘要 1476 字符
We identify occlusion reasoning as a fundamental yet overlooked aspect for 3D layout-conditioned generation. It is essential for synthesizing partially occluded objects with depth-consistent geometry and scale. While existing methods can generate realistic scenes that follow input layouts, they often fail to model precise inter-object occlusions. We propose SeeThrough3D, a model for 3D layout conditioned generation that explicitly models occlusions. We introduce an occlusion-aware 3D scene representation (OSCR), where objects are depicted as translucent 3D boxes placed within a virtual environment and rendered from desired camera viewpoint. The transparency encodes hidden object regions, enabling the model to reason about occlusions, while the rendered viewpoint provides explicit camera control during generation. We condition a pretrained flow based text-to-image image generation model by introducing a set of visual tokens derived from our rendered 3D representation. Furthermore, we apply masked self-attention to accurately bind each object bounding box to its corresponding textual description, enabling accurate generation of multiple objects without object attribute mixing. To train the model, we construct a synthetic dataset with diverse multi-object scenes with strong inter-object occlusions. SeeThrough3D generalizes effectively to unseen object categories and enables precise 3D layout control with realistic occlusions and consistent camera control.