文本-图像条件驱动的3D生成
Text-Image Conditioned 3D Generation
论文提出文本-图像联合条件的3D生成任务,并给出TIGON双分支基线:分别保留图像条件与文本条件骨干,再用轻量跨模态融合模块整合两路预测。对做3D生成的研究者,这提供了一个可复用的条件融合范式,也提示可用晚期融合作为诊断手段来量化不同条件模态的互补性。
原文依据 even a simple late fusion of text- and image-conditioned predictions improves over single-modality models
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论高质量3D资产对VR/AR、工业设计与娱乐至关重要,这推动了能够根据用户提供的提示生成3D内容的生成模型的日益增长的兴趣。然而,现有大多数3D生成器依赖单一条件模态:图像条件模型通过利用像素对齐线索实现高视觉保真度,但当输入视图有限或存在歧义时,会遭受视角偏差;而文本条件模型受益于广泛的语义引导,却缺乏低层视觉细节。这限制了用户表达意图的方式,并引出一个自然的问题:能否将两种模态结合起来,以实现更灵活、更忠实的3D生成?我们的诊断性研究表明,即使是对文本条件与图像条件预测进行简单的晚期融合,也能优于单模态模型,这证明了强烈的跨模态互补性。基于这一发现,我们形式化了文本-图像条件3D生成任务,该任务要求在生成过程中对视觉样例与文本规格进行联合推理。为解决该任务,我们提出了TIGON,一个极简的双分支基线,它保持独立的图像条件与文本条件骨干网络,并采用轻量级跨模态融合。大量实验表明,文本-图像条件相比单模态方法带来了一致的增益,这表明互补的视觉-语言引导是未来3D生成研究的一个有前景的方向。
英文原文摘要 1474 字符
High-quality 3D assets are critical for VR/AR, industrial design, and entertainment, driving growing interest in generative models that can create 3D content from user-provided prompts. Most existing 3D generators, however, rely on a single conditioning modality: image-conditioned models deliver high visual fidelity by exploiting pixel-aligned cues but suffer from viewpoint bias when the input view is limited or ambiguous, whereas text-conditioned models benefit from broad semantic guidance yet lack low-level visual detail. This restricts how users can express their intent and raises a natural question: can the two modalities be combined to yield more flexible and faithful 3D generation? Our diagnostic study shows that even a simple late fusion of text- and image-conditioned predictions improves over single-modality models, evidencing strong cross-modal complementarity. Building on this finding, we formalize the task of Text-Image Conditioned 3D Generation, which requires joint reasoning over a visual exemplar and a textual specification during generation. To address this task, we introduce TIGON, a minimalist dual-branch baseline that maintains separate image- and text-conditioned backbones with lightweight cross-modal fusion. Extensive experiments demonstrate that text-image conditioning yields consistent gains over single-modality methods, suggesting complementary vision-language guidance as a promising direction for future 3D generation research.