逐 token 对齐文本、图像与三维结构
Aligning Text, Images and 3D Structure Token-by-Token
把结构化 3D 场景当作一种可 tokenize 的模态，用统一的 LLM 自回归建模语言、图像与 3D 场景，为单图重建完整 3D 场景和 3D 目标识别提供了可复用的 tokenizer 与训练配方。
原文依据 we propose a unified LLM framework that aligns language, images, and 3D scenes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论创造能够以三维方式理解世界的机器，对于协助设计师构建和编辑三维环境、以及帮助机器人在三维空间中导航与交互至关重要。受语言与图像建模进展的启发，我们探索自回归模型在一种新模态上的潜力：结构化三维场景。为此，我们提出一个统一的 LLM 框架，用于对齐语言、图像与三维场景，并给出详细的“操作指南”，阐明实现最优训练与性能的关键设计选择，回答与数据表示、模态特定目标等相关的问题。我们展示了如何对复杂三维物体进行 tokenize 以纳入我们的结构化三维场景模态。我们在四项核心三维任务——渲染、识别、指令跟随与问答——以及四个合成与真实世界的三维数据集上评估性能。我们展示了模型在从单张图像重建由复杂物体构成的完整三维场景、以及在真实世界三维物体识别任务上的有效性。项目主页：https://glab-caltech.github.io/kyvo/
英文原文摘要 1146 字符
Creating machines capable of understanding the world in 3D is essential in assisting designers that build and edit 3D environments and robots navigating and interacting within a three-dimensional space. Inspired by advances in language and image modeling, we investigate the potential of autoregressive models for a new modality: structured 3D scenes. To this end, we propose a unified LLM framework that aligns language, images, and 3D scenes and provide a detailed "cookbook" outlining critical design choices for achieving optimal training and performance addressing key questions related to data representation, modality-specific objectives, and more. We show how to tokenize complex 3D objects to incorporate into our structured 3D scene modality. We evaluate performance across four core 3D tasks - rendering, recognition, instruction-following, and question-answering - and four 3D datasets, synthetic and real-world. We show our model's effectiveness on reconstructing complete 3D scenes consisting of complex objects from a single image and on real-world 3D object recognition tasks. Project webpage: https://glab-caltech.github.io/kyvo/