EmbodiedSplat：面向开放词表三维场景理解的在线前馈语义 3DGS
EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding
把 CLIP 嵌入绑定到每个 3D 高斯上，并用 3D U-Net 聚合部分点云生成几何感知的 CLIP 特征，为在线开放词表 3D 语义重建提供了可复用的「2D 语言特征 + 3D 几何先验」融合方案。
原文依据 we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在具身任务中，智能体必须在线且近乎实时地构建并理解三维表示，因此随着场景探索即时理解三维场景至关重要。本研究提出 EmbodiedSplat，一种面向开放词表场景理解的在线前馈 3DGS，能够从流式图像中同时实现在线三维重建与三维语义理解。与现有的开放词表 3DGS 方法不同，我们的目标有两方面：1）以在线方式从超过 300 张流式图像重建整个场景的语义嵌入 3DGS；2）借助前馈设计高度泛化到新场景，并在与实时二维模型结合时支持近乎实时的三维语义重建。为实现上述目标，我们提出一种带有 CLIP 全局码本的在线稀疏系数场，它将二维 CLIP 嵌入绑定到每个三维高斯上，同时最小化内存消耗并保留 CLIP 完整的语义泛化能力。此外，我们通过 3D U-Net 聚合 3DGS 的部分点云来生成三维几何感知的 CLIP 特征，从而为面向二维的语言嵌入补充三维几何先验。我们的代码将在项目网站上公开。
英文原文摘要 1276 字符
Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D representation in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Our code will be publicly available in project website.