REALM：面向高斯泼溅开放世界 3D 推理分割与编辑的 MLLM 智能体框架
REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting
提出全局到局部的空间视觉定位策略，利用多视角渲染与 MLLM 智能体聚合实现无需 3D 后训练的开放世界推理分割，可直接用于高斯泼溅场景的指令驱动物体定位与编辑。
原文依据 We perform segmentation directly on 3D Gaussian Splatting representations
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论弥合复杂人类指令与精确 3D 物体视觉定位之间的鸿沟，在视觉与机器人领域仍是一项重大挑战。现有的 3D 分割方法往往难以解读模糊的、基于推理的指令，而擅长此类推理的 2D 视觉语言模型又缺乏内在的 3D 空间理解能力。本文提出 REALM，一种创新的 MLLM 智能体框架，无需大量 3D 特定后训练即可实现开放世界推理分割。我们直接在高斯泼溅表示上执行分割，利用其渲染照片级逼真新视角的能力，这些视角非常适合 MLLM 理解。由于直接向 MLLM 输入一个或多个渲染视角可能导致对视角选择高度敏感，我们提出一种新颖的全局到局部空间视觉定位策略。具体而言，首先将多个全局视角并行输入 MLLM 智能体进行粗粒度定位，聚合响应以稳健地识别目标物体。然后，合成该物体的若干特写新视角以执行细粒度局部分割，产生准确且一致的 3D 掩码。大量实验表明，REALM 在 LERF、3D-OVS 以及我们新引入的 REALM3D 基准上，在解读显式和隐式指令方面均取得显著性能。此外，我们的智能体框架无缝支持一系列 3D 交互任务，包括物体移除、替换和风格迁移，展示了其实际效用与多功能性。
英文原文摘要 1604 字符
Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility.