MVGGT：用于多视图 3D 指代表达分割的多模态视觉几何接地 Transformer
MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation
MVGGT 展示了如何用双分支 Transformer 从稀疏多视图 RGB 直接进行 3D 指代分割，避免了两阶段重建-分割的延迟与几何退化；其提出的前景梯度稀释问题与 PVSO 优化策略对稀疏视图下的 3D 密集预测训练有直接借鉴意义。
原文依据 We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning through a dual-branch design.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的 3D 指代表达分割（3DRES）方法大多依赖稠密、高质量的点云，而现实中的智能体（如机器人和手机）仅能获取少量稀疏的 RGB 视图，并面临严格的延迟约束。我们提出了多视图 3D 指代表达分割（MV-3DRES）任务，要求模型直接从稀疏多视图图像中恢复场景结构并分割所指对象。传统的两阶段流程（先重建点云再分割）往往几何质量低、目标区域粗糙或退化，且运行缓慢。我们提出了多模态视觉几何接地 Transformer（MVGGT），一种高效的端到端框架，通过双分支设计将语言信息融入稀疏视图几何推理。在此设定下训练会暴露一个关键的优化障碍，称为前景梯度稀释（FGD），即稀疏 3D 信号导致监督微弱。为解决此问题，我们引入了逐视图无目标抑制优化（PVSO），它在各视图间提供更强且更平衡的梯度，从而实现稳定高效的学习。为支持一致的评估，我们构建了 MVRefer 基准，为 MV-3DRES 定义了标准化设置和指标。实验表明，MVGGT 建立了首个强基线，并实现了高精度与快速推理，优于现有替代方案。代码见 https://mvggt.github.io/。
英文原文摘要 1511 字符
Most existing 3D referring expression segmentation (3DRES) methods rely on dense, high-quality point clouds, while real-world agents such as robots and mobile phones operate with only a few sparse RGB views and strict latency constraints. We introduce Multi-view 3D Referring Expression Segmentation (MV-3DRES), where the model must recover scene structure and segment the referred object directly from sparse multi-view images. Traditional two-stage pipelines, which first reconstruct a point cloud and then perform segmentation, often yield low-quality geometry, produce coarse or degraded target regions, and run slowly. We propose the Multimodal Visual Geometry Grounded Transformer (MVGGT), an efficient end-to-end framework that integrates language information into sparse-view geometric reasoning through a dual-branch design. Training in this setting exposes a critical optimization barrier, termed Foreground Gradient Dilution (FGD), where sparse 3D signals lead to weak supervision. To resolve this, we introduce Per-view No-target Suppression Optimization (PVSO), which provides stronger and more balanced gradients across views, enabling stable and efficient learning. To support consistent evaluation, we build MVRefer, a benchmark that defines standardized settings and metrics for MV-3DRES. Experiments show that MVGGT establishes the first strong baseline and achieves both high accuracy and fast inference, outperforming existing alternatives. The code is available at https://mvggt.github.io/.