OmniVGGT：全模态驱动的视觉几何基础 Transformer
OmniVGGT: Omni-Modality Driven Visual Geometry Grounded Transformer
GeoAdapter 通过零初始化卷积将深度与相机内外参注入空间基础模型，配合随机模态子集训练，使模型在推理时可灵活利用任意数量辅助几何输入而不牺牲速度，为多视图三维重建与相机姿态估计提供了可复用的多模态融合范式。
原文依据 a GeoAdapter is proposed to encode depth and camera intrinsics/extrinsics into a spatial foundation model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论通用三维基础模型已开始引领统一多种视觉任务的趋势，但大多数模型假设仅使用 RGB 输入，并忽略了 readily available 的几何线索（例如相机内参、姿态和深度图）。为解决这一问题，我们提出了 OmniVGGT，这是一个新颖的框架，能够在训练和推理期间有效地受益于任意数量的辅助几何模态。在我们的框架中，提出了 GeoAdapter 将深度和相机内参/外参编码到空间基础模型中。它采用零初始化卷积来逐步注入几何信息，而不破坏基础模型的表示空间。这种设计确保了稳定的优化且开销可忽略不计，即使有多个额外输入，也能保持与 VGGT 相当的推理速度。此外，提出了一种随机多模态融合方案，在训练期间对每个实例随机采样模态子集。这使得在测试时能够输入任意数量的模态，并促进学习鲁棒的空间表示，而不是对辅助线索过拟合。在单目/多视图深度估计、多视图立体和相机姿态估计上的大量实验表明，OmniVGGT 在带有辅助输入时优于先前方法，并且即使在仅使用 RGB 输入时也达到了最优结果。为了进一步突出其实际效用，我们将 OmniVGGT 集成到视觉-语言-动作（VLA）模型中。由 OmniVGGT 增强的 VLA 模型不仅在主流基准上优于基于点云的普通基线，而且有效地利用可访问的辅助输入在机器人任务上实现了持续增益。项目页面：https://livioni.github.io/OmniVGGT-official/
英文原文摘要 1774 字符
General 3D foundation models have started to lead the trend of unifying diverse vision tasks, yet most assume RGB-only inputs and ignore readily available geometric cues (e.g., camera intrinsics, poses, and depth maps). To address this issue, we introduce OmniVGGT, a novel framework that can effectively benefit from an arbitrary number of auxiliary geometric modalities during both training and inference. In our framework, a GeoAdapter is proposed to encode depth and camera intrinsics/extrinsics into a spatial foundation model. It employs zero-initialized convolutions to progressively inject geometric information without disrupting the foundation model's representation space. This design ensures stable optimization with negligible overhead, maintaining inference speed comparable to VGGT even with multiple additional inputs. Additionally, a stochastic multimodal fusion regimen is proposed, which randomly samples modality subsets per instance during training. This enables an arbitrary number of modality inputs during testing and promotes learning robust spatial representations instead of overfitting to auxiliary cues. Extensive experiments on monocular/multi-view depth estimation, multi-view stereo, and camera pose estimation demonstrate that OmniVGGT outperforms prior methods with auxiliary inputs and achieves state-of-the-art results even with RGB-only input. To further highlight its practical utility, we integrated OmniVGGT into vision-language-action (VLA) models. The enhanced VLA model by OmniVGGT not only outperforms the vanilla point-cloud-based baseline on mainstream benchmarks, but also effectively leverages accessible auxiliary inputs to achieve consistent gains on robotic tasks. Project Page: https://livioni.github.io/OmniVGGT-official/