从无位姿多视图图像中学习面向空间智能的三维表征
Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images
在无相机位姿的多视图输入下，用双掩码迫使编码器从残缺视觉线索推断结构，并用位姿条件重投影把预测的3D点图与语义图对齐到图像平面，可作为无位姿前馈3D表征预训练的可复用范式，直接服务场景理解与具身感知的下游任务。
原文依据 a dual-masking strategy that strengthens geometry induction in the encoder
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论鲁棒的三维表征学习构成了空间智能的感知基础，支撑场景理解与具身智能中的下游任务。然而，直接从无位姿的多视图图像中学习此类表征仍然具有挑战性。近期的自监督方法试图以前馈方式统一几何、外观与语义，但往往存在几何归纳能力弱、外观细节有限以及几何与语义不一致的问题。我们提出 UniSplat，一个前馈框架，通过三个互补组件来应对上述局限。首先，我们提出一种双掩码策略，以增强编码器中的几何归纳能力。通过对编码器与解码器的 token 同时施加掩码，并将解码器掩码指向几何信息丰富的区域，模型被迫从残缺的视觉线索中推断结构信息，从而即使在无位姿输入下也能得到几何感知的表征。其次，我们设计了一种由粗到细的高斯泼溅策略，通过逐步细化辐射场来减少外观与语义之间的不一致。最后，为强化几何—语义一致性，我们引入一种位姿条件的重校准机制，该机制利用估计的相机参数将预测的三维点图与语义图重投影到图像平面，并将其与对应的 RGB 预测和语义预测对齐，从而关联多个头的输出，确保跨任务一致性并消除几何—语义失配。这些组件共同产生了统一的三维表征，对无位姿、稀疏视图输入具有鲁棒性，并能泛化到多种任务，为空间智能奠定了感知基础。
英文原文摘要 1733 字符
Robust 3D representation learning forms the perceptual foundation of spatial intelligence, enabling downstream tasks in scene understanding and embodied AI. However, learning such representations directly from unposed multi-view images remains challenging. Recent self-supervised methods attempt to unify geometry, appearance, and semantics in a feed-forward manner, but they often suffer from weak geometry induction, limited appearance detail, and inconsistencies between geometry and semantics.We introduce UniSplat, a feed-forward framework designed to address these limitations through three complementary components. First, we propose a dual-masking strategy that strengthens geometry induction in the encoder. By masking both encoder and decoder tokens, and targeting decoder masks toward geometry-rich regions, the model is forced to infer structural information from incomplete visual cues, yielding geometry-aware representations even under unposed inputs.Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies by progressively refining the radiance field.Finally, to enforce geometric-semantic consistency, we introduce a pose-conditioned recalibration mechanism that interrelates the outputs of multiple heads by reprojecting predicted 3D point and semantic maps into the image plane using estimated camera parameters, and aligning them with corresponding RGB and semantic predictions to ensure cross-task consistency and resolving geometry-semantic mismatches. Together, these components yield unified 3D representations that are robust to unposed, sparse-view inputs and generalize across diverse tasks, laying a perceptual foundation for spatial intelligence.