面向数据高效零样本仿真到真实室内全景深度估计的 SO(3) 等变 ViT-Adapter
SO(3)-Equivariant ViT-Adapter for Data-Efficient Zero-Shot Sim-to-Real Indoor Panoramic Depth Estimation
把 SO(3) 等变归纳偏置以可变形交叉注意力的形式注入冻结的透视预训练 ViT 适配器，使视觉研究者能在不改动骨干的前提下把透视基础模型的零样本能力迁移到球面/全景输入，这一思路可复用于其他存在投影失真的成像域（鱼眼、事件相机、多相机环视）。
原文依据 introduces an SO(3) deformable cross-attention mechanism to effectively align SO(3)-equivariant features with perspective features
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论全景深度估计能够实现对三维环境的完整 360° 理解，但在泛化到真实场景时面临显著挑战。尽管 Depth Anything 等近期零样本深度模型在透视图像上取得了出色的泛化能力，但由于投影畸变以及缺乏球面几何感知，它们在全景图上的性能急剧下降。此外，采集大规模全景 RGB-D 数据成本高昂，阻碍了全景基础模型的大规模训练。为解决这些问题，我们提出 SO(3) 等变 ViT-Adapter，通过显式引入旋转等变归纳偏置，将透视预训练 ViT 强大的零样本能力迁移到全景深度估计。该适配器引入 SO(3) 可变形交叉注意力机制，有效对齐 SO(3) 等变特征与透视特征，在不修改 ViT 骨干的前提下增强旋转一致性。仅在合成全景图上训练，我们的框架在 Matterport3D 与 Stanford2D3D 等真实室内基准上取得了稳健的零样本仿真到真实性能，在全景深度估计上同时体现出数据高效性与强泛化能力。
英文原文摘要 1258 字符
Panoramic depth estimation enables a complete 360^\circ understanding of 3D environments but faces significant challenges in generalizing to real-world scenes. While recent zero-shot depth models like Depth Anything achieve remarkable generalization on perspective images, their performance sharply degrades on panoramas due to projection distortions and the lack of spherical geometric awareness. Moreover, collecting large-scale panoramic RGB-D data is costly, hindering the large-scale training of panoramic foundation models. To address these issues, we propose an SO(3)-Equivariant ViT-Adapter, which transfers the powerful zero-shot capability of the perspective pre-trained ViT to panoramic depth estimation by explicitly incorporating a rotation-equivariant inductive bias. Our adapter introduces an SO(3) deformable cross-attention mechanism to effectively align SO(3)-equivariant features with perspective features, enhancing rotational consistency without modifying the ViT backbone. Trained solely on synthetic panoramas, our framework achieves robust zero-shot sim-to-real performance on real indoor benchmarks, including Matterport3D and Stanford2D3D, demonstrating both data efficiency and strong generalization for panoramic depth estimation.