Sky2Ground：变高度场景建模基准
Sky2Ground: A Benchmark for Site Modeling under Varying Altitude
提出 SkyNet 双流网络与 Masked-Satellite-Attention 机制，解决卫星图像在多视角相机定位与渲染中引入噪声的问题，并采用课程学习式相机采样策略；对做多视角几何、跨视角定位与神经渲染的研究者可复用其注意力隔离与训练采样思路。
原文依据 We propose a restricted-attention mechanism, termed as `Masked-Satellite-Attention' which prevents ground/aerial images from interacting with satellite images.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文提出一个问题：给定从地面、航空和卫星视角拍摄的多张图像，定位相机并生成场景渲染。我们引入一个名为 Sky2Ground 的数据集，包含覆盖全部 3 种视角的合成/真实图像，以及相机参数和稠密深度图/表面法线。近期工作表明，VGGT 等基于 Transformer 的网络能够在单次前向传播中推断场景参数。然而，我们正式揭示出简单微调此类模型会降低性能，且无法仅靠暴力扩展解决。我们发现罪魁祸首是卫星图像，其在学习过程中注入过多噪声。因此，我们提出 SkyNet 以支持使用卫星图像进行学习。SkyNet 是一个双流神经网络，一个流显式处理卫星图像，另一个流处理所有模态。我们提出一种受限注意力机制，称为“掩码卫星注意力”，阻止地面/航空图像与卫星图像交互。此外，SkyNet 采用受课程学习启发的策略进行优化：在训练期间采样彼此相距较远的相机。在 Sky2Earth 数据集上的大量实验表明，SkyNet 在绝对性能上优于现有方法 23%。我们的数据集和代码将在 huggingface 上公开。
英文原文摘要 1425 字符
In this work, we propose the problem of localizing cameras and producing renders of a scene, given multiple images captured from ground/aerial/satellite viewpoints. We introduce a dataset called Sky2Ground, which contains synthetic/real images across all 3 viewpoints, along with camera parameters, and dense depth-maps/surface-normals. Recent works have shown that transformer-based nets like VGGT are capable of inferring scene-parameters in a single-forward pass. However, we formally reveal that simply fine-tuning such models reduces performance, and can't be solved simply by bruteforce-scaling. We find the culprit to be satellite images, which inject too much noise during the learning process. Therefore, we propose SkyNet to enable learning using satellite-images. SkyNet is a two-stream neural-net, with one stream explicitly processing satellite, and another processing all modalities together.We propose a restricted-attention mechanism, termed as `Masked-Satellite-Attention' which prevents ground/aerial images from interacting with satellite images. Further, our SkyNet is optimized with strategies inspired from curriculum-learning: sampling cameras which are far-away from each other during training. Extensive experiments on our Sky2Earth dataset reveal that SkyNet outperforms existing methods by 23% in terms of absolute performance. Our dataset, and code shall be made publicly available on huggingface.