自监督与表征学习 · CVPR2026 ·2026.06.03

RAMEN:面向地球观测的分辨率可调多模态编码器

RAMEN: Resolution-Adjustable Multimodal Encoder for Earth Observation

Nicolas Houdré、Diego Marcos、Hugo Riffaud de Turckheim、Dino Ienco、Laurent Wendling、Camille Kurtz 等 7 人

与你有关的部分

将空间分辨率作为可控输出参数,使单一Transformer编码器能在推理时按需调整细节层次,为多传感器、多分辨率遥感数据的统一表征学习提供了可复用的预训练范式。

原文依据 We train a single, unified transformer encoder reconstructing masked multimodal EO data drawn from diverse sources
可迁移到 遥感图像分类多模态表征学习掩码图像重建

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

地球观测(EO)数据涵盖广泛的空间、光谱和时间分辨率,从高分辨率光学影像到低分辨率多光谱产品或雷达时间序列。尽管近期的基础模型改进了多模态集成以学习有意义的表征,但它们通常期望固定的输入分辨率,或基于特定传感器的编码器,限制了在异构EO模态间的泛化能力。为克服这些限制,我们提出RAMEN,一种分辨率可调的多模态编码器,以完全传感器无关的方式学习EO数据的共享视觉表征。RAMEN将模态以及空间和时间分辨率视为关键输入数据特征,从而在统一潜变量空间内实现跨模态的一致分析。其方法论主要贡献在于将空间分辨率定义为可控输出参数,使用户在推理时能直接控制所需的细节层次,并允许在空间精度与计算成本之间进行显式权衡。我们训练一个单一的统一Transformer编码器,重建来自多样来源的掩码多模态EO数据,确保跨传感器和分辨率的泛化能力。预训练后,RAMEN能有效迁移至已知和未见过的传感器配置,并在包含多种多传感器和多分辨率下游任务的社区标准PANGAEA基准上优于更大的最优模型。我们的代码和预训练模型将在接收后发布。

英文原文摘要 1552 字符

Earth observation (EO) data spans a wide range of spatial, spectral, and temporal resolutions, from high-resolution optical imagery to low resolution multispectral products or radar time series. While recent foundation models have improved multimodal integration for learning meaningful representations, they often expect fixed input resolutions or are based on sensor-specific encoders limiting generalization across heterogeneous EO modalities. To overcome these limitations we introduce RAMEN, a resolution-adjustable multimodal encoder that learns a shared visual representation across EO data in a fully sensor-agnostic manner. RAMEN treats the modality and spatial and temporal resolutions as key input data features, enabling coherent analysis across modalities within a unified latent space. Its main methodological contribution is to define spatial resolution as a controllable output parameter, giving users direct control over the desired level of detail at inference and allowing explicit trade-offs between spatial precision and computational cost. We train a single, unified transformer encoder reconstructing masked multimodal EO data drawn from diverse sources, ensuring generalization across sensors and resolutions. Once pretrained, RAMEN transfers effectively to both known and unseen sensor configurations and outperforms larger state-of-the-art models on the community-standard PANGAEA benchmark, containing various multi-sensor and multi-resolution downstream tasks. Our code and pretrained model will be released upon acceptance.

标签

会议 CVPR2026
页码 27838-27848
Bibkey
来源 CVF Open Access