ProM3E：面向生态学的概率掩码多模态嵌入模型
ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
在嵌入空间做掩码模态重建、支持任意模态到任意模态的生成与模态反转，这种「学什么该融合」的概率式融合思路可直接迁移到视觉-文本-音频等多模态检索与缺失模态补全任务。
原文依据 a novel cross-modal retrieval approach that mixes inter-modal and intra-modal similarities
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 ProM3E，一个面向生态学的概率掩码多模态嵌入模型，用于多模态表征的任意模态到任意模态生成。ProM3E 基于嵌入空间中的掩码模态重建，学习在给定少量上下文模态的条件下推断缺失模态。按设计，我们的模型支持嵌入空间中的模态反转。模型的概率特性使我们能够分析针对给定下游任务融合多种模态的可行性，本质上是在学习「融合什么」。利用模型的这些特性，我们提出一种新颖的跨模态检索方法，混合模态间与模态内相似度，从而在所有检索任务上取得更优性能。我们进一步利用模型的隐表征执行线性探测任务，并证明模型具有更优的表征学习能力。我们的全部代码、数据集与模型将在 https://vishu26.github.io/prom3e 发布。
英文原文摘要 977 字符
We introduce ProM3E, a probabilistic masked multimodal embedding model for any-to-any generation of multimodal representations for ecology. ProM3E is based on masked modality reconstruction in the embedding space, learning to infer missing modalities given a few context modalities. By design, our model supports modality inversion in the embedding space. The probabilistic nature of our model allows us to analyze the feasibility of fusing various modalities for given downstream tasks, essentially learning what to fuse. Using these features of our model, we propose a novel cross-modal retrieval approach that mixes inter-modal and intra-modal similarities to achieve superior performance across all retrieval tasks. We further leverage the hidden representation from our model to perform linear probing tasks and demonstrate the superior representation learning capability of our model. All our code, datasets and model will be released at https://vishu26.github.io/prom3e.