MOON2.0：面向电商商品理解的动态模态均衡多模态表征学习
MOON2.0: Dynamic Modality-balanced Multimodal Representation Learning for E-commerce Product Understanding
模态驱动的混合专家按样本模态构成自适应路由，配合产品内图文双层对齐，可直接迁移到图文检索与商品表征学习；其图文协同增强与动态样本过滤为多模态训练数据清洗提供可复用流程。
原文依据 a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期的多模态大语言模型显著推进了电商商品理解。然而，它们仍面临三个挑战：(i) 模态混合训练所引发的模态不均衡；(ii) 商品内部视觉与文本信息之间内在对齐关系未被充分利用；(iii) 对电商多模态数据中噪声的处理能力有限。为解决这些问题，我们提出 MOON2.0，一个面向电商商品理解的动态模态均衡多模态表征学习框架。它包含：(1) 模态驱动的混合专家（MoE），依据输入样本的模态构成对其进行自适应处理，从而实现多模态联合学习以缓解模态不均衡；(2) 一种双层对齐方法，以更好地利用单个商品内部的语义对齐特性；(3) 一种基于多模态大语言模型的图文协同增强策略，将文本丰富与视觉扩展相结合，并配合动态样本过滤以提升训练数据质量。我们进一步发布了 MBE2.0，一个用于电商表征学习与评估的协同增强多模态表征基准，地址为 https://huggingface.co/datasets/ZHNie/MBE2.0。实验表明，MOON2.0 在 MBE2.0 及多个公开数据集上取得了最优的零样本性能。此外，基于注意力的热力图可视化为 MOON2.0 改进的多模态对齐提供了定性证据。
英文原文摘要 1476 字符
Recent Multimodal Large Language Models (MLLMs) have significantly advanced e-commerce product understanding. However, they still face three challenges: (i) the modality imbalance induced by modality mixed training; (ii) underutilization of the intrinsic alignment relationships among visual and textual information within a product; and (iii) limited handling of noise in e-commerce multimodal data. To address these, we propose MOON2.0, a dynamic modality-balanced MultimOdal representation learning framework for e-commerce prOduct uNderstanding. It comprises: (1) a Modality-driven Mixture-of-Experts (MoE) that adaptively processes input samples by their modality composition, enabling Multimodal Joint Learning to mitigate the modality imbalance; (2) a Dual-level Alignment method to better leverage semantic alignment properties inside individual products; and (3) an MLLM-based Image-text Co-augmentation strategy that integrates textual enrichment with visual expansion, coupled with Dynamic Sample Filtering to improve training data quality. We further release MBE2.0, a co-augmented Multimodal representation Benchmark for E-commerce representation learning and evaluation at https://huggingface.co/datasets/ZHNie/MBE2.0. Experiments show that MOON2.0 delivers state-of-the-art zero-shot performance on MBE2.0 and multiple public datasets. Furthermore, attention-based heatmap visualization provides qualitative evidence of improved multimodal alignment of MOON2.0.