面向少标签的多模态域泛化
Towards Multimodal Domain Generalization with Few Labels
把半监督伪标签与域泛化结合到多模态设定：用融合后各单模态一致的高置信预测生成伪标签，并对不一致样本单独建模，再用跨模态原型对齐约束域不变与模态不变表征——这套「一致性伪标签 + 分歧感知 + 原型对齐」可直接迁移到多模态分类或检索在少标注、跨域、模态缺失时的训练流程。
原文依据 Semi-Supervised Multimodal Domain Generalization (SSMDG), which aims to learn robust multimodal models from multi-source data with few labeled samples
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论理想的多模态模型应当能够泛化到未见域，同时保持数据高效性以降低标注成本。为此，我们引入并研究一个新问题：半监督多模态域泛化（SSMDG），其目标是从带有少量标注样本的多源数据中学习鲁棒的多模态模型。我们观察到现有方法无法有效应对这一设定：多模态域泛化方法无法利用无标注数据，半监督多模态学习方法忽略了域偏移，而半监督域泛化方法则局限于单模态输入。为克服这些局限，我们提出一个统一框架，包含三个关键组件：共识驱动的一致性正则化，通过置信的融合单模态共识获得可靠的伪标签；分歧感知正则化，有效利用模糊的非共识样本；以及跨模态原型对齐，在强制域不变与模态不变表征的同时，通过跨模态翻译提升缺失模态情形下的鲁棒性。我们进一步建立了首个 SSMDG 基准，在该基准上，我们的方法在标准场景与缺失模态场景下均持续优于强基线。我们的基准与代码已公开于 https://github.com/lihongzhao99/SSMDG。
英文原文摘要 1364 字符
Multimodal models ideally should generalize to unseen domains while remaining data-efficient to reduce annotation costs. To this end, we introduce and study a new problem, Semi-Supervised Multimodal Domain Generalization (SSMDG), which aims to learn robust multimodal models from multi-source data with few labeled samples. We observe that existing approaches fail to address this setting effectively: multimodal domain generalization methods cannot exploit unlabeled data, semi-supervised multimodal learning methods ignore domain shifts, and semi-supervised domain generalization methods are confined to single-modality inputs. To overcome these limitations, we propose a unified framework featuring three key components: Consensus-Driven Consistency Regularization, which obtains reliable pseudo-labels through confident fused-unimodal consensus; Disagreement-Aware Regularization, which effectively utilizes ambiguous non-consensus samples; and Cross-Modal Prototype Alignment, which enforces domain- and modality-invariant representations while promoting robustness under missing modalities via cross-modal translation. We further establish the first SSMDG benchmarks, on which our method consistently outperforms strong baselines in both standard and missing-modality scenarios. Our benchmarks and code are available at https://github.com/lihongzhao99/SSMDG.