基于保形预测自校准的低质量数据多模态学习
Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration
把保形预测当作在线可靠性打分器，用它对单模态特征做选择性融合、对反向传播梯度按样本可靠性重加权，可直接迁移到视觉多模态（RGB-深度、图像-文本）训练中应对模态失衡与标签噪声。
原文依据 recalibrates the gradient flow during backpropagation based on instance-wise reliability scores
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态学习常常面临低质量数据的挑战，这主要体现在两个方面：模态不平衡和噪声污染。尽管这两个问题通常被孤立地研究，我们认为它们在学习过程中对单个模态和样本可靠性的预测不确定性上具有共同的根源。本文提出了一个统一框架，称为保形预测自校准（CPSC），利用保形预测使模型具备即时自引导校准的能力。CPSC 的核心在于一种新颖的自校准训练循环，它无缝集成了两个关键模块：（1）表征自校准，将单模态特征分解为多个分量，选择性地融合由保形预测器识别出的最鲁棒分量，以增强特征韧性。（2）梯度自校准，在反向传播过程中基于样本级可靠性分数重新校准梯度流，引导优化朝向更可信的方向。此外，我们还为保形预测器设计了一种自更新策略，以确保整个系统在训练过程中一致地协同进化。在不平衡和噪声设置下对六个基准数据集的大量实验表明，我们的 CPSC 框架始终优于现有的最优方法。代码可在 https://github.com/XunCHN/CPSC 获取。
英文原文摘要 1499 字符
Multimodal learning often grapples with the challenge of low-quality data, which predominantly manifests as two facets: modality imbalance and noisy corruption. While these issues are often studied in isolation, we argue that they share a common root in the predictive uncertainty towards the reliability of individual modalities and instances during learning. In this paper, we propose a unified framework, termed Conformal Predictive Self-Calibration (CPSC), which leverages conformal prediction to equip the model with the ability to perform self-guided calibration on-the-fly. The core of our proposed CPSC lies in a novel self-calibrating training loop that seamlessly integrates two key modules: (1) Representation Self-Calibration, which decomposes unimodal features into components, selectively fuses the most robust ones identified by a conformal predictor to enhance feature resilience. (2) Gradient Self-Calibration, which recalibrates the gradient flow during backpropagation based on instance-wise reliability scores, steering the optimization towards more trustworthy directions. Furthermore, we also devise a self-update strategy for the conformal predictor to ensure the entire system co-evolves consistently throughout the training process. Extensive experiments on six benchmark datasets under both imbalanced and noisy settings demonstrate that our CPSC framework consistently outperforms existing state-of-the-art methods. Our code is available at https://github.com/XunCHN/CPSC.