面向 3D 人体姿态估计的平衡多模态学习
Towards Balanced Multi-Modal Learning in 3D Human Pose Estimation
用 Shapley 值量化各模态（RGB/LiDAR/mmWave/WiFi）对 3D 姿态估计的边际贡献来诊断模态不平衡，并配合训练早期的学习节奏调控策略，可迁移到任何多模态融合的视觉任务中做模态权重诊断与平衡训练。
原文依据 we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论3D 人体姿态估计（3D HPE）已成为一个突出的研究课题，尤其是在基于 RGB 的方法领域。然而，RGB 图像的使用往往受到遮挡和隐私约束等问题的限制。因此，利用非侵入式传感器的多模态感知正日益受到关注。尽管如此，多模态 3D HPE 仍面临包括模态不平衡在内的诸多挑战。在本工作中，我们提出了一种新颖的面向 3D HPE 的平衡多模态学习方法，充分利用 RGB、LiDAR、毫米波和 WiFi 的能力。具体而言，我们提出了一种基于 Shapley 值的贡献算法来评估各模态的贡献并检测模态不平衡。为解决这种不平衡，我们设计了一种模态学习调控策略，在训练的早期阶段减缓学习过程。我们在广泛采用的多模态数据集 MM-Fi 上进行了大量实验，证明了我们的方法在复杂条件下增强 3D 姿态估计的优越性。我们的源代码可在 https://github.com/MICLAB-BUPT/AWC 获取。
英文原文摘要 1114 字符
3D human pose estimation (3D HPE) has emerged as a prominent research topic, particularly in the realm of RGB-based methods. However, the use of RGB images is often limited by issues such as occlusion and privacy constraints. Consequently, multi-modal sensing, which leverages non-intrusive sensors, is gaining increasing attention. Nevertheless, multi-modal 3D HPE still faces challenges, including modality imbalance. In this work, we introduce a novel balanced multi-modal learning method for 3D HPE, which harnesses the power of RGB, LiDAR, mmWave, and WiFi. Specifically, we propose a Shapley value-based contribution algorithm to assess the contribution of each modality and detect modality imbalance. To address this imbalance, we design a modality learning regulation strategy that decelerates the learning process during the early stages of training. We conduct extensive experiments on the widely adopted multi-modal dataset, MM-Fi, demonstrating the superiority of our approach in enhancing 3D pose estimation under complex conditions. Our source code is available at https://github.com/MICLAB-BUPT/AWC.