CaReFlow：用于多模态融合的循环自适应整流流
CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion
把整流流当作跨模态分布映射器，用一对多映射让源模态每个样本看到目标模态全局分布，配合自适应松弛对齐与循环整流流约束，可作为视觉-文本/视觉-音频特征对齐的通用预对齐模块，替代扩散或对抗式一对一对齐。
原文依据 leveraging the characteristic of rectified flow that can map one distribution to another via a straight trajectory
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论模态鸿沟显著制约了多模态融合的效果。已有方法常采用扩散模型与对抗学习等技术来缩小模态鸿沟，但它们通常聚焦于一对一的对齐，未能让源模态的数据点接触到目标模态的全局分布信息。为此，本文利用整流流能够通过直线轨迹将一个分布映射到另一个分布的特性，将整流流扩展用于模态分布映射。具体而言，我们利用整流流中的「一对多映射」策略，使源模态的每个数据点都能观测到目标模态的整体分布。这也缓解了每个样本内部配对数据不足的问题，从而实现更稳健的分布变换。此外，为实现更精确的分布映射并解决一对多映射中流向模糊的问题，我们设计了「自适应松弛对齐」：对属于同一样本的模态对施加更严格的对齐，而对不属于同一样本或同一类别的模态对施加松弛映射。同时，为防止分布映射过程中的信息损失，我们引入「循环整流流」，确保迁移后的特征能够被还原回原始特征，从而使多模态表征学习到充分的模态特有信息。在完成分布对齐后，即使采用简单的融合方法，我们的方法在多模态情感计算的多个任务上也取得了极具竞争力的结果，可视化结果验证了其能够有效缩小模态鸿沟。
英文原文摘要 1698 字符
Modality gap significantly restricts the effectiveness of multimodal fusion. Previous methods often use techniques such as diffusion models and adversarial learning to reduce the modality gap, but they typically focus on one-to-one alignment without exposing the data points of the source modality to the global distribution information of the target modality. To this end, leveraging the characteristic of rectified flow that can map one distribution to another via a straight trajectory, we extend rectified flow for modality distribution mapping. Specifically, we leverage the `one-to-many mapping' strategy in rectified flow that allows each data point of the source modality to observe the overall target distribution. This also alleviates the issue of insufficient paired data within each sample, enabling a more robust distribution transformation. Moreover, to achieve more accurate distribution mapping and address the ambiguous flow directions in one-to-many mapping, we design `adaptive relaxed alignment', enforcing stricter alignment for modality pairs belonging to the same sample, while applying relaxed mapping for pairs not belonging to the same sample or category. Additionally, to prevent information loss during distribution mapping, we introduce `cyclic rectified flow' to ensure the transferred features can be translated back to the original features, allowing multimodal representations to learn sufficient modality-specific information. After distribution alignment, our approach achieves very competitive results on multiple tasks of multimodal affective computing even with a simple fusion method, and visualizations verify that it can effectively reduce the modality gap.