VLM 与 IRM 的因果联姻：从理解到推理
A Causal Marriage between VLM and IRM from Understanding to Reasoning
把 CLIP 的对比目标在 token 粒度上重新解释为 IRM 的不变性准则，并给出一个无需改结构的中训练方案来注入不变学习信号；做视觉-语言预训练或 OOD 泛化的人可以直接复用这套 token 分解与词表约束 InfoNCE 等价性，以及用不变对齐分数作为 MLLM 强化学习过程奖励的思路。
原文依据 a vocabulary-constrained InfoNCE objective becomes formally equivalent to IRM's invariance criterion
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论像 CLIP 这样的视觉-语言模型（VLM）展现出非凡的分布外（OOD）泛化能力，而这种鲁棒性背后的理论基础在很大程度上仍未被探索。本工作通过 token 级因果表征学习，在 CLIP 与不变风险最小化（IRM）——克服 OOD 问题的原则性范式——之间建立了联系。我们的关键洞见是：CLIP 的对比目标在最优训练时，会在词与短语粒度上恢复模态不变的因果因子。通过将文本提示分解为类别特定的 token（因果因子）与类别无关的上下文 token（环境因子），我们证明：受词表约束的 InfoNCE 目标在形式上等价于 IRM 的不变性准则。基于这一等价性，我们提出一种中训练范式，旨在不修改架构的前提下向预训练 CLIP 注入不变学习信号，从而得到 OOD 性能更优的 CLIP-IRM。我们进一步将这一因果对齐扩展到多模态推理：把 CLIP-IRM 的不变对齐分数用作强化学习中的过程级奖励，从而有效地将 IRM 的保证迁移到多模态大语言模型的鲁棒序列决策中。大量实验验证了我们的理论框架，并在多模态 OOD 理解与推理任务上均呈现出显著提升。
英文原文摘要 1433 字符
Vision-Language Models (VLMs) like CLIP exhibit extraordinary out-of-distribution (OOD) generalization, while the theoretical foundations underlying this robustness remain largely unexplored. This work establishes a connection between CLIP and Invariant Risk Minimization (IRM), the principled paradigm to overcome OOD problems, through token-level causal representation learning. Our key insight is that CLIP's contrastive objective, when optimally trained, recovers modality-invariant causal factors at the word-and-phrase granularity. By decomposing text prompts into class-specific tokens (causal factors) and class-agnostic context tokens (environmental factors), we prove that a vocabulary-constrained InfoNCE objective becomes formally equivalent to IRM's invariance criterion. Grounded in this equivalence, we propose a mid-training paradigm aiming to inject invariant learning signals into pre-trained CLIP without architectural modification, yielding CLIP-IRM with superior OOD performance. We further extend this causal alignment to multimodal reasoning via using CLIP-IRM's invariant alignment scores as process-level rewards in reinforcement learning, effectively transplanting IRM's guarantees to robust sequential decision-making in Multimodal Large Language Models. Extensive experiments validate our theoretical framework and present substantial improvements in both multimodal OOD understanding and reasoning tasks.