面向多模态大语言模型的不确定性感知探索式直接偏好优化
Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models
把词元级认知不确定性作为探索强度信号，替代模型自评的视觉敏感度来分配 DPO 训练权重，可直接迁移到任何需要细粒度视觉 grounding 监督的 MLLM 偏好对齐流程中。
原文依据 guided by token-level epistemic uncertainty
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论直接偏好优化（DPO）已被证明是一种有效方案，它通过从偏好对中学习来缓解多模态大语言模型（MLLM）中的幻觉。其关键挑战之一在于如何将序列级偏好转化为对视觉保真度的细粒度监督。为保护易产生幻觉的视觉相关词元，现有方法通常依据模型自评估的视觉敏感度信号来分配训练权重。然而，这种由仍在训练中的模型估计出的敏感度会引入自指偏差：它强化了模型已充分学到的视觉线索，却忽视了难以感知但至关重要的细节，从而限制了更深层次的对齐。在本工作中，我们提出一种面向 MLLM 的不确定性感知探索式直接偏好优化（UE-DPO）方法，该方法以词元级认知不确定性为引导，使模型能够发现自身的认知缺陷并主动探索以实现自我纠正。具体而言，我们首先从模型无法将词元预测建立在给定图像之上这一失败中量化不确定性。然后，基于不确定性感知的探索强度，我们在偏好样本中对视觉上有缺陷的词元施加更大的学习压力，并缓解对非偏好样本中有益知识的过度惩罚。此外，我们为该提供了理论依据，大量实验证明了其有效性与鲁棒性。
英文原文摘要 1495 字符
Direct Preference Optimization (DPO) has proven to be an effective solution for mitigating hallucination in Multimodal Large Language Models (MLLMs) by learning from preference pairs. One of its key challenges lies in how to transfer the sequence-level preference into fine-grained supervision on visual fidelity. To safeguard vision-related tokens that are prone to hallucination, existing methods typically allocate training emphasis according to the model's self-assessed visual sensitivity signals. However, such sensitivity, estimated by a model still under training, introduces self-referential bias: reinforcing already well-learned visual cues while neglecting hard-to-perceive but critical details, thereby limiting deeper alignment. In this work, we propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for MLLMs, which enables the model to uncover its cognitive deficiencies and actively explore for self-correction, guided by token-level epistemic uncertainty. Specifically, we first quantify the uncertainty from the model's failure to ground token predictions in the given image. Then, based on an uncertainty-aware exploration intensity, we encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples. Further, we provide a theoretical justification for our method, and extensive experiments demonstrate its effectiveness and robustness.