MoD-DPO：利用模态解耦偏好优化缓解全模态大模型中的跨模态幻觉
MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization
给做多模态大模型（尤其是视听/视觉-语言模型）的研究者提供了一种可复用的偏好优化正则化思路：通过对无关模态加扰动、对相关模态加敏感度约束，显式解耦模态间虚假关联，可直接迁移到视觉-语言模型的视觉定位与幻觉抑制训练中。
原文依据 modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论全模态大语言模型（omni LLM）近期在视听理解任务上取得了强劲性能，但它们仍然极易受到由虚假相关和占主导地位的语言先验所引发的跨模态幻觉的影响。在本工作中，我们提出模态解耦直接偏好优化（MoD-DPO），这是一个简单而有效的框架，用于提升全模态大模型中的模态视觉定位能力。MoD-DPO 引入模态感知正则化项，显式地强制模型对无关模态中的扰动保持不变性、对相关模态中的扰动保持敏感性，从而减少非预期的跨模态交互。为进一步缓解对文本先验的过度依赖，我们加入了一项语言先验去偏惩罚，以抑制容易产生幻觉的纯文本回复。在多个视听幻觉基准上的大量实验表明，MoD-DPO 在相近训练预算下持续提升感知准确率与抗幻觉能力，优于此前的偏好优化基线。我们的发现凸显了模态忠实对齐的重要性，并展示了一条通往更可靠、更具韧性的多模态基础模型的可扩展路径。
英文原文摘要 1253 字符
Omni-modal large language models (omni LLMs) have recently achieved strong performance across audiovisual understanding tasks, yet they remain highly susceptible to cross-modal hallucinations arising from spurious correlations and dominant language priors. In this work, we propose Modality-Decoupled Direct Preference Optimization (MoD-DPO), a simple and effective framework for improving modality grounding in omni LLMs. MoD-DPO introduces modality-aware regularization terms that explicitly enforce invariance to corruptions in irrelevant modalities and sensitivity to perturbations in relevant modalities, thereby reducing unintended cross-modal interactions. To further mitigate over-reliance on textual priors, we incorporate a language-prior debiasing penalty that discourages hallucination-prone text-only responses. Extensive experiments across multiple audio-visual hallucination benchmarks demonstrate that MoD-DPO consistently improves perception accuracy and hallucination resistance, outperforming previous preference optimization baselines under similar training budgets. Our findings underscore the importance of modality-faithful alignment and demonstrate a scalable path toward more reliable and resilient multimodal foundation models.