MAD：面向多模态大语言模型跨模态幻觉缓解的模态自适应解码
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
对做多模态大模型推理的人来说，这是一种免训练的解码期干预手段：用模型自评估得到的模态相关性概率去加权各模态的对比解码分支，可直接套用到任意音视频/视觉语言模型的推理流程中抑制跨模态幻觉，无需重训。
原文依据 The extracted modality probabilities are then used to adaptively weight contrastive decoding branches
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）存在跨模态幻觉问题，即一种模态会不恰当地影响关于另一种模态的生成，从而导致捏造输出。这暴露出模态交互控制方面更为根本的缺陷。为解决该问题，我们提出模态自适应解码（MAD），这是一种免训练方法，可依据任务需求对模态特定的解码分支进行自适应加权。MAD 利用模型自身评估模态相关性的固有能力，通过查询每个任务需要哪些模态来实现这一点。随后，提取出的模态概率被用于对对比解码分支进行自适应加权，使模型能够聚焦于相关信息，同时抑制跨模态干扰。在 CMM 和 AVHBench 上的大量实验表明，MAD 显著降低了多种音视频语言模型的跨模态幻觉（VideoLLaMA2-AV 分别提升 7.8% 和 2.0%，Qwen2.5-Omni 分别提升 8.7% 和 4.7%）。我们的方法表明，通过自评估实现的显式模态感知对于稳健的多模态推理至关重要，并为现有对比解码方法提供了一种有原则的扩展。
英文原文摘要 1214 字符
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8% and 2.0% improvements for VideoLLaMA2-AV, 8.7% and 4.7% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods.