多模态大语言模型中对抗视觉表征退化的预测正则化
Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
提出预测正则化（PRe）强制中间层退化视觉特征预测初始视觉特征，为提升MLLM内部视觉表征质量提供可复用训练策略，可缓解语言驱动训练导致的视觉保真度下降。
原文依据 we propose Predictive Regularization (PRe) to force degraded intermediate features to predict initial visual features
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管多模态大语言模型（MLLMs）在视觉-语言任务上表现出色，但其语言驱动的训练对内部视觉基础能力造成的代价仍不明确。本文通过详细的诊断分析揭示了一个普遍问题：MLLMs中的视觉表征退化。具体而言，我们发现与初始视觉特征相比，LLM中间层的视觉表征在全局功能和图像块结构上均出现退化。我们将此现象归因于单一文本生成目标驱动的视觉牺牲，即模型为优化答案生成而牺牲视觉保真度。我们认为一个鲁棒的MLLM需要强大的跨模态推理和核心视觉能力，并提出预测正则化（PRe）来强制退化的中间特征预测初始视觉特征，从而保持MLLM内部表征的固有视觉属性。大量实验证实，缓解这种视觉退化能有效提升视觉-语言性能，凸显了在MLLMs中培养鲁棒内部视觉表征对于全面多模态理解的关键重要性。
英文原文摘要 1259 字符
While Multimodal Large Language Models (MLLMs) excel at vision-language tasks, the cost of their language-driven training on internal visual foundational competence remains unclear. In this paper, we conduct a detailed diagnostic analysis to unveil a pervasive issue: visual representation degradation in MLLMs. Specifically, we find that compared to the initial visual features, the visual representation in the middle layers of LLM exhibits both a degradation in global function and patch structure. We attribute this phenomenon to a visual sacrifice driven by the singular text-generation objective, where the model compromises its visual fidelity to optimize for answer generation. We argue that a robust MLLM requires both strong cross-modal reasoning and core visual competence, and propose Predictive Regularization (PRe) to force degraded intermediate features to predict initial visual features, thereby maintaining the inherent visual attributes of the MLLM's internal representations. Extensive experiments confirm that mitigating this visual degradation effectively boosts vision-language performance, underscoring the critical importance of fostering robust internal visual representations within MLLMs for comprehensive multimodal understanding.