DeepAlign：通过模态特定对齐缓解模态冲突
DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment
针对视觉-语言表征错位与模态细节流失，提出表征干预加结构诱导知识蒸馏的对齐框架，可直接用于改进任意 MLLM 的视觉编码器到语言模型的投影对齐环节，并解锁交错图文序列上的多模态上下文学习。
原文依据 we propose DeepAlign, a novel multimodal alignment framework to mitigate modality conflict, which employs representation intervention and structure-induced knowledge distillation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）在增强大语言模型理解视觉输入的能力方面已展现出可观的进展。然而，视觉与文本之间的模态错位仍是 MLLM 中的关键挑战，这可归因于两个方面：模态特定表征的错位以及模态特定细节的流失。为解决模态错位问题，我们提出 DeepAlign，一种新的多模态对齐框架，用以缓解模态冲突；该框架采用表征干预与结构诱导的知识蒸馏，以防止模态特定信息的错位与流失。大量实验表明，DeepAlign 显著缓解了模态冲突，在多个视觉-语言任务上相较骨干模型取得了大幅性能提升。它同时激发了 MLLM 的一些涌现能力，例如在交错文本-图像序列上的多模态上下文学习。
英文原文摘要 1009 字符
Multimodal Large Language Models (MLLMs) have demonstrated promising advancements in augmenting the capabilities of LLMs to comprehend visual input. However, modality misalignment between vision and text remains a key challenge in MLLM, which can be attributed to two aspects: misalignment of modality-specific representations and depletion of modality-specific details. To address the issue of modality misalignment, we propose DeepAlign, a novel multimodal alignment framework to mitigate modality conflict, which employs representation intervention and structure-induced knowledge distillation to prevent the misalignment and depletion of modality-specific information. Extensive experiments demonstrate that DeepAlign significantly mitigates modality conflicts, leading to substantial performance improvements compared to backbone models across multiple vision-language tasks. It also stimulates some emergent abilities in MLLMs, such as multimodal in-context learning on interleaved text-image sequences.