通过隐式模态分解实现多模态大模型的参数高效适配
Parameter-Efficient Adaptation for MLLMs via Implicit Modality Decomposition
把 LoRA 的可学习参数按模态拆成文本专属、非文本专属与模态共享三部分，并加解耦/对齐约束，可直接迁移到视觉-语言模型（如 LLaVA 类）的 PEFT 微调中，缓解文本模态主导更新、视觉表征学不充分的问题。
原文依据 decomposes the learnable parameters into the non-overlapped text-specific, non-text-specific and modality-sharing components
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论参数高效微调（PEFT）已成为将大语言模型（LLM）适配为多模态大语言模型（MLLM）的重要途径，使其能以显著更低的内存与计算开销处理多种模态。然而，现有大多数 PEFT 方法忽视了模态不平衡学习问题，该问题表现为文本模态在参数更新中过度占据主导，从而导致非文本模态学习不足并引发性能下降。为解决该问题，我们提出一种面向 MLLM 的新型参数高效适配方法，即基于 LoRA 的隐式模态分解（IMoD）。该方法首先将可学习参数分解为互不重叠的文本专属、非文本专属与模态共享三部分，从而缓解模态不平衡。为进一步引导这些成分朝特定模态优化，我们提出模态特定解耦约束，用以抑制模态特定参数之间的跨模态干扰；以及模态无关对齐约束，用以促使模态共享成分捕获良好对齐、模态不变的语义。在多种多模态设定与 LLM 架构上的大量实验表明，我们的方法持续带来显著性能提升，尤其是在音视频-文本任务上平均提升 3.3%，且不牺牲参数与推理效率。代码已发布于 https://github.com/mmffzzz/IMoD.git。
英文原文摘要 1581 字符
Parameter-efficient fine-tuning (PEFT) has become a compelling approach for adapting large language models (LLMs) into multimodal large language models (MLLMs), enabling them to handle diverse modalities with substantially lower memory and computational costs. However, most existing PEFT methods neglect the issue of modality-imbalanced learning, which is characterized by the excessive dominance of text modality in updating parameters, thus incurring insufficient learning of non-text modalities and leading to performance degradation. To address this issue, we propose a novel parameter-efficient adaptation method for MLLMs, namely Implicit Modality Decomposition (IMoD), based on LoRA. It firstly decomposes the learnable parameters into the non-overlapped text-specific, non-text-specific and modality-sharing components, thereby alleviating modality imbalance. To further guide the optimization of these components toward specific modalities, we propose Modality-Specific Decoupling Constraint that suppresses cross-modal interference among modality-specific parameters, and Modality-Agnostic Alignment Constraint that encourages modality-sharing component to capture well-aligned, modality-invariant semantics. Extensive experiments across diverse multimodal settings and LLM architectures demonstrate that our method consistently delivers significant performance gains, particularly achieving an averaged 3.3% improvements on the audio-visual-text tasks without sacrificing the parameter and inference efficiency. Code is available at https://github.com/mmffzzz/IMoD.git.