面向高效跨模态迁移的解耦可复用适配
Decoupled and Reusable Adaptation for Efficient Cross-Modal Transfer
把跨模态迁移拆成「一次性任务无关的模态知识迁移（自监督 + 结构重建得到可复用 LoRA）」与「轻量任务知识注入（Task-Prompted MoE 融合）」两段，视觉研究者可把 Stage 1 的模态 LoRA 当作可插拔资产，在深度/事件/热成像等非 RGB 输入上复用同一套 RGB 基础模型，避免每个下游任务重训重存。
原文依据 decouples the process into a one-time general modality knowledge transfer and a flexible task knowledge transfer
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论跨模态迁移方法在将基于 RGB 的基础模型扩展到非 RGB 模态方面已取得显著进展。然而，现有迁移范式主要以任务为导向，这意味着更换任务就需要重新训练和重新存储，从而在数据、计算和存储上造成大量冗余。为解决这一局限，我们提出一种高效的跨模态迁移范式，将整个过程解耦为一次性的通用模态知识迁移与灵活的任务知识迁移。在第一阶段，我们提出渐进式自监督调优策略，将模态感知的结构重建与语义判别学习相结合，仅使用无标注数据、通过一次训练过程即可实现与任务无关的模态知识学习，从而得到可复用的目标模态 LoRA。在第二阶段，我们引入模态 LoRA，并进一步提出任务提示的模态专家混合模块。该设计在实现轻量级任务知识注入的同时，能够在多模态融合过程中为多样化的下游任务有效平衡任务特定知识、模态通用知识与模态特定知识。在六个跨模态迁移场景上的大量实验，以及对数据、计算和存储效率的分析，验证了本方法的优越性。
英文原文摘要 1356 字符
Cross-modal transfer methods have achieved significant progress in extending RGB-based foundation models to non-RGB modalities. However, existing transfer paradigms are primarily task-oriented, meaning that changing tasks requires re-training and re-storing, leading to substantial redundancy in data, computation and storage. To address this limitation, we propose an efficient cross-modal transfer paradigm that decouples the process into a one-time general modality knowledge transfer and a flexible task knowledge transfer. In Stage 1, we propose a Progressive Self-Supervised Tuning strategy that integrates modality-aware structural reconstruction with semantic discriminative learning, which enables task-agnostic modality knowledge learning using only unlabeled data through a one-time training process, resulting in reusable target-modality LoRAs. In Stage 2, we incorporate the modality LoRAs and further propose a Task-Prompted Mixture-of-Modality Experts module. This design enables lightweight task knowledge injection while effectively balancing task-specific, modality-general and modality-specific knowledge in multimodal fusion process for diverse downstream tasks. Extensive experiments across six cross-modal transfer scenarios, along with analyses of data, computation, and storage efficiency, demonstrate the superiority of our method.