M4-SAM：面向 RGB-D 视频显著性目标检测的多模态混合专家与记忆增强 SAM
M4-SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection
把 SAM2 的线性 LoRA 换成带卷积专家的模态感知 MoE-LoRA，并用伪掩码引导记忆库实现无提示初始化，这套 PEFT + 记忆引导思路可直接迁移到其他需要多模态适配的 SAM2 下游分割任务。
原文依据 we inject Modality-Aware MoE-LoRA, which employs convolutional experts to encode local spatial priors
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论分割一切模型 2（SAM2）已成为通用分割的基础模型。凭借其可泛化的视觉表征，SAM2 已被成功应用于多种下游任务。然而，将 SAM2 扩展到 RGB-D 视频显著性目标检测（RGB-D VSOD）任务面临三项挑战：线性 LoRA 的空间建模能力有限、SAM 多尺度特征利用不足，以及初始化依赖显式提示。为解决这些问题，我们提出多模态混合专家与记忆增强 SAM（M4-SAM），为 SAM2 配备模态相关的参数高效微调、层次化特征融合与无提示记忆初始化。首先，我们在 SAM2 的编码器中注入模态感知 MoE-LoRA，它采用卷积专家编码局部空间先验，并引入模态调度器以实现高效的多模态微调。其次，我们部署门控多级特征融合，以自适应门控机制层次化聚合多尺度编码器特征，从而平衡空间细节与语义上下文。最后，为在无人工提示的情况下实现零样本 VSOD，我们采用伪引导初始化，将粗掩码视为伪先验并用于引导记忆库。大量实验表明，M4-SAM 在三个公开 RGB-D VSOD 数据集上于所有评价指标均取得最优性能。
英文原文摘要 1454 字符
The Segment Anything Model 2 (SAM2) has emerged as a foundation model for universal segmentation. Owing to its generalizable visual representations, SAM2 has been successfully applied to various downstream tasks. However, extending SAM2 to the RGB-D video salient object detection (RGB-D VSOD) task encounters three challenges including limited spatial modeling of linear LoRA, insufficient employment of SAM's multi-scale features, and dependence of initialization on explicit prompts. To address the issues, we present Multi-Modal Mixture-of-Experts with Memory-Augmented SAM (M4-SAM), which equips SAM2 with modality-related PEFT, hierarchical feature fusion, and prompt-free memory initialization. Firstly, we inject Modality-Aware MoE-LoRA, which employs convolutional experts to encode local spatial priors and introduces a modality dispatcher for efficient multi-modal fine-tuning, into SAM2's encoder. Secondly, we deploy Gated Multi-Level Feature Fusion, which hierarchically aggregates multi-scale encoder features with an adaptive gating mechanism, to balance spatial details and semantic context. Finally, to conduct zero-shot VSOD without manual prompts, we utilize a Pseudo-Guided Initialization, where a coarse mask is regarded as a pseudo prior and used to bootstrap the memory bank. Extensive experiments demonstrate that M4-SAM achieves the state-of-the-art performance across all evaluation metrics on three public RGB-D VSOD datasets.