用于语义场景补全的多模态频率分解网络
Multi-modal Frequency Decomposition Network for Semantic Scene Completion
针对 RGB-D 语义场景补全中的多模态对齐问题，提出频率分解与全局到局部的对齐范式，可复用于其他多模态融合任务（如深度补全、3D 检测）的特征对齐与细节保留。
原文依据 By integrating frequency processing with limited layers of convolution and downsampling, MFDNet achieves a balance between modalities alignment and detail retainment.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于 RGB-D 图像对，语义场景补全（SSC）通过预测 3D 语义占用图来提供对 3D 场景理解的描述。近期方法提取 RGB-D 多模态特征并在空间域融合，忽视了不完美原始多模态数据导致的多模态特征学习中的错位问题。此外，它们所采用的高层特征提取操作容易引入特征平滑和细节丢失，加剧了上述错位。为解决这些问题，本文提出 MFDNet，一种基于多模态频率分解策略的轻量级语义场景补全网络。通过将频率处理与有限层数的卷积和下采样相结合，MFDNet 在模态对齐和细节保留之间实现了平衡。该网络配备了多模态自适应频率融合（MAFF）和频率细节补偿（FDC）。MAFF 从全局角度建模模态内多频带依赖和模态间关系，在实现模态特定校准的同时促进多模态特征的对齐融合。FDC 挖掘浅层特征中的高频线索，以补偿融合特征缺失的局部细节，并实现用于补全的细粒度对齐。MAFF 和 FDC 为多模态 SSC 构建了从全局到局部的对齐与补全范式。大量实验表明，MFDNet 在 NYUv2 和 NYUCAD 数据集上实现了最优性能，同时减少了 54.4% 的参数量。
英文原文摘要 1593 字符
Based on an RGB-D image pair, semantic scene completion (SSC) provides a description for 3D scene understanding by predicting 3D semantic occupancy map. Recent methods extract RGB-D multi-modal features and fuse them in spatial domain, which disregards the misalignment caused by the imperfect raw multi-modal data and the multi-modal feature learning. Moreover, the operations of extracting high-level features they utilized tend to introduce feature smoothing and detail loss, exacerbating the above misalignment. To tackle these problems, this paper introduces MFDNet, a lightweight semantic scene completion network based on a multi-modal frequency decomposition strategy. By integrating frequency processing with limited layers of convolution and downsampling, MFDNet achieves a balance between modalities alignment and detail retainment. The network is equipped with Multi-modal Adaptive Frequency Fusion (MAFF) and Frequency Detail Compensation (FDC). MAFF models the intra-modal multi-bands dependencies and inter-modal relationships from a global perspective, enabling modality-specific calibration while facilitating the aligned fusion of multi-modal features. FDC excavates the high-frequency cues in shallow features to compensate for the missing local details of the fused feature and achieve fine-grained alignment for completion. MAFF and FDC formulate a global-to-local alignment and completion paradigm for multi-modal SSC. Extensive experiments demonstrate that MFDNet reduces parameters by 54.4% while achieving state-of-the-art performance on the NYUv2 and NYUCAD datasets.