MagicFuse：用于视觉与语义增强的单图像融合
MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement
利用扩散模型从单张退化可见光图像生成跨光谱场景表示，通过融合两个扩散分支的噪声实现知识级融合，为单模态输入下的多模态融合提供了新思路。
原文依据 MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论本文关注一个高度实用的场景：在仅有可见光成像传感器的苛刻条件下，如何继续受益于多模态图像融合的优势。为实现这一目标，我们提出了单图像融合的新概念，将传统的数据级融合扩展到知识级。具体而言，我们开发了 MagicFuse，一种新颖的单图像融合框架，能够从单张低质量可见光图像中推导出全面的跨光谱场景表示。MagicFuse 首先引入了基于扩散模型的谱内知识强化分支和跨光谱知识生成分支。它们分别挖掘可见光谱中遮蔽的场景信息，并学习迁移到红外光谱的热辐射分布模式。在此基础上，我们设计了一个多域知识融合分支，该分支整合来自这两个分支的扩散流中的概率噪声，通过连续采样从中获得跨光谱场景表示。然后，我们施加视觉和语义约束，以确保该场景表示既能满足人类观察，又能支持下游语义决策。大量实验表明，尽管仅依赖单张退化可见光图像，我们的 MagicFuse 在视觉和语义表示性能上仍可与多模态输入的最优融合方法相媲美甚至更优。代码已公开于 https://github.com/zhayanping/MagicFuse。
英文原文摘要 1601 字符
This paper focuses on a highly practical scenario: how to continue benefiting from the advantages of multi-modal image fusion under harsh conditions when only visible imaging sensors are available. To achieve this goal, we propose a novel concept of single image fusion, which extends conventional data-level fusion to the knowledge level. Specifically, we develop MagicFuse, a novel single image fusion framework capable of deriving a comprehensive cross-spectral scene representation from a single low-quality visible image. MagicFuse first introduces an intra-spectral knowledge reinforcement branch and a cross-spectral knowledge generation branch based on the diffusion models. They mine scene information obscured in the visible spectrum and learn thermal radiation distribution patterns transferred to the infrared spectrum, respectively. Building on them, we design a multi-domain knowledge fusion branch that integrates the probabilistic noise from the diffusion streams of these two branches, from which a cross-spectral scene representation can be obtained through successive sampling. Then, we impose both visual and semantic constraints to ensure that this scene representation can satisfy human observation while supporting downstream semantic decision-making. Extensive experiments show that our MagicFuse achieves visual and semantic representation performance comparable to or even better than state-of-the-art fusion methods with multi-modal inputs, despite relying solely on a single degraded visible image. The code is publicly available at https://github.com/zhayanping/MagicFuse.