医学影像与生物视觉 · CVPR2026 ·2026.06.03

MUST:面向缺失模态扩散增强生存预测的模态特异表征感知 Transformer

MUST: Modality-Specific Representation-Aware Transformer for Diffusion-Enhanced Survival Prediction with Missing Modality

Kyungwon Kim、Dosik Hwang

与你有关的部分

把每个模态的表征显式拆成模态特异分量与跨模态上下文分量,再用条件潜变量扩散模型补全缺失模态的特异信息——这套「分解+条件生成补全」思路可直接迁移到视觉多模态任务(如RGB-深度、图像-文本)中的模态缺失场景。

原文依据 we employ conditional latent diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors
可迁移到 多模态生存预测缺失模态补全医学图像分类

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

从多模态医学数据中进行准确的生存预测对精准肿瘤学至关重要,然而临床部署面临一个长期挑战:由于成本限制、技术局限或回顾性数据可得性,模态常常是不完整的。尽管近期方法试图通过特征对齐或联合分布学习来应对缺失模态,但它们从根本上缺乏对每个模态独特贡献的显式建模——即该模态相对于可从其他模态推导出的信息而言所独有的部分。我们提出了一个新框架,在学习到的低秩共享子空间中通过代数约束,将每个模态的表征显式分解为模态特异分量与跨模态上下文分量。这一分解使我们能够精确识别当某个模态缺失时究竟丢失了哪些信息。对于无法从可用模态推断的真正模态特异信息,我们采用条件潜变量扩散模型,以恢复的共享信息和学习到的结构先验为条件,生成高质量表征。在五个 TCGA 癌症数据集上的大量实验表明,所提方法在完整数据下达到最优性能,并在病理缺失与基因组缺失两种条件下均保持稳健的预测。

英文原文摘要 1362 字符

Accurate survival prediction from multimodal medical data is essential for precision oncology, yet clinical deployment faces a persistent challenge: modalities are frequently incomplete due to cost constraints, technical limitations, or retrospective data availability. While recent methods attempt to address missing modalities through feature alignment or joint distribution learning, they fundamentally lack explicit modeling of the unique contributions of each modality as opposed to the information derivable from other modalities. We propose a novel framework that explicitly decomposes each modality's representation into modality-specific and cross-modal contextualized components through algebraic constraints in a learned low-rank shared subspace. This decomposition enables us to identify precisely what information is missing when a modality is absent. For the truly modality-specific information that cannot be inferred from available modalities, we employ conditional latent diffusion models to generate high-quality representations conditioned on recovered shared information and learned structural priors. Extensive experiments on five TCGA cancer datasets demonstrate that the proposed method achieves state-of-the-art performance with complete data while maintaining robust predictions in both missing pathology and missing genomics conditions.

标签

会议 CVPR2026
页码 30312-30321
Bibkey
来源 CVF Open Access