Sketch2CT：面向结构感知三维医学体数据生成的多模态扩散模型
Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation
把「2D 草图 + 文本」作为多模态条件，先由噪声生成 3D 器官分割掩码、再以其引导潜变量扩散模型合成 3D CT 体数据，这套「先结构后外观」的两阶段条件生成流程可直接迁移到其他需要解剖结构约束的医学体数据合成与数据增广任务。
原文依据 The synthesized segmentation masks subsequently guide a latent diffusion model for 3D CT volume synthesis
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩散概率模型在生成高质量、逼真的医学图像方面已展现出显著潜力，为解决医学领域长期存在的数据稀缺问题提供了有前景的方案。然而，在多模态条件下生成具有解剖学一致结构的三维医学体数据仍是一个复杂且尚未解决的问题。本文提出 Sketch2CT，一个用于结构感知三维医学体数据生成的多模态扩散框架，由用户提供的二维草图与刻画三维几何语义的文本描述共同引导。该框架首先以这两种模态为条件，从随机噪声生成目标器官的三维分割掩码。为有效对齐并融合这些输入，我们提出两个关键模块：一个利用局部化文本线索细化草图特征，另一个整合全局的草图—文本表示。这两个模块构建于胶囊注意力骨干之上，充分利用草图与文本的互补优势，生成解剖学上准确的器官形状。随后，合成的分割掩码引导潜变量扩散模型进行三维 CT 体数据合成，从而实现对与用户指定草图及描述相一致的器官外观的逼真重建。在公开 CT 数据集上的大量实验表明，Sketch2CT 在多模态医学体数据生成方面取得了更优的性能。其可控、低成本的生成流程能够对医学数据集进行有原则且高效的增广。代码已发布于 https://github.com/adlsn/Sketch2CT。
英文原文摘要 1592 字符
Diffusion probabilistic models have demonstrated significant potential in generating high-quality, realistic medical images, providing a promising solution to the persistent challenge of data scarcity in the medical field. Nevertheless, producing 3D medical volumes with anatomically consistent structures under multimodal conditions remains a complex and unresolved problem. We introduce Sketch2CT, a multimodal diffusion framework for structure-aware 3D medical volume generation, jointly guided by a user-provided 2D sketch and a textual description that captures 3D geometric semantics. The framework initially generates 3D segmentation masks of the target organ from random noise, conditioned on both modalities. To effectively align and fuse these inputs, we propose two key modules that refine sketch features with localized textual cues and integrate global sketch-text representations. Built upon a capsule-attention backbone, these modules leverage the complementary strengths of sketches and text to produce anatomically accurate organ shapes. The synthesized segmentation masks subsequently guide a latent diffusion model for 3D CT volume synthesis, enabling realistic reconstruction of organ appearances that are consistent with user-defined sketches and descriptions. Extensive experiments on public CT datasets demonstrate that Sketch2CT achieves superior performance in generating multimodal medical volumes. Its controllable, low-cost generation pipeline enables principled, efficient augmentation of medical datasets. Code is available at https://github.com/adlsn/Sketch2CT.