生成模型与扩散 · CVPR2026 ·2026.06.03

SounDiT:地理上下文声景到景观生成

SounDiT: Geo-Contextual Soundscape-to-Landscape Generation

Junbo Wang、Haofeng Tan、Bowen Liao、Albert Jiang、Teng Fei、Qixing Huang 等 10 人

与你有关的部分

把环境声景作为条件、地理上下文作为先验,用 DiT 生成地理上合理的景观图像;其 PSS 评测框架(元素/场景/人类感知三级一致性)可迁移到其他跨模态生成任务的条件一致性度量。

原文依据 we propose SounDiT, a diffusion transformer (DiT)-based model that incorporates acoustic environments and geo-contextual scene conditioning
可迁移到 audio-to-image generationconditional image synthesiscross-modal generationtext-to-image generation

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

近期的音频到图像模型在以对应声音为条件生成特定物体图像方面表现出色。然而,这些模型无法以声学环境为条件重建真实世界的景观。为应对这一挑战,我们提出地理上下文声景到景观(GeoS2L)生成,这是一项新颖且具有实际意义的任务,旨在从环境声景合成地理上真实的景观图像。为支撑该任务,我们构建了两个大规模地理上下文多模态数据集 SoundingSVI 与 SonicUrban,将多样化的环境声景与真实世界景观图像配对。我们进一步提出 SounDiT,一个基于扩散 Transformer(DiT)的模型,融合声学环境与地理上下文场景条件以合成地理上连贯的景观图像。此外,我们提出地点相似度分数(PSS),一个面向实际应用的地理上下文评测框架,用于衡量输入声景与生成景观图像之间的一致性。大量实验表明,SounDiT 在 GeoS2L 任务上优于现有基线,同时 PSS 能有效捕捉元素、场景与人类感知等多层次的生成一致性。

英文原文摘要 1289 字符

Recent audio-to-image models have shown impressive performance in generating images of specific objects conditioned on their corresponding sounds. However, these models fail to reconstruct real-world landscapes conditioned on acoustic environments. To address this challenge, we present Geo-contextual Soundscape-to-Landscape (GeoS2L) generation, a novel and practically significant task that aims to synthesize geographically realistic landscape images from environmental soundscapes. To support this task, we construct two large-scale geo-contextual multi-modal datasets, SoundingSVI and SonicUrban, which pair diverse environmental soundscapes with real-world landscape images. We further propose SounDiT, a diffusion transformer (DiT)-based model that incorporates acoustic environments and geo-contextual scene conditioning to synthesize geographically coherent landscape images. Furthermore, we propose the Place Similarity Score (PSS), a practically-informed geo-contextual evaluation framework to measure consistency between input soundscapes and generated landscape images. Extensive experiments demonstrate that SounDiT outperforms existing baselines in the GeoS2L, while the PSS effectively captures multi-level generation consistency across element, scene,and human perception.

标签

会议 CVPR2026
页码 32659-32670
Bibkey
来源 CVF Open Access