AudioStory：利用大语言模型生成长篇叙事音频
AudioStory: Generating Long-Form Narrative Audio with Large Language Models
AudioStory 将 LLM 的指令分解与扩散模型的音频生成通过解耦桥接机制（桥接查询和残差查询）结合，并实现端到端训练，这种思路可迁移到视觉领域的可控长视频生成，例如利用 LLM 分解复杂叙事为时序子任务，再通过扩散模型生成连贯视频。
原文依据 AudioStory, a unified framework that integrates large language models (LLMs) with TTA systems to generate structured, long-form audio narratives.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期文本到音频（TTA）生成进展在合成短音频片段方面表现出色，但在处理长篇叙事音频时仍面临困难，后者需要时序连贯性和组合推理能力。为填补这一空白，我们提出 AudioStory，一个将大语言模型（LLM）与 TTA 系统集成以生成结构化长篇音频叙事的统一框架。AudioStory 具备强大的指令遵循推理生成能力。它利用 LLM 将复杂叙事查询分解为具有上下文线索的时序有序子任务，从而实现连贯的场景转换和情感基调一致性。AudioStory 有两个吸引人的特点：（1）解耦桥接机制：AudioStory 将 LLM-扩散器协作解耦为两个专门组件，即用于事件内语义对齐的桥接查询和用于事件间连贯性保持的残差查询。（2）端到端训练：通过在单一端到端框架内统一指令理解和音频生成，AudioStory 消除了模块化训练流程的需求，同时增强了组件间的协同作用。此外，我们建立了一个基准 AudioStory-10K，涵盖动画音景和自然声音叙事等多个领域。大量实验表明，AudioStory 在单音频生成和叙事音频生成方面均具有优越性，在指令遵循能力和音频保真度上均超越了先前的 TTA 基线。我们的代码和数据集将公开可用。
英文原文摘要 1541 字符
Recent advances in text-to-audio (TTA) generation excel at synthesizing short audio clips but struggle with long-form narrative audio, which requires temporal coherence and compositional reasoning. To fill this gap, we propose AudioStory, a unified framework that integrates large language models (LLMs) with TTA systems to generate structured, long-form audio narratives. AudioStory possesses strong instruction-following reasoning generation capabilities. It employs LLMs to decompose complex narrative queries into temporally ordered sub-tasks with contextual cues, enabling coherent scene transitions and emotional tone consistency. AudioStory has two appealing features: (1) Decoupled bridging mechanism: AudioStory disentangles LLM-diffuser collaboration into two specialized components, i.e., a bridging query for intra-event semantic alignment and a residual query for inter-event coherence preservation. (2) End-to-end training: By unifying instruction comprehension and audio generation within a single end-to-end framework, AudioStory eliminates the need for modular training pipelines while enhancing synergy between components. Furthermore, we establish a benchmark AudioStory-10K, encompassing diverse domains such as animated soundscapes and natural sound narratives. Extensive experiments show the superiority of AudioStory on both single-audio generation and narrative audio generation, surpassing prior TTA baselines in both instruction-following ability and audio fidelity. Our code and dataset will be publicly available.