FoleyDirector：通过结构化脚本实现视频到音频生成的细粒度时序引导
FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts
FoleyDirector 在 DiT 架构上通过结构化时序脚本和时序脚本注意力实现细粒度时序控制，为视觉研究者提供了一种可复用的时序条件注入机制，尤其适用于多事件或视觉线索不足（如小区域、画外音、遮挡）的视频到音频生成场景。
原文依据 FoleyDirector introduces Structured Temporal Scripts(STS), a set of captions corresponding to short temporal segments, to provide richer temporal information.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频到音频（V2A）方法取得了显著进展，能够合成逼真、高质量的音频。然而，在多事件场景或视觉线索不足（如小区域、画外音、遮挡/部分可见物体）时，它们难以实现细粒度的时序控制。本文提出 FoleyDirector，该框架首次在基于 DiT 的 V2A 生成中实现精确的时序引导，同时保持基础模型的音频质量，并允许在 V2A 生成与时序控制合成之间无缝切换。FoleyDirector 引入结构化时序脚本（STS），即一组对应于短时序片段的描述，以提供更丰富的时序信息。这些特征通过脚本引导的时序融合模块进行整合，该模块采用时序脚本注意力来连贯地融合 STS 特征。为处理复杂的多事件场景，我们进一步提出双帧声音合成，实现帧内与帧外音频的并行生成，并提升可控性。为支持训练与评估，我们构建了 DirectorSound 数据集，并引入 VGGSound-Director 和 DirectorBench。实验表明，FoleyDirector 在保持高音频保真度的同时显著增强了时序可控性，使用户能够扮演 Foley 导演的角色，推动 V2A 向更具表现力和可控性的方向发展。
英文原文摘要 1452 字符
Recent Video-to-Audio (V2A) methods have achieved remarkable progress, enabling the synthesis of realistic, high-quality audio. However, they struggle with fine-grained temporal control in multi-event scenarios or when visual cues are insufficient, such as small regions, off-screen sounds, or occluded/partially visible objects.In this paper, we propose FoleyDirector, a framework that, for the first time, enables precise temporal guidance in DiT-based V2A generation while preserving the base model's audio quality and allowing seamless switching between V2A generation and temporally controlled synthesis. FoleyDirector introduces Structured Temporal Scripts(STS), a set of captions corresponding to short temporal segments, to provide richer temporal information. These features are integrated via the Script-Guided Temporal Fusion Module, which employs Temporal Script Attention to fuse STS features coherently. To handle complex multi-event scenarios, we further propose Bi-Frame Sound Synthesis, enabling parallel in-frame and out-of-frame audio generation and improving controllability.To support training and evaluation, we construct the DirectorSound dataset and introduce VGGSound-Director and DirectorBench. Experiments demonstrate that FoleyDirector substantially enhances temporal controllability while maintaining high audio fidelity, empowering users to act as Foley directors and advancing V2A toward more expressive and controllable.