EchoFoley：以事件为中心的层次化控制用于视频接地创意音效生成
EchoFoley: Event-Centric Hierarchical Control for Video Grounded Creative Sound Generation
提出以事件为中心的符号化表示（何时、何声、如何发声）作为细粒度控制信号，可迁移到视频生成中的局部事件控制；其慢快思维智能体框架为多模态条件平衡提供新思路。
原文依据 we introduce EchoFoley (Event-Centric Hierarchical cOntrol Foley), a new task designed for video-grounded sound generation with both event-level local control and hierarchical semantic control.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论音效构建了多模态叙事的关键层次，塑造视频的情感氛围与叙事语义。尽管视频-文本到音频（VT2A）近期取得进展，当前范式仍面临三个关键局限：（1）视觉与文本条件之间的不平衡导致视觉主导；（2）缺乏对细粒度可控生成的具体定义；（3）指令理解与跟随能力弱，因现有数据集依赖简短类别标签。为解决这些局限，我们引入EchoFoley（以事件为中心的层次化控制音效），这是一个新任务，旨在实现兼具事件级局部控制与层次化语义控制的视频接地音效生成。我们用于发声事件的符号化表示指定了视频或指令中每个声音产生的时间、内容与方式，从而支持声音生成、插入与编辑等细粒度控制。为支持该任务，我们构建了EchoFoley-6k，一个大规模、专家整理的基准，包含超过6,000个视频-指令-标注三元组和42,000个细粒度发声事件标注。在此基础上，我们提出EchoVidia，一个以发声事件为中心的智能体生成框架，采用慢快思维策略。实验表明，EchoVidia在可控性上超越近期VT2A模型40.7%，在感知质量上超越12.5%。
英文原文摘要 1421 字符
Sound effects build an essential layer of multimodal storytelling, shaping the emotional atmosphere and the narrative semantics of videos. Despite recent advancement in video-text-to-audio (VT2A), the current formulation faces three key limitations: (1) an imbalance between visual and textual conditioning that leads to visual dominance; (2) the absence of a concrete definition for fine-grained controllable generation; (3) weak instruction understanding and following, as existing datasets rely on brief categorical tags. To address these limitations, we introduce EchoFoley (Event-Centric Hierarchical cOntrol Foley), a new task designed for video-grounded sound generation with both event-level local control and hierarchical semantic control. Our symbolic representation for sounding events specifies when, what, and how each sound is produced within a video or instruction, enabling fine-grained controls like sound generation, insertion, and editing. To support this task, we construct EchoFoley-6k, a large-scale, expert-curated benchmark containing over 6,000 video-instruction-annotation triplets and 42,000 fine-grained sounding event annotations.Building upon this foundation, we propose EchoVidia, a sounding-event-centric agentic generation framework with slow-fast thinking strategy. Experiments show that EchoVidia surpasses recent VT2A models by 40.7% in controllability and 12.5% in perceptual quality.