时间回响：解锁视频到音频生成模型中的长度泛化能力
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
论文提出的非因果 Mamba 层次化架构，为视觉研究者提供了一种在测试时把短视频训练模型外推到长视频序列的思路，可直接迁移到长视频生成、长时序多模态对齐等需要长度泛化的场景。
原文依据 we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩展视频与音频之间的多模态对齐具有挑战性，这尤其源于数据有限以及文本描述与帧级视频信息之间的不匹配。在本工作中，我们应对多模态到音频生成中的扩展挑战，考察在短样本上训练的模型能否在测试时泛化到更长的样本。为应对这一挑战，我们提出多模态层次化网络，称为 MMHNet，它是对最优视频到音频模型的增强扩展。我们的方法整合了层次化方法与非因果 Mamba，以支持长时音频生成。所提方法显著提升了长音频生成能力，可生成长达 5 分钟以上的音频。我们还证明了在视频到音频生成任务中，无需在更长时长上训练，即可实现短时训练、长时测试。我们在实验中表明，所提方法在长视频到音频基准上取得了显著结果，超越了视频到音频任务中的先前工作。此外，我们展示了模型生成超过 5 分钟音频的能力，而先前的视频到音频方法在生成长时长音频时表现不足。项目主页：https://echoesovertime.github.io
英文原文摘要 1228 字符
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations. Our project page: https://echoesovertime.github.io