听见你所见：基于扩散 Transformer 与语义-时序对齐排序直接偏好优化的视频到音频生成
Hear What You See: Video-to-Audio Generation with Diffusion Transformer and Semantic-Temporal Alignment-Ranked Direct Preference Optimization
把视频帧与文本线索作为多模态条件注入 co-attention 扩散 Transformer，并用按语义-时序对齐排序构造偏好对的 DPO 微调扩散采样过程，这套「自动排序候选 + 偏好微调」的流程可直接迁移到其他以视觉为条件的生成任务（如视频生成、可控图像生成）中提升条件对齐度。
原文依据 ranks them based on both semantic and temporal alignment, and subsequently fine-tunes the diffusion model using the derived preference pairs
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论生成既具有语义意义、又与无声视频在时序上同步的高保真音频，仍是视频到音频生成中的一个难题。现有方法往往无法捕捉视觉事件与音频动态之间细粒度的时序对应关系，导致输出不真实或不同步。为解决这些局限，我们提出 VisioSonic，一个统一的视频对齐声音生成框架，将流匹配扩散与偏好引导的对齐相结合。VisioSonic 引入多模态条件模块，联合利用视频帧与文本线索，提供语义层面与帧级的时序引导。一个协同注意力扩散 Transformer 高效融合视觉与音频表征，以极低的计算开销实现内容感知的声音合成。为进一步在监督训练之外增强对齐，我们提出语义-时序对齐排序直接偏好优化（STAR-DPO），这是一种新的偏好学习范式：它自动生成音频候选，依据语义与对齐两方面对候选排序，再利用所得的偏好对微调扩散模型。在多个基准上的大量实验表明，VisioSonic 在实现最优的音视频同步与音频保真度的同时，所用可训练参数在同类方法中最少。项目主页：https://kaiw7.github.io/VisioSonic/
英文原文摘要 1475 字符
Generating high-fidelity audio that is both semantically meaningful and temporally synchronized with silent videos remains a challenging problem in video-to-audio generation. Existing approaches often fail to capture fine-grained temporal correspondence between visual events and audio dynamics, leading to unrealistic or desynchronized outputs. To address these limitations, we propose VisioSonic, a Video-Aligned Sound generation framework that unifies flow-matching diffusion and preference-guided alignment. VisioSonic introduces a multimodal conditioning module that jointly leverages video frames and textual cues to provide semantic and frame-level temporal guidance. A co-attention diffusion transformer efficiently fuses visual and audio representations, enabling content-aware sound synthesis with minimal computation costs. To further enhance alignment beyond supervised training, we introduce Semantic-Temporal Alignment Ranked Direct Preference Optimization (STAR-DPO), a novel preference-learning paradigm that automatically generates audio candidates, ranks them based on both semantic and temporal alignment, and subsequently fine-tunes the diffusion model using the derived preference pairs. Extensive experiments on various benchmarks demonstrate that VisioSonic achieves state-of-the-art audio-video synchronization and audio fidelity while using the fewest trainable parameters among competing approaches. Project page: https://kaiw7.github.io/VisioSonic/