PAVAS：物理感知的视频到音频合成
PAVAS: Physics-Aware Video-to-Audio Synthesis
把 VLM 估计的物体质量与基于分割的动态三维重建得到的运动轨迹作为物理条件注入潜变量扩散，为视觉研究者提供了一条「从视频中显式提取物理量再驱动跨模态生成」的可复用管线，也给出了用 APCC 衡量生成结果物理一致性的评测思路。
原文依据 incorporates physical reasoning into a latent diffusion-based V2A generation through the Physics-Driven Audio Adapter
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频到音频（V2A）生成在感知质量与时序同步方面取得了令人瞩目的进展，但多数模型仍由外观驱动，只捕捉视觉与声学之间的相关性，而未考虑塑造真实世界声音的物理因素。本文提出物理感知视频到音频合成方法 PAVAS，通过物理驱动音频适配器（Phy-Adapter）将物理推理引入基于潜变量扩散的 V2A 生成过程。该适配器接收由物理参数估计器（PPE）估计的物体级物理参数：PPE 利用视觉语言模型（VLM）推断运动物体的质量，并借助基于分割的动态三维重建模块恢复其运动轨迹以计算速度。这些物理线索使模型能够合成反映潜在物理因素的声音。为评估物理真实感，我们构建了聚焦物体间交互的基准 VGG-Impact，并提出音频-物理相关系数（APCC），该指标衡量物理属性与听觉属性之间的一致性。大量实验表明，PAVAS 生成的音频在物理上合理且感知上连贯，在定量与定性评估中均优于现有 V2A 模型。详见 https://physics-aware-video-to-audio-synthesis.github.io。
英文原文摘要 1383 字符
Recent advances in Video-to-Audio (V2A) generation have achieved impressive perceptual quality and temporal synchronization, yet most models remain appearance-driven, capturing visual-acoustic correlations without considering the physical factors that shape real-world sounds. We present Physics-Aware Video-to-Audio Synthesis (PAVAS), a method that incorporates physical reasoning into a latent diffusion-based V2A generation through the Physics-Driven Audio Adapter (Phy-Adapter). The adapter receives object-level physical parameters estimated by the Physical Parameter Estimator (PPE), which uses a Vision-Language Model (VLM) to infer the moving-object mass and a segmentation-based dynamic 3D reconstruction module to recover its motion trajectory for velocity computation. These physical cues enable the model to synthesize sounds that reflect underlying physical factors. To assess physical realism, we curate VGG-Impact, a benchmark focusing on object-object interactions, and introduce Audio-Physics Correlation Coefficient (APCC), an evaluation metric that measures consistency between physical and auditory attributes. Comprehensive experiments show that PAVAS produces physically plausible and perceptually coherent audio, outperforming existing V2A models in both quantitative and qualitative evaluations. Visit https://physics-aware-video-to-audio-synthesis.github.io.