面向视频到语音生成的层级编解码扩散模型
Hierarchical Codec Diffusion for Video-to-Speech Generation
把 RVQ 编解码器的层级结构直接映射为扩散 Transformer 的低层/高层 block，并用唇动、身份、表情分别条件化不同层级，这种「按 token 层级分配视觉条件」的做法可迁移到其他跨模态生成任务（如视频到动作、面部到语音）。
原文依据 we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频到语音（VTS）生成旨在从无声视频中合成语音，无需音频信号，在电影配音和失声者声音恢复等应用中具有广阔前景。然而，现有 VTS 方法忽视了语音的层级特性，即从粗粒度的说话人感知语义到细粒度的韵律细节。这一疏忽阻碍了在属性匹配过程中视觉特征与语音特征在特定层级上的直接对齐。本文利用基于残差向量量化（RVQ）的编解码器的层级结构，提出 HiCoDiT，一种新颖的层级编解码扩散 Transformer，利用离散语音 token 的固有层级实现强音视频对齐。具体而言，由于低层 token 编码粗粒度的说话人感知语义、高层 token 捕捉细粒度韵律，HiCoDiT 采用低层与高层 block 分别生成不同层级的 token。低层 block 以唇同步运动和面部身份为条件来捕捉说话人感知内容，高层 block 则利用面部表情调制韵律动态。最后，为实现更有效的由粗到细条件化，我们提出双尺度自适应实例层归一化，通过通道维归一化捕捉全局声音风格、通过时间维归一化捕捉局部韵律动态。大量实验表明，HiCoDiT 在保真度和表现力上优于基线，凸显了离散建模在 VTS 中的潜力。代码与语音演示见 https://github.com/Jiaxin-Ye/HiCoDiT。
英文原文摘要 1708 字符
Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals, and holds substantial promise for applications such as film dubbing and voice restoration for individuals with aphonia. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalisation and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.