InstantViR:基于蒸馏扩散先验的实时视频逆问题求解器
InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior
把双向视频扩散教师蒸馏成因果自回归学生,单次前向即可完成视频修复,无需测试时迭代优化;这套「先验驱动蒸馏 + 轻量变分自编码器」的范式可直接迁移到其他需要实时推理的扩散先验逆问题求解器。
原文依据 We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频逆问题(如修复补全、去模糊和超分辨率)是流媒体、远程呈现和 AR/VR 的基础,在这些场景中,高感知质量必须与严格的延迟约束共存。基于扩散的先验目前能给出最优的重建结果,但现有方法要么用临时设计的时序正则项改造图像扩散模型——导致时序伪影——要么依赖原生视频扩散模型,而其迭代后验采样对于实时使用而言过慢。我们提出 InstantViR,一个由预训练视频扩散先验驱动的、用于超快速视频重建的摊销推理框架。我们将一个强大的双向视频扩散模型(教师)蒸馏为一个因果自回归学生模型,该学生模型在单次前向传播中把退化视频直接映射为其复原版本,继承教师强大的时序建模能力,同时完全去除测试时的迭代优化。该蒸馏是先验驱动的:它只需要教师扩散模型和已知的退化算子,不依赖外部成对的干净/含噪视频数据。为进一步提升吞吐量,我们用高效的 LeanVAE 替换视频扩散骨干网络中的标准变分自编码器,从而实现低延迟的潜空间处理。在流式随机修复补全、高斯去模糊和超分辨率任务上,InstantViR 的重建质量与基于扩散的基线相当或更优,同时在 NVIDIA A100 GPU 上以超过 35 FPS 运行,相比迭代视频扩散先验最高实现 100 倍加速。这些结果表明,基于扩散的视频重建与实时、交互式、可编辑的流式场景相兼容,使高质量视频复原成为现代视觉系统中的实用组件。
英文原文摘要 1810 字符
Video inverse problems such as inpainting, deblurring and super-resolution are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers--leading to temporal artifacts--or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the standard VAE in video diffusion backbone with a highly efficient LeanVAE, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100x speedups over iterative video diffusion priors. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.