细节决定成败:通过关键帧驱动的细节注入增强视频虚拟试穿
The devil is in the details: Enhancing Video Virtual Try-On via Keyframe-Driven Details Injection
提出关键帧驱动的细节注入策略,在不修改 DiT 架构的前提下将服装动态与背景一致性信息注入标准 DiT 块,为视频生成中的细粒度控制与背景保持提供可复用方案。
原文依据 keyframe-driven details injection strategy, motivated by the fact that keyframes inherently contain both foreground dynamics and background consistency
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管基于扩散 Transformer(DiT)的视频虚拟试穿(VVT)在合成逼真视频方面取得了显著进展,现有方法仍难以捕捉细粒度的服装动态并在视频帧间保持背景完整性。此外,由于在 DiT 中引入了额外的交互模块,计算成本高昂,而现有公开数据集规模和质量有限,也制约了模型的泛化能力和有效训练。为解决这些挑战,我们提出了一个新颖的框架 KeyTailor,以及一个大规模高清数据集 ViT-HD。KeyTailor 的核心思想是关键帧驱动的细节注入策略,其动机在于关键帧本身同时包含前景动态和背景一致性。具体而言,KeyTailor 采用指令引导的关键帧采样策略从输入视频中筛选信息丰富的帧。随后,两个定制化的关键帧驱动模块——服装细节增强模块和协同背景优化模块——被用于将服装动态蒸馏到服装相关的潜在表示中,并优化背景潜在表示的完整性,两者均以关键帧为指导。这些丰富的细节随后与姿态、掩码和噪声潜在表示一起被注入标准 DiT 块,从而实现高效且逼真的试穿视频合成。该设计在无需显式修改 DiT 架构的情况下确保了一致性,同时避免了额外的复杂性。此外,我们的数据集 ViT-HD 包含 15,070 个高质量视频样本,分辨率为 810 x 1080,覆盖多样化的服装。大量实验表明,KeyTailor 在动态和静态场景下的服装保真度和背景完整性方面均优于最优基线。数据集和代码将公开释放。
英文原文摘要 1894 字符
Although diffusion transformer (DiT)-based video virtual try-on (VVT) has made significant progress in synthesizing realistic videos, existing methods still struggle to capture fine-grained garment dynamics and preserve background integrity across video frames. They also incur high computational costs due to additional interaction modules introduced into DiTs, while the limited scale and quality of existing public datasets also restrict model generalization and effective training. To address these challenges, we propose a novel framework, KeyTailor, along with a large-scale, high-definition dataset, ViT-HD. The core idea of KeyTailor is a keyframe-driven details injection strategy, motivated by the fact that keyframes inherently contain both foreground dynamics and background consistency. Specifically, KeyTailor adopts an instruction-guided keyframe sampling strategy to filter informative frames from the input video. Subsequently, two tailored keyframe-driven modules--the garment details enhancement module and the collaborative background optimization module--are employed to distill garment dynamics into garment-related latents and to optimize the integrity of background latents, both guided by keyframes. These enriched details are then injected into standard DiT blocks together with pose, mask, and noise latents, enabling efficient and realistic try-on video synthesis. This design ensures consistency without explicitly modifying the DiT architecture, while simultaneously avoiding additional complexity. In addition, our dataset ViT-HD comprises 15,070 high-quality video samples at a resolution of 810 x 1080, covering diverse garments. Extensive experiments demonstrate that KeyTailor outperforms state-of-the-art baselines in terms of garment fidelity and background integrity across both dynamic and static scenarios. The dataset and code will be publicly released.