渐进式超网络训练用于高效视觉自回归建模
Progressive Supernet Training for Efficient Visual Autoregressive Modeling
提出一种基于超网络的动态深度调整方法，通过渐进式训练策略在单个模型内实现不同深度的子网络，从而在视觉自回归生成中大幅减少KV缓存和推理时间，为资源受限场景下的高效生成提供了可复用的技术方案。
原文依据 we obtain 50% cache reduction and 1.8x inference speedup with minimal quality loss
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉自回归（VAR）模型通过采用“下一尺度”预测范式显著减少了推理步骤，在图像生成方面表现出与扩散模型相竞争的性能。然而，VAR的渐进式多尺度生成由于跨所有尺度的KV缓存累积而导致严重的内存开销，限制了实际部署。现有解决方案要么需要训练和部署多个专用模型，要么牺牲生成质量。我们观察到VAR中一个关键的尺度-深度非对称依赖关系：小尺度（低分辨率token）对网络深度高度敏感，需要深层来捕获全局语义信息，而大尺度（高分辨率token）对深度减少表现出显著的鲁棒性。受此启发，我们提出了VARiant，一个统一的超网络框架，通过参数共享在单个模型内实现动态深度调整。我们的方法采用均匀间隔层选择策略来构建保持质量的子网络，并引入动态比例渐进训练策略，从联合优化（全网络与子网络比例2:8）逐渐过渡到子网络优化（比例10:0），有效解决了超网络训练中全网络与子网络之间固有的优化冲突。在ImageNet上的大量实验表明，我们的方法在生成质量和推理效率之间实现了帕累托最优权衡：通过前7个尺度使用全深度（30层），后续尺度使用16层子网络（50%深度），我们获得了50%的缓存减少和1.8倍的推理加速，而质量损失极小（FID仅增加0.3）。与需要部署多个模型的方法不同，我们的单模型解决方案消除了部署复杂性，支持零成本的运行时深度切换，并无缝集成到标准Transformer推理框架中，使其在资源受限场景下具有高度实用性。
英文原文摘要 2038 字符
Visual Autoregressive (VAR) models have demonstrated competitive performance with diffusion models in image generation by adopting a "next-scale" prediction paradigm that significantly reduces inference steps. However, VAR's progressive multi-scale generation leads to severe memory overhead due to KV cache accumulation across all scales, limiting practical deployment. Existing solutions either require training and deploying multiple specialized models or sacrifice generation quality.We observe a critical scale-depth asymmetric dependency in VAR: small scales (low-resolution tokens) are highly sensitive to network depth and require deep layers to capture global semantic information, while large scales (high-resolution tokens) exhibit remarkable robustness to depth reduction.Motivated by this insight, we propose VARiant, a unified supernet framework that enables dynamic depth adjustment within a single model through parameter sharing. Our approach employs an even-spacing layer selection strategy to construct quality-preserving subnetworks, and introduces a dynamic-ratio progressive training strategy that gradually transitions from joint optimization (full network to subnetwork ratio 2:8) to subnetwork optimization (ratio 10:0), effectively resolving the inherent optimization conflicts between full network and subnetworks in supernet training.Extensive experiments on ImageNet demonstrate that our method achieves Pareto-optimal trade-offs between generation quality and inference efficiency: by using full depth (30 layers) for the first 7 scales and a 16-layer subnetwork (50% depth) for subsequent scales, we obtain 50% cache reduction and 1.8x inference speedup with minimal quality loss (FID increases by only 0.3).Unlike approaches requiring deployment of multiple models, our single-model solution eliminates deployment complexity, supports zero-cost runtime depth switching, and seamlessly integrates into standard transformer inference frameworks, making it highly practical for resource-constrained scenarios.