没有图像也能学会看？视觉Transformer的程序化预热
Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
提出用程序化生成的无视觉语义数据对ViT进行预热预训练，绕过图像块嵌入，可作为一种新的数据高效预训练策略，用于提升下游视觉任务的数据效率和收敛速度。
原文依据 allocating just 1% of the training budget to procedural data improves final accuracy by over 1.7%
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Transformer 具有极强的通用性，暗示存在跨模态有益的通用归纳偏置。在本工作中，我们探索了一种新方法，通过在无视觉或语义内容的程序化生成数据上进行预训练，将此类偏置注入视觉 Transformer（ViT）。我们使用形式文法之类的简单算法生成这些数据，因此结果与自然图像或合成图像均无关联。我们利用这种程序化生成的数据在预热阶段预训练 ViT，该阶段绕过了其视觉图像块嵌入机制，从而鼓励模型内化抽象的计算先验。随后进行标准的基于图像的训练时，这种预热显著提升了数据效率、收敛速度和下游性能。例如，在 ImageNet-1K 上，仅将 1% 的训练预算分配给程序化数据，最终准确率就提升了超过 1.7%。就性能影响而言，1% 的程序化生成数据相当于 28% 的 ImageNet-1K 数据。这些发现指向了一条通往新的数据高效且领域无关的预训练策略的有前景的道路。
英文原文摘要 1154 字符
Transformers are remarkably versatile, suggesting the existence of generic inductive biases beneficial across modalities. In this work, we explore a new way to instil such biases in vision transformers (ViTs) through pretraining on procedurally generated data devoid of visual or semantic content. We generate this data with simple algorithms such as formal grammars, so the results bear no relationship to either natural or synthetic images. We use this procedurally generated data to pretrain ViTs in a warm-up phase that bypasses their visual patch embedding mechanisms, thus encouraging the models to internalise abstract computational priors. When followed by standard image-based training, this warm-up significantly improves data efficiency, convergence speed, and downstream performance. On ImageNet-1K, for example, allocating just 1% of the training budget to procedural data improves final accuracy by over 1.7%. In terms of its effect on performance, 1% procedurally generated data is thus equivalent to 28% of the ImageNet-1K data. These findings suggest a promising path toward new data-efficient and domain-agnostic pretraining strategies.