将并行序列模型扩展至视觉基础模型
Scaling Parallel Sequence Models to Vision Foundation Models
用 C-GSPN 替换 ViT 中的自注意力块，可在 2D 网格上做线性时间上下文传播并去掉位置编码，配合融合 CUDA kernel 与两阶段蒸馏，让视觉基础模型预训练在同等算力下获得约 2 倍加速；做高分辨率或长序列视觉骨干时可直接复用该算子与蒸馏流程。
原文依据 C-GSPN achieves up to 2x speedup, maintains competitive zero-shot accuracy, and improves segmentation by +2.1%
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论扩展视觉基础模型受限于自注意力的二次复杂度。尽管线性注意力变体与状态空间模型等次二次注意力替代方案成功降低了模型复杂度，但它们通常将图像序列化为 1D 词元序列，损害了空间连贯性与效率。广义空间传播网络（GSPN）提供了一种线性时间的替代方案，通过线扫描传播直接在 2D 网格上传播上下文，并去除了位置嵌入，然而原始设计触及 GPU 扩展瓶颈：增大的批大小/通道数会饱和 SM 并发度，使扫描串行化并导致延迟飙升。我们提出紧凑 GSPN（C-GSPN），一种 ViT 块，它压缩传播空间以保持精度，同时将传播延迟降低近 10 倍。我们进一步通过轻量投影与融合 CUDA 内核提升效率。为支持大规模预训练，我们采用两阶段跨算子蒸馏策略，将逐层监督与端到端对齐相结合。在代表性的 1K 配置（批大小 32，C=1152）下，C-GSPN 实现最高 2 倍加速，保持有竞争力的零样本精度，并将分割性能提升 +2.1%。大量实验与消融表明，所提出的压缩与两阶段蒸馏对于强迁移至关重要，同时大幅降低计算量，首次将次二次算子扩展至基础模型规模（CLIP 风格）的视觉预训练。
英文原文摘要 1506 字符
Scaling vision foundation models is constrained by the quadratic complexity of self-attention. Although subquadratic attention alternatives like linear attention variants and state-space models successfully reduce the model complexity, they typically serialize images into 1D token sequences, compromising spatial coherence and efficiency. Generalized Spatial Propagation Networks (GSPN) offer a linear-time alternative that propagates context directly on the 2D grid via line-scan propagation and removes positional embeddings, yet the original design hits GPU-scaling limits: growing batch/channels saturate SM concurrency, serializing scans, and spiking latency. We introduce Compact GSPN (C-GSPN), a ViT block that compresses the propagation space to preserve accuracy while cutting propagation latency by nearly 10x. We further improve efficiency with lightweight projections and fused CUDA kernels. To enable large-scale pretraining, we adopta two-stage cross-operator distillation strategy that combines layer-wise supervision with end-to-end alignment. In a representative 1K configuration (batch 32, C=1152), C-GSPN achieves up to 2x speedup, maintains competitive zero-shot accuracy, and improves segmentation by +2.1%. Extensive experiments and ablations show that the proposed compression and two-stage distillation are criticalfor strong transfer while substantially reducing compute, enabling the first extension of a subquadratic operator to foundation-scale (CLIP-style) vision pretraining.