VidEoMT：你的 ViT 其实也是视频分割模型
VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
把视频分割的时序建模从专用跟踪模块转移到 ViT 编码器内部：复用上一帧的 query 做传播、再与时序无关的可学习 query 融合，可直接迁移到任何基于 query 的视频分割/跟踪管线，用极简结构换取 5-10 倍加速。
原文依据 VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有的在线视频分割模型通常将逐帧分割器与复杂的专用跟踪模块相结合。这类方法虽然有效，但会带来显著的架构复杂度和计算开销。近期研究表明，plain Vision Transformer（ViT）编码器在具备足够容量并经过大规模预训练后，无需专用模块即可完成准确的图像分割。受此观察启发，我们提出 Video Encoder-only Mask Transformer（VidEoMT），一种无需专用跟踪模块的简洁纯编码器视频分割模型。为在纯编码器 ViT 中实现时序建模，VidEoMT 引入轻量级的查询传播机制，通过复用上一帧的查询在帧间传递信息。为在这一点与对新内容的适应性之间取得平衡，它采用查询融合策略，将传播得到的查询与一组时序无关的可学习查询相结合。由此，VidEoMT 在不增加复杂度的前提下获得了跟踪器的优势，在保持有竞争力精度的同时速度提升 5 至 10 倍，以 ViT-L 骨干网络运行时可达 160 FPS。代码：https://www.tue-mps.org/videomt/。
英文原文摘要 1278 字符
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: \href https://www.tue-mps.org/videomt/ https://www.tue-mps.org/videomt/ .