当 Transformer 遇见 Mamba：一种用于视频目标检测的混合 Transformer-Mamba 网络
When Transformers Meet Mamba: A Hybrid Transformer-Mamba Network for Video Object Detection
将 Mamba 的线性复杂度长序列建模能力与 Transformer 的长程依赖建模能力结合，用于视频目标检测中的跨帧特征聚合；其空间自适应可变形 Transformer 编码器和时序级联双向 Mamba 编码器可作为通用模块迁移到其他需要长时序建模的视频任务中。
原文依据 a temporal cascaded bidirectional Mamba encoder to efficiently capture the long-range dependencies across frames in video sequences with linear complexity
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着 Transformer 的出现，视频目标检测取得了显著进展。尽管 Transformer 擅长建模长程上下文依赖，但其二次复杂度限制了长序列处理中的效率。相比之下，Mamba 在建模长序列时具有更高的效率，但与 Transformer 相比，其上下文学习能力相对有限，且其在视频目标检测中的应用尚未被探索。为了利用 Transformer 和 Mamba 的互补优势，我们提出了一种用于视频目标检测的混合 Transformer-Mamba 网络（TMambaDet），这是该领域的一个开创性框架，结合了 Transformer 的长程建模能力和 Mamba 的高效长序列处理能力。我们的 TMambaDet 具有三个核心组件：1）空间自适应可变形 Transformer 编码器，用于有效建模每帧内的长程依赖，实现帧内特征聚合，从而显著提升目标的空间特征表示；2）时序级联双向 Mamba 编码器，以线性复杂度高效捕获视频序列中跨帧的长程依赖，实现帧间特征聚合，有效增强目标的时序特征表示；3）Mamba 纠缠 Transformer 解码器，充分探索目标查询与时空特征之间的交互，实现细粒度的查询-特征对齐，有效丰富目标查询的实例级表示。我们在 ImageNet VID 和 EPIC-KITCHENS-55 数据集上进行了实验，表明 TMambaDet 取得了最优结果。
英文原文摘要 1743 字符
Video object detection has gained notable progress with the advent of transformers. While transformers excel at modeling long-range contextual dependencies, the quadratic complexity limits their efficiency in long-sequence processing. In contrast, Mamba offers greater efficiency in modeling long sequences but tends to exhibit relatively limited contextual learning capability compared with transformers, and its application to video object detection remains unexplored. To harness the complementary strengths of transformers and Mamba, we propose a hybrid Transformer-Mamba network for video object detection (TMambaDet), a pioneering framework in this domain that combines the long-range modeling power of transformers with the efficient long-sequence processing capability of Mamba. Our TMambaDet is characterized by three core components: 1) a spatial adaptive deformable transformer encoder to effectively model the long-range dependencies within each frame, enabling intra-frame feature aggregation that substantially improves the spatial feature representations of objects; 2) a temporal cascaded bidirectional Mamba encoder to efficiently capture the long-range dependencies across frames in video sequences with linear complexity, enabling inter-frame feature aggregation that effectively enhances the temporal feature representations of objects; 3) a Mamba entangled transformer decoder to fully explore the interactions between object queries and spatial-temporal features, enabling fine-grained query-feature alignment that effectively enriches the instance-level representations of object queries. We conduct experiments on the ImageNet VID and EPIC-KITCHENS-55 datasets, showing that TMambaDet achieves state-of-the-art results.