RNN 即线性 Transformer：对视觉 Mamba 模型表征潜力的深入考察
RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
把 Mamba 解释为 Softmax 注意力的低秩近似，并给出用二值分割指标定量评估激活图长程依赖能力的方法；做视觉骨干网络选型或可解释性分析时，可复用这套理论视角与评估协议。
原文依据 confirming that it can be viewed as a low-rank approximation of Softmax Attention
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论Mamba 最初为语言建模而提出，近来作为视觉任务的有效骨干网络受到关注。然而，其在视觉领域中的内在机制仍鲜有理解。在本工作中，我们系统性地考察了 Mamba 的表征性质，并做出三项主要贡献。第一，我们从理论上分析了 Mamba 与 Softmax 注意力及线性注意力之间的关系，确认 Mamba 可被视为 Softmax 注意力的低秩近似，从而弥合了 Softmax 形式与线性形式之间的表征鸿沟。第二，我们提出了一种新的二值分割指标用于激活图评估，将定性评估扩展为定量度量，并借此证明 Mamba 具备建模长程依赖的能力。第三，通过利用 DINO 进行自监督预训练，我们获得的激活图比标准监督方法所生成的更为清晰，凸显了 Mamba 在可解释性方面的潜力。值得注意的是，我们的模型在 ImageNet 上还取得了 78.5% 的线性探测准确率，彰显了其强劲性能。我们希望本工作能为未来基于 Mamba 的视觉架构研究提供有价值的洞见。
英文原文摘要 1214 字符
Mamba, originally introduced for language modeling, has recently garnered attention as an effective backbone for vision tasks. However, its underlying mechanism in visual domains remains poorly understood. In this work, we systematically investigate Mamba's representational properties and make three primary contributions. First, we theoretically analyze Mamba's relationship to Softmax and Linear Attention, confirming that it can be viewed as a low-rank approximation of Softmax Attention and thereby bridging the representational gap between Softmax and Linear forms. Second, we introduce a novel binary segmentation metric for activation map evaluation, extending qualitative assessments to a quantitative measure that demonstrates Mamba's capacity to model long-range dependencies. Third, by leveraging DINO for self-supervised pretraining, we obtain clearer activation maps than those produced by standard supervised approaches, highlighting Mamba's potential for interpretability. Notably, our model also achieves a 78.5% linear probing accuracy on ImageNet, underscoring its strong performance. We hope this work can provide valuable insights for future investigations of Mamba-based vision architectures.