随机分组胜出：重新审视视觉词元的分组策略
Random Wins All: Rethinking Grouping Strategies for Vision Tokens
用随机分组替代精心设计的词元分组策略来降低自注意力的二次复杂度，可直接替换 ViT 类骨干中的分组注意力模块，并在检测等下游任务上验证了迁移收益。
原文依据 we propose the random grouping strategy, which involves a simple and fast random grouping strategy for vision tokens
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论自 Transformer 被引入视觉架构以来，其二次复杂度一直是一个重要问题，许多研究工作致力于解决它。一类代表性做法是对词元进行分组，并在每个组内执行自注意力计算。为此，人们提出了各种精心设计的分组策略以提升视觉 Transformer 的性能。本文提出以下问题：这些精心设计的分组方法真的必要吗？是否存在一种更简单、更统一的词元分组方法可以替代这些多样的方法？为此，我们提出随机分组策略，即一种简单快速的视觉词元随机分组策略。我们在多个基线上验证了该方法，实验表明随机分组几乎优于所有其他分组方法。迁移到目标检测等下游任务时，随机分组的优势更加明显。针对这一现象，我们从多个角度详细分析了随机分组的优势，并识别出分组策略设计中的若干关键要素：位置信息、检测头特征多样性、全局感受野以及固定分组模式。我们证明，只要满足这四个条件，视觉词元只需一种极其简单的分组策略即可高效且有效地处理各类视觉任务。我们还在多种模态上验证了所提出的随机方法的有效性，包括视觉任务、点云处理和视觉语言模型。代码将发布于 https://github.com/qhfan/random。
英文原文摘要 1718 字符
Since Transformers are introduced into vision architectures, their quadratic complexity has always been a significant issue that many research efforts aim to address. A representative approach involves grouping tokens, performing self-attention calculations within each group. To this end, various carefully designed grouping strategies have been proposed to enhance the performance of Vision Transformers. Here, we pose the following questions: Are these carefully designed grouping methods truly necessary? Is there a simpler and more unified token grouping method that can replace these diverse methods? Therefore, we propose the random grouping strategy, which involves a simple and fast random grouping strategy for vision tokens. We validate this approach on multiple baselines, and experiments show that random grouping almost outperforms all other grouping methods. When transferred to downstream tasks, such as object detection, random grouping demonstrates even more pronounced advantages. In response to this phenomenon, we conduct a detailed analysis of the advantages of random grouping from multiple perspectives and identify several crucial elements for the design of grouping strategies: positional information, head feature diversity, global receptive field, and fixed grouping pattern. We demonstrate that as long as these four conditions are met, vision tokens require only an extremely simple grouping strategy to efficiently and effectively handle various visual tasks. We also validate the effectiveness of our proposed random method across multiple modalities, including visual tasks, point cloud processing, and vision-language models. Code will be available at https://github.com/qhfan/random.