PowerCLIP：面向对比预训练的幂集对齐
PowerCLIP: Powerset Alignment for Contrastive Pre-Training
把图文对齐从「整图-整句」和「单 token-单 patch」推进到「区域子集-短语子树」的幂集级对齐，并用非线性聚合器把 2^M 复杂度降到 O(M)，这套聚合思路可直接迁移到任何需要枚举组合子集的细粒度跨模态对齐或区域级对比学习框架中。
原文依据 we introduce efficient non-linear aggregators (NLAs) that reduce complexity from \mathit \mathcal O (2^ M ) to \mathit \mathcal O (M)
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论CLIP 等对比预训练框架已在多种视觉-语言任务上展现出令人印象深刻的零样本性能。近期研究表明，将单个文本 token 与特定图像块或区域对齐，能够增强细粒度的组合式理解。然而，捕捉跨越多个图像区域的组合语义仍然具有挑战性。为克服这一局限，我们提出 PowerCLIP，一种由幂集对齐增强的新型对比预训练框架，它通过在图像区域的幂集与文本解析树之间定义损失并加以最小化，穷尽式地优化区域到短语的对齐。由于区域子集数量的组合爆炸会使该方法的计算复杂度呈指数增长，我们引入高效的非线性聚合器（NLA），将复杂度从关于区域数 M 的 O(2^M) 降至 O(M)，并可证明地以任意精度逼近精确损失值。大量实验表明，PowerCLIP 在零样本分类与检索任务上优于最优方法，凸显了我们方法的组合性与鲁棒性。代码将公开。
英文原文摘要 1282 字符
Contrastive pre-training frameworks such as CLIP have demonstrated impressive zero-shot performance across a range of vision-language tasks. Recent studies have shown that aligning individual text tokens with specific image patches or regions enhances fine-grained compositional understanding. However, it remains challenging to capture compositional semantics spanning multiple image regions.To address this limitation, we propose PowerCLIP, a novelcontrastive pre-training framework enhanced by powerset alignment, which exhaustively optimizes region-to-phrase alignments by minimizing the loss defined between powersets of image regions and textual parse trees.As this approach increases computational complexity exponentially due to the combinatorial explosion in the number of region subsets, we introduce efficient non-linear aggregators (NLAs) that reduce complexity from \mathit \mathcal O (2^ M ) to \mathit \mathcal O (M) with respect to the number of regions M, provably approximating the exact loss value with arbitrary precision.Our extensive experiments demonstrate that PowerCLIPoutperforms state-of-the-art methodsin zero-shot classification and retrieval tasks, underscoring compositionality and robustness of our approach. Our code will be made publicly available.