通过概念解释 CLIP 的零样本预测
Explaining CLIP Zero-shot Predictions Through Concepts
把 CLIP 的图像-文本联合嵌入投影到由语言描述学到的概念空间，用对齐+重建目标训练，可在不改 CLIP 权重、不加概念监督的前提下为开放词表零样本预测给出概念级解释；做 VLM 可解释性或概念瓶颈方向可直接复用该投影与训练目标。
原文依据 projects CLIP's joint image-text embeddings into a concept space learned from language descriptions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大规模视觉语言模型（如 CLIP）在零样本图像识别上取得了显著成功，但其预测结果对人类理解而言仍基本不透明。相比之下，概念瓶颈模型通过人类定义的概念进行推理，从而提供可解释的中间表示，但它们依赖概念监督，且缺乏泛化到未见类别的能力。我们提出 EZPC，通过人类可理解的概念来解释 CLIP 的零样本预测，从而连接这两种范式。本方法将 CLIP 的图像-文本联合嵌入投影到一个由语言描述学习得到的概念空间，从而在无需额外监督的情况下实现忠实且透明的解释。模型通过对齐与重建目标的组合来学习该投影，确保概念激活既保留 CLIP 的语义结构又保持可解释性。在 CIFAR-100、CUB-200-2011、Places365、ImageNet-100 和 ImageNet-1k 五个基准数据集上的大量实验表明，本方法在保持 CLIP 强大零样本分类精度的同时，提供了有意义的概级解释。通过将开放词表预测视觉定位在显式语义概念之上，本方法为可解释且可信的视觉语言模型迈出了有原则的一步。代码见 https://github.com/oonat/ezpc。
英文原文摘要 1406 字符
Large-scale vision-language models such as CLIP have achieved remarkable success in zero-shot image recognition, yet their predictions remain largely opaque to human understanding. In contrast, Concept Bottleneck Models provide interpretable intermediate representations by reasoning through human-defined concepts, but they rely on concept supervision and lack the ability to generalize to unseen classes. We introduce EZPC that bridges these two paradigms by explaining CLIP's zero-shot predictions through human-understandable concepts. Our method projects CLIP's joint image-text embeddings into a concept space learned from language descriptions, enabling faithful and transparent explanations without additional supervision. The model learns this projection via a combination of alignment and reconstruction objectives, ensuring that concept activations preserve CLIP's semantic structure while remaining interpretable. Extensive experiments on five benchmark datasets, CIFAR-100, CUB-200-2011, Places365, ImageNet-100, and ImageNet-1k, demonstrate that our approach maintains CLIP's strong zero-shot classification accuracy while providing meaningful concept-level explanations. By grounding open-vocabulary predictions in explicit semantic concepts, our method offers a principled step toward interpretable and trustworthy vision-language models. Code is available at https://github.com/oonat/ezpc.