语言模型可通过引导解释视觉特征
Language Models Can Explain Visual Features via Steering
对做视觉可解释性的人，这篇给出了一个不依赖输入样本的因果式特征解释范式：在视觉编码器里直接引导单个 SAE 特征、再让语言模型描述其语义，可作为探测视觉表征概念的新工具；Steering-informed Top-k 混合策略还能零额外算力提升解释质量。
原文依据 we present a fundamentally different alternative based on causal interventions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论稀疏自编码器在视觉模型中揭示了数千个特征，然而在无需人工干预的情况下解释这些特征仍是一个开放挑战。以往工作提出了基于高激活输入样本生成相关性解释的方法，而我们提出了一种基于因果干预的全新替代方案。我们利用视觉语言模型的结构，在输入空图像后对视觉编码器中的单个 SAE 特征进行引导。随后，我们提示语言模型解释它“看到”了什么，从而有效引出每个特征所代表的视觉概念。结果表明，引导提供了一种可扩展的替代方案，与基于输入样本的传统方法互补，成为视觉模型自动化可解释性的新维度。此外，解释质量随语言模型规模的增大而持续提升，凸显了我们的方法作为未来研究方向的前景。最后，我们提出了 Steering-informed Top-k，一种混合方法，结合了因果干预与基于输入的方法的优势，在无需额外计算成本的情况下达到最优的解释质量。
英文原文摘要 1219 字符
Sparse Autoencoders uncover thousands of features in vision models, yet explaining these features without requiring human intervention remains an open challenge. While previous work has proposed generating correlation-based explanations based on top activating input examples, we present a fundamentally different alternative based on causal interventions. We leverage the structure of Vision-Language Models and steer individual SAE features in the vision encoder after providing an empty image. Then, we prompt the language model to explain what it "sees", effectively eliciting the visual concept represented by each feature. Results show that Steering offers an scalable alternative that complements traditional approaches based on input examples, serving as a new axis for automated interpretability in vision models. Moreover, the quality of explanations improves consistently with the scale of the language model, highlighting our method as a promising direction for future research. Finally, we propose Steering-informed Top-k, a hybrid approach that combines the strengths of causal interventions and input-based approaches to achieve state-of-the-art explanation quality without additional computational cost.