视觉语言模型中目标定位的机制
Mechanisms of Object Localization in Vision-Language Models
对做视觉定位与多模态定位的研究者，这篇给出了可复用的机制分析工具箱（词元消融、注意力 knockout、因果中介分析），并揭示定位由「容器化」词元决定空间范围、仅极少数注意力头承担因果作用，可用于指导 VLM 定位目标设计与注意力头级剪枝/干预。
原文依据 localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉接地的语言模型（VLM）在关联视觉与文本信息方面非常有效，但它们常常在基本的分类与定位任务上表现不佳。分类机制已被较为广泛地研究，而支撑目标定位的过程仍鲜为人知。在本工作中，我们使用一套机制可解释性工具，包括词元消融、注意力 knockout 与因果中介分析，考察了两个代表性模型家族 LLaVA-1.5 与 InternVL-3.5。我们发现定位由一种容器化机制驱动：与目标对齐的词元界定了目标的空间范围，而这些边界内词元的语义排布对预测框基本无关。只有极少数注意力头对分类与定位的因果效应起中介作用，在 LLaVA 中集中于早中期层，在 InternVL 中集中于中后期层。两个任务共享部分早期处理，但最终依赖 largely 不同的专门化注意力头。总体而言，我们首次给出了 VLM 中定位的层级与头级解释，揭示了狭窄的计算通路，可指导未来的模型设计与视觉定位目标。
英文原文摘要 1262 字符
Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object localization remain poorly understood. In this work, we investigate two representative families, LLaVA-1.5 and InternVL-3.5, using a suite of mechanistic interpretability tools, including token ablations, attention knockout, and causal mediation analysis. We find that localization is driven by a containerization mechanism in which object-aligned tokens define the spatial extent of the object, while the semantic arrangement of tokens within those boundaries is largely irrelevant to the predicted box. Only a very small set of attention heads mediates the causal effect for both classification and localization, concentrating in early-mid layers for LLaVA and mid-late layers for InternVL. The two tasks share some early processing but ultimately depend on largely distinct specialized heads. Overall, we provide the first layer- and head-level account of localization in VLMs, revealing narrow computational pathways that can guide future model design and grounding objectives.