See What I Mean：对齐视觉与语言表示用于视频细粒度物体理解
See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding
通过训练时使用掩码监督来对齐视觉与语言表示，使多模态大语言模型在推理时仅凭文本提示即可自动关注用户指定的物体，为细粒度物体理解提供了一种无需视觉提示的跨模态注意力对齐方法。
原文依据 SWIM leverages mask supervision only during training to guide cross-modal attention, allowing the model to automatically attend to the user-specified object at inference.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论我们提出 SWIM（See What I Mean），一种新颖的训练策略，通过对齐视觉与语言表示，使得仅凭文本提示即可实现细粒度物体理解。与需要显式视觉提示（如掩码或点）的现有方法不同，SWIM 仅在训练时利用掩码监督来引导跨模态注意力，从而使模型在推理时能够自动关注用户指定的物体。我们对预训练多模态大语言模型（MLLMs）的交叉注意力分析揭示了一个系统性差异：属性词在视觉模态中产生尖锐、局部化的激活，而物体名词由于语义指代偏差和分布式高层表示，产生弥散且分散的模式。为了解决这种不对齐问题，我们构建了 NL-Refer，一个丰富的数据集，其中每个物体掩码都与一个精确的自然语言指代表达配对。SWIM 从物体名词中提取多层交叉注意力图，并强制其与真实掩码保持空间一致性。实验结果表明，SWIM 显著提升了文本-视觉对齐，并在细粒度物体理解基准上取得了优于基于视觉提示的方法的性能。代码和数据可在 https://github.com/HumanMLLM/SWIM 获取。
英文原文摘要 1330 字符
We present SWIM (See What I Mean), a novel training strategy that aligns vision and language representations to enable fine-grained object understanding solely from textual prompts. Unlike existing approaches that require explicit visual prompts, such as masks or points, SWIM leverages mask supervision only during training to guide cross-modal attention, allowing the model to automatically attend to the user-specified object at inference.Our cross-attention analysis of pretrained multimodal large language models (MLLMs) reveals a systematic discrepancy: Attribute words produce sharp, localized activations in the visual modality, whereas object nouns yield diffuse and scattered patterns due to semantic reference bias and distributed high-level representations. To address this misalignment, we construct NL-Refer, an enriched dataset, in which each object mask is paired with a precise natural language referring expression. SWIM extracts multi-layer cross-attention maps from object nouns and enforces spatial consistency with ground-truth masks. Experimental results demonstrate that SWIM substantially improves text-visual alignment and achieves superior performance over visual-prompt-based methods on fine-grained object understanding benchmarks. The code and data are available at https://github.com/HumanMLLM/SWIM.