纯视频心智理论：增强多模态大语言模型中的心智理论能力
Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models
提出一种通过计算干预向量来对齐视觉表征与正确语义目标的方法，可引导模型注意力跨层关注视觉特征，减少对语言先验的依赖；该技术可复用于其他需要增强视觉 grounding 的多模态推理任务。
原文依据 compute intervention vectors that align visual representations with the correct semantic targets, thereby steering the model's attention through different layers of visual features
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论随着大语言模型（LLMs）的持续发展，人们对其推断人类心理状态并展现类人心智理论（ToM）的能力日益关注。然而，现有大多数ToM评估以文本输入为中心，而仅依赖视觉信息的场景则受到远为较少的关注。这留下了一个空白，因为现实世界中的人机交互通常需要多模态理解。此外，许多当前方法将模型视为黑箱，很少探究其内部注意力在多选题问答（QA）中的行为。从可解释性角度看，LLM幻觉对此类任务的影响也尚未得到充分探索。为解决这些问题，我们引入了VisionToM，一个面向视觉的干预框架，旨在强化任务感知推理。其核心思想是计算干预向量，使视觉表征与正确的语义目标对齐，从而通过不同层的视觉特征引导模型的注意力。这种引导减少了模型对虚假语言先验的依赖，从而产生更可靠的多模态语言模型（MLLM）输出和更好的QA性能。在EgoToM基准——一个具有三种多选题QA设置的自我中心真实世界视频ToM数据集——上的实验表明，我们的方法大幅提升了MLLM的ToM能力。此外，在另一个开放式生成任务上的结果表明，VisionToM使MLLM能够生成更准确捕捉智能体心理状态的自由形式解释，推动人机协作朝着更高的一致性迈进。
英文原文摘要 1660 字符
As large language models (LLMs) continue to advance, there is increasing interest in their ability to infer human mental states and demonstrate a human-like Theory of Mind (ToM). Most existing ToM evaluations, however, are centered on text-based inputs, while scenarios relying solely on visual information receive far less attention. This leaves a gap, since real-world human-AI interaction typically requires multimodal understanding. In addition, many current methods regard the model as a black box and rarely probe how its internal attention behaves in multiple-choice question answering (QA). The impact of LLM hallucinations on such tasks is also underexplored from an interpretability perspective. To address these issues, we introduce VisionToM, a vision-oriented intervention framework designed to strengthen task-aware reasoning. The core idea is to compute intervention vectors that align visual representations with the correct semantic targets, thereby steering the model's attention through different layers of visual features. This guidance reduces the model's reliance on spurious linguistic priors, leading to more reliable multimodal language model (MLLM) outputs and better QA performance. Experiments on the EgoToM benchmark--an egocentric, real-world video dataset for ToM with three multiple-choice QA settings--demonstrate that our method substantially improves the ToM abilities of MLLMs. Furthermore, results on an additional open-ended generation task show that VisionToM enables MLLMs to produce free-form explanations that more accurately capture agents' mental states, pushing machine-human collaboration toward greater alignment.