Mario：基于大语言模型的多模态图推理
Mario: Multimodal Graph Reasoning with Large Language Models
把图拓扑作为结构先验注入视觉-语言特征对齐，并用可学习路由器为每个节点挑选最有信息量的模态配置，这一「图条件化 VLM + 模态自适应路由」思路可迁移到视觉场景图、多模态知识图谱以及图文检索中的结构化对齐问题。
原文依据 a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大语言模型（LLM）的最新进展为多模态推理开辟了新途径。然而，现有方法大多仍依赖预训练的视觉-语言模型（VLM）孤立地编码图文对，忽略了真实世界多模态数据天然形成的关系结构。这促使我们在多模态图（MMG）上进行推理，其中每个节点具有文本与视觉属性，边提供结构线索。在此类异构多模态信号上实现基于 LLM 的推理并同时保持图拓扑，带来两个关键挑战：解决弱跨模态一致性问题，以及处理异构模态偏好问题。为此，我们提出 Mario，一个统一框架，可同时解决上述两个挑战，并在 MMG 上实现有效的基于 LLM 的推理。Mario 包含两个创新阶段。首先是一种图条件化的 VLM 设计，通过由图拓扑引导的细粒度跨模态对比学习联合精炼文本与视觉特征。其次是一种模态自适应的图指令微调机制，将对齐后的多模态特征组织为图感知的指令视图，并采用可学习路由器，为每个节点及其邻域向 LLM 呈现最具信息量的模态配置。在多种 MMG 基准上的大量实验表明，Mario 在监督与零样本场景下的节点分类和链接预测任务中均持续优于最优的图模型。代码将发布于 https://github.com/sunyuanfu/Mario。
英文原文摘要 1600 字符
Recent advances in large language models (LLMs) have opened new avenues for multimodal reasoning. Yet, most existing methods still rely on pretrained vision-language models (VLMs) to encode image-text pairs in isolation, ignoring the relational structure that real-world multimodal data naturally form. This motivates reasoning on multimodal graphs (MMGs), where each node has textual and visual attributes and edges provide structural cues. Enabling LLM-based reasoning on such heterogeneous multimodal signals while preserving graph topology introduces two key challenges: resolving weak cross-modal consistency and handling heterogeneous modality preference. To address this, we propose Mario, a unified framework that simultaneously resolves the two above challenges and enables effective LLM-based reasoning over MMGs. Mario consists of two innovative stages. Firstly, a graph-conditioned VLM design that jointly refines textual and visual features through fine-grained cross-modal contrastive learning guided by graph topology. Secondly, a modality-adaptive graph instruction tuning mechanism that organizes aligned multimodal features into graph-aware instruction views and employs a learnable router to surface, for each node and its neighborhood, the most informative modality configuration to the LLM. Extensive experiments across diverse MMG benchmarks demonstrate that Mario consistently outperforms state-of-the-art graph models in both supervised and zero-shot scenarios for node classification and link prediction. The code will be made available at https://github.com/sunyuanfu/Mario.