面向意图识别的层次语义表示驱动的进化式多模态推理
Evolutionary Multimodal Reasoning via Hierarchical Semantic Representation for Intent Recognition
把多模态语义组织成 token 级、概念级、概念关系级三层表示，再用 CoT 提示注入 MLLM 做逐步推理，并借 MLLM 反馈在推理时迭代精炼表示——这套「层次语义 + 自进化提示」流程可直接迁移到视觉问答、视觉定位等需要多粒度视觉语义推理的任务。
原文依据 These hierarchical representations are then injected into MLLM via CoT-driven prompting, enabling step-wise reasoning.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态意图识别旨在通过联合建模多种模态来推断人类意图，在真实对话系统中起着关键作用。然而，现有方法难以对复杂意图背后的层次语义进行建模，也缺乏对多模态表示进行自进化推理的能力。为解决这些问题，我们提出 HIER，一种将层次语义表示与基于多模态大语言模型（MLLM）的进化式推理相结合的新方法。受人类认知启发，HIER 引入一种结构化推理范式，将多模态语义组织为三个逐步抽象的层次。它首先以模态专属 token 捕捉局部语义线索，随后通过标签引导策略对这些 token 进行聚类，形成中层语义概念。为捕捉更高阶的结构，概念间关系通过 JS 散度分数进行筛选，以突出概念之间显著的依赖关系。这些层次化表示随后通过思维链（CoT）驱动的提示注入 MLLM，从而实现逐步推理。此外，HIER 采用自进化机制，借助 MLLM 的反馈精炼语义表示，使其在推理过程中能够动态适应。在三个具有挑战性的基准上的实验表明，HIER 在所有指标上均稳定优于最优方法与 MLLM，提升幅度为 1-3%。代码与更多结果见 https://github.com/thuiar/HIER。
英文原文摘要 1489 字符
Multimodal intent recognition aims to infer human intents by jointly modeling various modalities, playing a pivotal role in real-world dialogue systems. However, current methods struggle to model hierarchical semantics underlying complex intents and lack the capacity for self-evolving reasoning over multimodal representations. To address these issues, we propose HIER, a novel method that integrates hierarchical semantic representation with evolutionary reasoning based on Multimodal Large Language Model (MLLM). Inspired by human cognition, HIER introduces a structured reasoning paradigm that organizes multimodal semantics into three progressively abstracted levels. It starts with modality-specific tokens capturing localized semantic cues, which are then clustered via a label-guided strategy to form mid-level semantic concepts. To capture higher-order structure, inter-concept relations are selected using JS divergence scores to highlight salient dependencies across concepts. These hierarchical representations are then injected into MLLM via CoT-driven prompting, enabling step-wise reasoning. Besides, HIER utilizes a self-evolution mechanism that refines semantic representations through MLLM feedback, allowing dynamic adaptation during inference. Experiments on three challenging benchmarks show that HIER consistently outperforms state-of-the-art methods and MLLMs with 1-3% gains across all metrics. Code and more results are available at https://github.com/thuiar/HIER.