CLiViS：借助语言-视觉协同释放认知地图以赋能具身视觉推理
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
提出免训练的 LLM 规划 + VLM 开放世界感知协同框架，并用动态认知地图作为结构化场景表示，可复用于长时第一人称视频的逐步组合推理与长程视觉依赖建模。
原文依据 a novel training-free framework that leverages LLMs for high-level task planning and orchestrates VLM-driven open-world visual perception
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论具身视觉推理（EVR）旨在依据第一人称视频遵循复杂、自由形式的指令，从而在动态环境中实现语义理解与时空推理。尽管潜力可观，EVR 仍面临显著挑战，其根源在于复杂指令的多样性以及长时第一人称视频中错综复杂的时空动态。已有方案要么在静态视频描述之上调用大语言模型（LLM），而这类描述往往遗漏关键视觉细节；要么依赖端到端视觉语言模型（VLM），而这类模型难以进行逐步的组合式推理。考虑到 LLM 在推理与 VLM 在感知上的互补优势，我们提出 CLiViS。它是一个新颖的免训练框架，利用 LLM 进行高层任务规划，并编排由 VLM 驱动的开放世界视觉感知，以迭代方式更新场景上下文。在这一协同之上，CLiViS 的核心是一个在推理过程中持续演化的动态认知地图。该地图构建出具身场景的结构化表示，桥接低层感知与高层推理。在多个基准上的大量实验证明了 CLiViS 的有效性与通用性，尤其是在处理长程视觉依赖方面。代码见 https://github.com/Teacher-Tom/CLiViS。
英文原文摘要 1377 字符
Embodied Visual Reasoning (EVR) seeks to follow complex, free-form instructions based on egocentric video, enabling semantic understanding and spatiotemporal reasoning in dynamic environments. Despite its promising potential, EVR encounters significant challenges stemming from the diversity of complex instructions and the intricate spatiotemporal dynamics in long-term egocentric videos. Prior solutions either employ Large Language Models (LLMs) over static video captions, which often omit critical visual details, or rely on end-to-end Vision-Language Models (VLMs) that struggle with stepwise compositional reasoning. Considering the complementary strengths of LLMs in reasoning and VLMs in perception, we propose CLiViS. It is a novel training-free framework that leverages LLMs for high-level task planning and orchestrates VLM-driven open-world visual perception to iteratively update the scene context. Building on this synergy, the core of CLiViS is a dynamic Cognitive Map that evolves throughout the reasoning process. This map constructs a structured representation of the embodied scene, bridging low-level perception and high-level reasoning. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generality of CLiViS, especially in handling long-term visual dependencies. Code is available at https://github.com/Teacher-Tom/CLiViS.