Minecraft 游戏中多模态大模型智能体的经验迁移
Experience Transfer for Multimodal LLM Agents in Minecraft Game
把过去交互经验拆成结构/属性/过程/功能/交互五个维度并做上下文类比检索，这套「可迁移经验表示 + 检索式上下文示例」的机制可直接搬到视觉具身智能体的长程任务记忆里，替代把历史帧当静态缓存的朴素做法。
原文依据 Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在复杂游戏环境中运行的多模态大模型智能体必须持续复用过往经验，才能高效求解新任务。本文提出 Echo，一个面向迁移的记忆框架，使智能体能够从先前的交互中提炼出可操作的知识，而不是把记忆当作静态记录的被动仓库。为使迁移显式化，Echo 将可复用知识分解为五个维度：结构、属性、过程、功能与交互。这一形式化使智能体能够识别不同任务间共享的重复模式，并推断哪些先验经验在新情境下仍然适用。在此形式化基础上，Echo 利用上下文类比学习（ICAL）检索相关经验，并通过上下文示例将其适配到未见任务。在 Minecraft 中的实验表明，在从零开始的学习设定下，Echo 在物体解锁任务上取得 1.3 倍至 1.7 倍的加速。此外，Echo 呈现出爆发式的链式解锁现象：在获得可迁移经验后，它能在短时间内迅速解锁多个相似物品。这些结果表明，经验迁移是提升多模态大模型智能体在复杂交互环境中效率与适应性的一个有前景的方向。
英文原文摘要 1318 字符
Multimodal LLM agents operating in complex game environments must continually reuse past experience to solve new tasks efficiently. In this work, we propose Echo, a transfer-oriented memory framework that enables agents to derive actionable knowledge from prior interactions rather than treating memory as a passive repository of static records. To make transfer explicit, Echo decomposes reusable knowledge into five dimensions: structure, attribute, process, function, and interaction. This formulation allows the agent to identify recurring patterns shared across different tasks and infer what prior experience remains applicable in new situations. Building on this formulation, Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples. Experiments in Minecraft show that, under a from-scratch learning setting, Echo achieves a 1.3x-1.7x speed-up on object-unlocking tasks. Moreover, Echo exhibits a burst-like chain-unlocking phenomenon, rapidly unlocking multiple similar items within a short time interval after acquiring transferable experience. These results suggest that experience transfer is a promising direction for improving the efficiency and adaptability of multimodal LLM agents in complex interactive environments.