ViLoMem：具备生长-精炼多模态语义记忆的智能体学习器
ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory
ViLoMem 提出双流语义记忆，分别编码视觉分心模式与逻辑推理错误，使多模态大模型在解题时能复用可泛化的视觉注意力策略并避免重复视觉错误；该记忆机制可直接用于视觉问答、多模态推理等任务中的智能体持续学习与跨域迁移。
原文依据 It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型在孤立查询上表现出强大的推理能力，但它们每次都是从头开始——独立解决每个问题，并经常重复同样的错误。现有的记忆增强智能体主要存储过去的轨迹以供重用。然而，基于轨迹的记忆存在简洁性偏差，会逐渐丢失关键的领域知识。更关键的是，即使在真正的多模态问题解决场景中，它也只记录过去行为的单模态痕迹，无法保留视觉注意力和逻辑推理如何共同促成解决方案。这与人类认知从根本上不一致：语义记忆是多模态且整合的，通过协调但不同的表征流来保存视觉和抽象知识。因此，我们引入了 ViLoMem，一个双流记忆框架，构建紧凑的、基于图式的记忆。它分别编码视觉分心模式和逻辑推理错误，使多模态大语言模型能够从成功和失败的经验中学习。遵循生长-精炼原则，系统增量地积累和更新多模态语义知识——保留稳定、可泛化的策略，同时避免灾难性遗忘。在九个多模态基准上，ViLoMem 一致地提高了 pass@1 准确率，并大幅减少了重复的视觉和逻辑错误。消融实验证实了双流记忆与显式分心-幻觉分离的必要性，展示了错误感知的多模态记忆在终身和跨域智能体学习中的价值。
英文原文摘要 1590 字符
MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo--solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a **single-modality** trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both **multimodal and integrated**, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce **ViLoMem**, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge--preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across nine multimodal benchmarks, **ViLoMem** consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction-hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning.