VisMem：潜变量视觉记忆释放视觉语言模型的潜力
VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
为 VLM 在长程生成中维持视觉证据提供了一种潜空间记忆机制：短期记忆模块保留细粒度感知特征、长期记忆模块固化语义，可在推理阶段即插即用地缓解视觉定位漂移，对多模态推理与视觉生成类工作都可直接复用。
原文依据 a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论尽管视觉语言模型（VLM）取得了显著成功，但其在一系列复杂视觉任务上的表现常常受制于「视觉处理瓶颈」：在长时间生成过程中，模型倾向于丢失对视觉证据的定位，并表现出情境化视觉经验的不足。受人类认知记忆理论的启发——该理论区分了以视觉为主导的短期记忆与以语义为主导的长期记忆——我们提出了 VisMem，一个与认知机制对齐的框架，为 VLM 配备动态的潜变量视觉记忆，其中包括用于细粒度感知保持的短期模块和用于抽象语义固化的长期模块。这些记忆在推理过程中被无缝调用，使 VLM 在思考与生成的全过程中同时保持感知保真度与语义一致性。在面向理解、推理与生成的多种视觉基准上的大量实验表明，VisMem 相较原始模型取得了 11.0% 的显著平均性能提升，并优于所有对比方法，确立了潜空间记忆增强的新范式。代码将发布于：https://github.com/YU-deep/VisMem.git。
英文原文摘要 1225 字符
Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git.