MM-SeR：面向轻量级图像描述的多模态自精炼
MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
把 MLLM 的语言部分换成 125M 小模型后仍能保持事实性描述能力，这一发现提示视觉研究者：做图像描述时不必依赖大语言模型的复杂推理；其「先粗后细」的自精炼流程（用粗描述定位显著区域再重描述）可作为轻量 VLM 的通用解码策略复用。
原文依据 we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频聊天机器人、导航机器人等系统常依赖流式图像描述来理解视觉输入。现有方法通常采用大型多模态语言模型（MLLM）完成这一任务，但其高昂的计算开销阻碍了实际应用。这一局限促使我们开发轻量级描述模型。我们的研究首先将 MLLM 中的大规模语言组件替换为紧凑的 125M 参数模型。令人意外的是，尽管规模缩小了 93 倍，该紧凑模型仍能达到与 MLLM 相当的性能，这表明事实性图像描述并不显著依赖 LLM 的复杂推理能力。尽管结果令人鼓舞，我们的轻量级模型仍缺乏可靠性。为此，我们从人类视觉过程中获得启发：先对场景形成全局而粗略的理解，再关注更精细的细节。据此，我们提出一种多模态自精炼框架，引导模型参考先前生成的粗略描述来定位显著区域的特征，并据此生成精炼后的描述。实验结果表明，我们的模型在单句描述与详细描述任务上均具有优越性，甚至可扩展到长程视频问答任务。
英文原文摘要 1322 字符
Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal language models (MLLMs) for this purpose, but their substantial computational cost hinders practical application.This limitation motivates our development of a lightweight captioning model. Our investigation begins by replacing the large-scale language component in MLLMs with a compact 125M-parameter model.Surprisingly, this compact model, despite a 93x reduction in size, achieves comparable performance to MLLMs, suggesting that factual image captioning does not significantly require the complex reasoning abilities of LLMs. Despite this promising result, our lightweight model still lacks reliability. To address this, we draw inspiration from the human visual process: perceiving a global and coarse understanding of the scene before attending to finer details. Accordingly, we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions, identified by referencing the previous coarse caption, and to produce a refined description. Experimental results demonstrate the superiority of our model in both single-sentence and detailed captioning, extending even to long-range video QA tasks.