跨模态恒等映射：通过强化学习最小化模态转换中的信息损失
Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
提出用图文检索相似度作为强化学习奖励信号，无需人工标注即可优化LVLM的图像描述，减少模态转换中的信息损失；该奖励设计可迁移到其他需要细粒度视觉理解的生成任务。
原文依据 the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论大型视觉语言模型（LVLM）在生成的图像描述中经常遗漏或错误呈现关键视觉内容。最小化此类信息损失将迫使LVLM关注图像细节以生成精确描述。然而，由于视觉内容与文本输出之间的模态鸿沟，衡量模态转换过程中的信息损失本质上具有挑战性。本文认为，图像描述的质量与使用该描述通过文本搜索检索到的图像之间的相似度正相关。基于这一见解，我们进一步提出跨模态恒等映射（CIM），一种无需额外标注即可增强图像描述的强化学习框架。具体而言，该方法从两个角度定量评估信息损失：图库表示一致性和查询-图库图像相关性。在这些指标的监督下，LVLM最小化信息损失，旨在实现从图像到描述的恒等映射。实验结果表明，我们的方法在图像描述上具有优越性能，甚至与监督微调相比也是如此。特别是在COCO-LN500基准上，CIM在Qwen2.5-VL-7B上实现了关系推理20%的提升。代码已在GitHub上发布。
英文原文摘要 1312 字符
Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B. The codes are released on GitHub.