学会聚焦与精确裁剪：面向多模态大语言模型的信息差与视觉定位损失强化学习框架
Learning to Focus and Precise Cropping:A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs
针对多模态大模型在裁剪区域上「看了却没用」的问题，用信息差机制（降低全局图粒度以制造信息增益）配合少量边界框的 grounding loss 做两阶段强化学习，可复用于任何需要模型真正依赖局部细节的高分辨率 VQA / 细粒度感知流程。
原文依据 we propose a novel two-stage reinforcement learning framework that does not require trajectory supervision
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论为提升多模态大语言模型在复杂视觉场景中的感知与推理能力，近期研究引入了基于智能体的工作流。在这些工作中，多模态大语言模型自主调用图像裁剪工具来分析感兴趣区域以完成问答。尽管现有的训练策略（如采用监督微调和强化学习的方法）已取得显著进展，我们的实证分析揭示了一个关键局限：模型对全局输入存在强依赖，而对裁剪区域内的细节依赖较弱。为解决该问题，我们提出了一种无需轨迹监督的新型两阶段强化学习框架。在第一阶段，我们通过调整全局图像的粒度引入「信息差」机制。该机制驱使模型聚焦于裁剪出的关键区域来回答问题，其动力来自这些区域所提供的信息增益。第二阶段利用少量边界框标注引入视觉定位损失，进一步提升裁剪精度。实验表明，我们的方法显著增强了模型对裁剪区域的注意力，使其在高分辨率视觉问答基准上达到最优性能。我们的方法为多模态大语言模型感知与推理细粒度细节提供了一种更高效的途径。代码已开源：https://github.com/XuanPu-Z/LFPC。
英文原文摘要 1498 字符
To enhance the perception and reasoning capabilities of multimodal large language models in complex visual scenes, recent research has introduced agent-based workflows. In these works, MLLMs autonomously utilize image cropping tool to analyze regions of interest for question answering. While existing training strategies, such as those employing supervised fine-tuning and reinforcement learning, have made significant progress, our empirical analysis reveals a key limitation. We demonstrate the model's strong reliance on global input and its weak dependence on the details within the cropped region. To address this issue, we propose a novel two-stage reinforcement learning framework that does not require trajectory supervision. In the first stage, we introduce the "Information Gap" mechanism by adjusting the granularity of the global image. This mechanism trains the model to answer questions by focusing on cropped key regions, driven by the information gain these regions provide. The second stage further enhances cropping precision by incorporating a grounding loss, using a small number of bounding box annotations. Experiments show that our method significantly enhances the model's attention to cropped regions, enabling it to achieve state-of-the-art performance on high-resolution visual question-answering benchmarks. Our method provides a more efficient approach for perceiving and reasoning fine-grained details in MLLMs. Code is available at: https://github.com/XuanPu-Z/LFPC.