拥抱视觉提示与分割令牌：多模态大语言模型中用于细粒度视觉理解的一致性学习
Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs
该论文提出的一致性学习损失和两阶段训练框架，可复用于其他需要联合优化视觉提示与分割令牌的多模态任务，为细粒度视觉理解提供了一种通用的训练策略。
原文依据 we propose FCLM, a large multimodal model designed to jointly support fine-grained visual understanding through consistency learning.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近年来，多模态大语言模型（MLLMs）在通用多模态任务中取得了显著成功。越来越多的注意力被投向利用MLLMs进行细粒度视觉理解，例如区域级描述生成和像素级视觉定位。然而，大多数现有方法是任务特定的。尽管一些近期的统一方法尝试同时处理这两种类型，但它们仍未深入探索任务之间的潜在关联。为了弥补这一差距，我们提出了FCLM，一个通过一致性学习联合支持细粒度视觉理解的大型多模态模型。其核心前提是像素级描述生成与视觉定位是互利互补的，二者在实现对视觉内容的细粒度理解中相互增强。具体而言，FCLM分析了这两类视觉任务所需的表示特征——视觉提示和分割令牌，并通过新设计的一致性学习损失和两阶段训练框架实现先进的推理与感知。此外，设计了一种混合区域提取器来增强视觉提示嵌入，为详细描述生成产生语义判别性表示。进一步，引入了一个新的详细局部指代表达分割（DL-RES）任务，以评估模型从详细文本描述中定位目标的能力。在七个视觉理解任务上的大量实验证明了FCLM的优越性能和强泛化能力。
英文原文摘要 1592 字符
Recently, multimodal large language models (MLLMs) have achieved remarkable success in general multimodal tasks. Increasing attention has been given to leveraging MLLMs for fine-grained visual understanding, such as region-level captioning and pixel-level grounding. However, most existing approaches are task-specific. While some recent unified approaches attempt to handle both types simultaneously, they still fall short of deeply exploring the underlying associations across tasks. To bridge this gap, we propose FCLM, a large multimodal model designed to jointly support fine-grained visual understanding through consistency learning. The central premise is that pixel-level captioning and grounding are mutually beneficial and complementary, each enhancing the other in achieving a fine-grained understanding of visual content. Specifically, FCLM analyzes the representation features - visual prompt and segmentation tokens - required for the two types of visual tasks, and achieves advanced reasoning and perception through a newly-designed consistency learning loss and a two-stage training framework. Moreover, a hybrid region extractor is designed to enhance visual prompt embeddings, yielding semantically discriminative representations for detailed captioning. Furthermore, a novel detailed localized referring expression segmentation (DL-RES) task is introduced to evaluate the model's ability to localize targets from detailed textual descriptions. Extensive experiments on seven visual understanding tasks demonstrate the superior performance and strong generalization of FCLM.