IBISAgent：强化多模态大模型中的像素级视觉推理，实现通用生物医学目标指代与分割
IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
把分割重构为多轮「文本点击动作 + 调用分割工具」的智能体决策过程，无需改动 MLLM 架构、也无需联合微调解码器，这种「MLLM 输出点击坐标驱动外部分割器」的范式可直接迁移到通用指代分割与交互式分割，并天然支持掩码迭代精修。
原文依据 generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期关于医学多模态大模型的研究重心已从图像级理解转向细粒度、像素级的理解。尽管分割是像素级理解的基础，现有方法仍面临两大挑战。其一，它们引入隐式分割 token，并需要对多模态大模型与外部像素解码器进行联合微调，这增加了灾难性遗忘的风险，并限制了域外泛化能力。其二，多数方法依赖单次推理，缺乏迭代精修分割结果的能力。为克服上述局限，我们提出 IBISAgent——一种新颖的智能体式多模态大模型，将分割重构为以视觉为中心的多步决策过程。IBISAgent 使多模态大模型能够生成交错的推理与基于文本的点击动作、调用分割工具，并在无需修改架构的情况下生成高质量掩码。我们还设计了一个两阶段训练框架，由冷启动监督微调与带有定制化细粒度奖励的智能体强化学习组成。通过迭代式多轮视觉推理，IBISAgent 自然地促进了掩码精修，并提升了在复杂医学指代与推理分割任务中的鲁棒性。大量实验表明，IBISAgent 持续优于闭源与开源的当前最优方法。
英文原文摘要 1359 字符
Recent research on medical MLLMs has shifted its focus from image-level understanding to fine-grained, pixel-level comprehension. Although segmentation serves as the foundation for pixel-level understanding, existing approaches face two major challenges. First, they introduce implicit segmentation tokens and require joint fine-tuning of the MLLM and external pixel decoders, increasing the risk of catastrophic forgetting and limiting out-of-domain generalization. Second, most methods rely on single-pass reasoning and lack the ability to iteratively refine segmentation results. To overcome these limitations, we propose IBISAgent--a novel agentic MLLM that reformulates segmentation as a vision-centric, multi-step decision-making process. IBISAgent enables MLLMs to generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications. We also design a two-stage training framework consisting of cold-start SFT and agentic RL with tailored, fine-grained rewards. Through iterative multi-turn visual reasoning, IBISAgent naturally facilitates mask refinement and enhances robustness in complex medical referring and reasoning segmentation tasks. Extensive experiments demonstrate that IBISAgent consistently outperforms both closed-source and open-source SOTA methods.