MedTVT-R1：赋能医学推理与诊断的多模态大语言模型
MedTVT-R1: A Multimodal LLM Empowering Medical Reasoning and Diagnosis
模态感知层对异构模态（影像、生理信号、文本）做自适应加权融合，配合 GRPO 强化微调与 Jaccard 奖励，可直接迁移到需要多源视觉证据联合推理的医学 VQA 与报告生成任务。
原文依据 MedTVT-R1 incorporates a modality perception layer to capture inter-modal dependencies and adaptively weight modality contributions.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论准确且可解释的多疾病诊断仍是医学研究中的关键挑战，尤其是在利用异构多模态医学数据时。现有方法往往依赖单模态数据，限制了其对复杂疾病的全面理解能力。为此，我们提出 MedTVT-R1，一种新颖的多模态大语言模型（MLLM）框架，旨在整合临床多模态数据进行多疾病推理与诊断。我们构建了 MedTVT-QA，这是一个经过精心整理的指令数据集，以证据链方式提供面向生理层面解读与疾病层面诊断的问答对。MedTVT-R1 引入模态感知层以捕获模态间依赖关系并自适应地加权各模态贡献。此外，我们采用基于组相对策略优化（GRPO）的强化微调，并配合 Jaccard 奖励函数，以增强诊断推理能力。实验结果表明，MedTVT-R1 在多模态特征利用与多疾病诊断方面具有优越性，为诊断报告生成与共病推理等临床应用提供了重要潜力。数据集与代码将在 GitHub 上公开。
英文原文摘要 1261 字符
Accurate and interpretable multi-disease diagnosis remains a critical challenge in medical research, particularly when leveraging heterogeneous multimodal medical data. Current approaches often rely on single-modal data, limiting their ability to comprehensively understand complex diseases. To address this, we propose MedTVT-R1, a novel Multimodal Large Language Model (MLLM) framework designed to integrate clinical multimodal data for reasoning and diagnosing multiple diseases. We construct MedTVT-QA, a curated instruction dataset that provides question-answer pairs for physiological-level interpretations and disease-level diagnoses with a Chain of Evidence approach. MedTVT-R1 incorporates a modality perception layer to capture inter-modal dependencies and adaptively weight modality contributions. Additionally, we employ Group Relative Policy Optimization (GRPO)-based Reinforcement Fine-Tuning with a Jaccard Reward function to enhance diagnostic reasoning. Experimental results demonstrate MedTVT-R1's superiority in multimodal feature utilization and multi-disease diagnosis, offering significant potential for clinical applications such as diagnostic report generation and comorbidity reasoning. The dataset and code will be available on GitHub.