FairLLaVA：面向大型视觉语言助手的美公平性感知参数高效微调
FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants
把互信息最小化当作表示层正则项，配合低秩适配器做插件式微调，可在不改动 MLLM 主干的前提下压制视觉指令微调中的群体偏差；做医学影像报告生成或 VQA 时，这套「公平性正则 + LoRA」的组合可直接迁移到自己的多模态模型上。
原文依据 By minimizing the mutual information between target attributes, FairLLaVA regularizes the model's representations to be demographic-invariant.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型虽然在图像条件生成上表现强大，但在不同人口群体上可能表现出不均衡的性能，这凸显了公平性风险。在安全攸关的临床场景中，此类差异可能导致诊断叙述不平等，并削弱对 AI 辅助决策的信任。公平性在纯视觉模型和纯语言模型中已得到广泛研究，但其对多模态大语言模型的影响仍鲜有探索。为缓解这些偏差，我们提出 FairLLaVA，一种参数高效微调方法，可在不损害整体性能的前提下缓解视觉指令微调中的群体差异。通过最小化目标属性之间的互信息，FairLLaVA 对模型表示施加正则，使其对人口统计属性不变。该方法可作为轻量级插件集成，借助低秩适配器微调保持高效性，并提供了一种与架构无关的公平视觉指令遵循方案。在大规模胸部放射学报告生成与皮肤镜视觉问答基准上的大量实验表明，FairLLaVA 在多种医学影像模态上持续缩小群体间差异，同时提升公平性加权的临床性能与自然语言生成质量。代码见 https://github.com/bhosalems/FairLLaVA。
英文原文摘要 1389 字符
While powerful in image-conditioned generation, multimodal large language models (MLLMs) can display uneven performance across demographic groups, highlighting fairness risks. In safety-critical clinical settings, such disparities risk producing unequal diagnostic narratives and eroding trust in AI-assisted decision-making. While fairness has been studied extensively in vision-only and language-only models, its impact on MLLMs remains largely underexplored. To address these biases, we introduce FairLLaVA, a parameter-efficient fine-tuning method that mitigates group disparities in visual instruction tuning without compromising overall performance. By minimizing the mutual information between target attributes, FairLLaVA regularizes the model's representations to be demographic-invariant. The method can be incorporated as a lightweight plug-in, maintaining efficiency with low-rank adapter fine-tuning, and provides an architecture-agnostic approach to fair visual instruction following. Extensive experiments on large-scale chest radiology report generation and dermoscopy visual question answering benchmarks show that FairLLaVA consistently reduces inter-group disparities while improving both equity-scaled clinical performance and natural language generation quality across diverse medical imaging modalities. Code can be accessed at https://github.com/bhosalems/FairLLaVA.