SafeGRPO：通过规则约束的策略优化实现自奖励多模态安全对齐
SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization
对做多模态大模型安全对齐的研究者，这篇给出了可复用的配方：用带视觉/文本/组合安全标签的数据集构造规则化可验证奖励，再接入 GRPO 做自奖励强化学习，并配合分步安全思维链约束推理轨迹；可直接迁移到其他 VLM 的安全微调或推理时风险感知训练。
原文依据 integrates rule-governed reward construction into GRPO, enabling interpretable and verifiable optimization of reasoning safety
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论多模态大语言模型（MLLMs）已展现出令人印象深刻的推理与指令遵循能力，但其扩展后的模态空间引入了由复杂图文交互产生的新型组合式安全风险。此类跨模态耦合即使在单个输入本身无害时也可能产生不安全的语义，暴露出当前 MLLMs 脆弱的安全意识。尽管近期工作通过引导模型推理潜在风险来增强安全性，但不受约束的推理轨迹可能损害对齐；虽然组相对策略优化（GRPO）提供了无需人工监督的自奖励式精炼，但它缺乏针对推理安全性的可验证信号。为此，我们提出 SafeGRPO，一个自奖励的多模态安全对齐框架，将规则约束的奖励构造集成到 GRPO 中，从而实现对推理安全性的可解释且可验证的优化。基于所构建的 SafeTag-VL-3K 数据集（带有显式的视觉、文本及组合安全标签），SafeGRPO 执行分步引导的安全思考，以强制结构化推理与行为对齐，在多种基准上显著提升多模态安全意识、组合鲁棒性与推理稳定性，同时不牺牲通用能力。
英文原文摘要 1309 字符
Multimodal large language models (MLLMs) have demonstrated impressive reasoning and instruction-following capabilities, yet their expanded modality space introduces new compositional safety risks that emerge from complex text-image interactions.Such cross-modal couplings can produce unsafe semantics even when individual inputs are benign, exposing the fragile safety awareness of current MLLMs.While recent works enhance safety by guiding models to reason about potential risks, unregulated reasoning traces may compromise alignment; although Group Relative Policy Optimization (GRPO) offers self-rewarded refinement without human supervision, it lacks verifiable signals for reasoning safety.To address this, we propose **SafeGRPO** a self-rewarded multimodal safety alignment framework that integrates rule-governed reward construction into GRPO, enabling interpretable and verifiable optimization of reasoning safety. Built upon the constructed **SafeTag-VL-3K** dataset with explicit visual, textual, and combined safety tags, SafeGRPO performs **step-guided safety thinking** to enforce structured reasoning and behavior alignment, substantially improving multimodal safety awareness, compositional robustness, and reasoning stability across diverse benchmarks without sacrificing general capabilities.