MapReduce LoRA：推进生成模型多偏好优化的帕累托前沿
MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
MapReduce LoRA 用并行训练偏好专属 LoRA 专家再迭代合并进共享基座，RaTE 用奖励专属词元嵌入在推理时组合，为文生图/文生视频模型提供可复用的多奖励对齐与推理时偏好切换手段。
原文依据 MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于人类反馈的强化学习（RLHF）配合奖励模型，已推动生成模型在人类审美与感知偏好上的对齐。然而，联合优化多个奖励往往会带来对齐税——提升某一维度的同时损害其他维度。为此，我们提出两种互补的方法：MapReduce LoRA 与奖励感知词元嵌入（RaTE）。MapReduce LoRA 并行训练偏好专属的低秩适配专家，并迭代地将其合并以精炼共享基座模型；RaTE 学习奖励专属的词元嵌入，在推理时进行组合以实现灵活的偏好控制。在文本到图像生成任务（Stable Diffusion 3.5 Medium 与 FLUX.1-dev）上，GenEval、PickScore 与 OCR 分别提升 36.1%、4.6%、55.7%，以及 32.7%、4.3%、67.1%。在文本到视频生成任务（HunyuanVideo）上，视觉质量与运动质量分别提升 48.1% 与 90.0%。在语言任务 Helpful Assistant 上，使用 Llama-2 7B，有用性与无害性分别提升 43.4% 与 136.7%。我们的框架在跨模态的多偏好对齐上树立了新的最优方案。
英文原文摘要 1158 字符
Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax--improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.