面向视觉语言模型的动态感知偏好优化
Dynamics-Aware Preference Optimization for Vision-Language Models
对做 VLM 偏好微调的人，这篇给出了一个可直接套用的两阶段训练配方：先用低权重“温和负样本”做约束 SFT 压平损失面，再按模型平均每 token 对数概率自适应缩放负样本梯度，从而抑制简单负样本带来的无信息梯度；该权重设计可迁移到任意 DPO 类视觉语言对齐流程。
原文依据 a competence-aware cooling weight that adaptively scales negative gradients according to the model's average per-token log-probability
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于偏好的视觉语言模型（VLM）微调以不稳定著称，因为平凡错误的负样本会注入无信息的梯度，扭曲优化过程并损害校准。本文从学习动力学的视角重新审视这一问题，识别出一种核心病理——挤压效应，即简单负样本尽管损失可忽略，却仍保留幅度大且方向错位的梯度。为此，我们提出冷却加权直接偏好优化（CW-DPO），这是一个先平滑、后稳定的两阶段框架。第一阶段采用带低权重“温和负样本”的约束式 SFT，以正则化过度自信的分布并压平损失面。第二阶段引入能力感知的冷却权重，依据模型平均每 token 对数概率自适应缩放负样本梯度，在抑制无信息更新的同时强调困难的同策略对比。这种动态感知加权有效缓解了挤压效应，并带来更平滑的收敛。在 COCO、Flickr30k、NoCaps、MMMU 和 MMBench1.1 等主流基准上的大量实验表明，CW-DPO 取得了最优性能，例如相较 PPO 提升 3.4 CIDEr、在 MMMU 上绝对准确率提升 2.4%，同时改善了校准并将收敛步数减半。这表明“先平滑、后冷却”构成了一条简单而通用的原则，可用于稳健的 VLM 偏好优化。
英文原文摘要 1485 字符
Preference-based finetuning of vision-language models (VLMs) is notoriously unstable, as trivially wrong negatives inject uninformative gradients that distort optimization and degrade calibration. This work revisits this issue through the lens of learning dynamics and identifies a core pathology, the squeezing effect, where easy negatives retain large, misaligned gradients despite having negligible loss.To address this, we propose Cooling-Weighted Direct Preference Optimization (CW-DPO), a two-stage framework that first smooths and then stabilizes the alignment process. Stage 1 employs a constrained SFT phase with low-weight "gentle negatives" to regularize overconfident distributions and flatten the loss landscape. Stage 2 introduces a competence-aware cooling weight that adaptively scales negative gradients according to the model's average per-token log-probability, suppressing uninformative updates while emphasizing hard, on-policy contrasts. This dynamics-aware weighting effectively mitigates the squeezing effect and enables smoother convergence.Extensive experiments on mainstream benchmarks--including COCO, Flickr30k, NoCaps, MMMU, and MMBench1.1--show that CW-DPO achieves state-of-the-art performance, for example +3.4 CIDEr over PPO and +2.4% absolute accuracy on MMMU, while improving calibration and halving convergence steps. This demonstrates that smoothing before cooling constitutes a simple yet general principle for robust VLM preference optimization.