AVATAR：通过强化学习在视频上看见、听见与推理
AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
对做视频多模态推理的人，这篇给出了一个可直接复用的强化学习后训练配方：用离策略样本回放缓解 GRPO 的组内奖励同质化（优势消失），并用时序优势塑形把信用集中到规划与综合两端阶段，可迁移到任意视频/音视频 VLM 的 RL 微调流程。
原文依据 an off-policy training architecture that improves sample efficiency and resolves vanishing advantages by reusing past experiences
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论长时序视频上的多模态推理具有挑战性，因为需要跨模态进行精确的时空融合与对齐。尽管组相对策略优化（GRPO）等近期方法在该领域展现出潜力，但它们存在三个关键局限：(1) 其同策略设计导致数据效率低下；(2) 优势消失问题，即组内相同或近乎相同的奖励会产生零值优势，从而消除学习信号；(3) 统一的信用分配无法突出关键推理步骤。我们提出 AVATAR（用于对齐与推理的音视频智能体），该框架通过两个核心组件解决上述局限：(1) 一种离策略训练架构，通过复用具有更大奖励多样性的过往经验来提升样本效率并解决优势消失问题；(2) 时序优势塑形（TAS），一种强调早期（规划）与后期（综合）推理阶段的信用分配策略。AVATAR 在多个基准上取得强劲性能，在 MMVU 上超过 Qwen2.5-Omni 基线 +5.4，在 OmniBench 上超过 +4.9，在 Video-Holmes 上超过 +4.5。此外，它在 OmniBench 上超过标准 GRPO +3.7，在 Video-Holmes 上超过 +1.9，同时展现出 5 倍的样本效率，达到目标性能所需的生成补全数减少 80 次。
英文原文摘要 1400 字符
Multimodal reasoning over long-horizon video is challenging due to the need for precise spatiotemporal fusion and alignment across modalities. While recent methods such as Group Relative Policy Optimization (GRPO) have shown promise in this domain, they suffer from three key limitations: (1) data inefficiency from their on-policy design, (2) a vanishing advantage problem, where identical or near-identical rewards within a group eliminate the learning signal by producing zero-valued advantages, and (3) uniform credit assignment that fails to emphasize critical reasoning steps. We introduce AVATAR (Audio-Video Agent for Alignment and Reasoning, a framework that addresses these limitations through two core components: (1) an off-policy training architecture that improves sample efficiency and resolves vanishing advantages by reusing past experiences with greater reward diversity, and (2) Temporal Advantage Shaping (TAS), a credit assignment strategy that emphasizes early (planning) and late (synthesis) reasoning phases. AVATAR achieves strong performance across various benchmarks, outperforming the Qwen2.5-Omni baseline by +5.4 on MMVU, +4.9 on OmniBench, and +4.5 on Video-Holmes. Furthermore, it surpasses standard GRPO by +3.7 on OmniBench and +1.9 on Video-Holmes, while demonstrating 5 times sample efficiency, requiring 80 fewer generated completions to reach target performance.