通过激活重放提升大型多模态模型的推理能力
Boosting Reasoning in Large Multimodal Models via Activation Replay
提出一种无需训练的测试时干预方法：通过操纵视觉 token，将基座 LMM 的低熵激活重放到 RLVR 后训练的 LMM 中，以提升多模态推理性能。该方法可复用于任何基于 RLVR 微调的视觉语言模型，以缓解推理覆盖变窄的问题。
原文依据 Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论最近，带可验证奖励的强化学习（RLVR）已成为激发大型多模态模型（LMM）推理能力的有效方法，但这一后训练范式背后的机制尚不清楚。我们首先从 logit lens 的视角探究输入激活如何受 RLVR 影响。通过对多个后训练 LMM 的系统性研究，我们发现 RLVR 会意外地改变低熵激活，而高熵激活受影响较小。我们进一步通过控制实验证明，这种现象与 LMM 的推理相关，表明调节低熵激活可能具有有益作用。为此，我们提出了激活重放（Activation Replay），这是一种新颖、简单且有效的免训练方法，无需昂贵的策略优化即可提升后训练 LMM 的多模态推理能力。我们的设计涉及在测试时操纵视觉 token，将基座 LMM 输入上下文中的低熵激活重放到 RLVR 对应模型中。激活重放在多种场景下触发更好的推理，包括数学、类 o3 的视觉智能体和视频推理。我们进一步表明，激活重放提升了 Pass@K 并缓解了 RLVR 推理覆盖范围变窄的问题。我们的设计与替代方案进行了比较，例如重放高熵激活而非低熵激活，或直接进行跨模型干预而非操纵输入 token，证明了我们实现的优越性。代码将公开。
英文原文摘要 1620 字符
Recently, Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach to incentivizing reasoning capability in Large Multimodal Models (LMMs), while the underlying mechanisms behind this post-training paradigm are poorly understood. We begin by exploring how input activations are affected by RLVR through the perspective of logit lens. Our systematic investigations across multiple post-trained LMMs suggest that RLVR shifts low-entropy activations unexpectedly, while high-entropy ones are less affected. We further demonstrate that such phenomena are associated with LMM reasoning by controlled experiments, suggesting a potentially beneficial role of modulating low-entropy activations. To this end, we propose Activation Replay, a novel simple yet effective training-free approach that boosts multimodal reasoning of post-trained LMMs without requiring expensive policy optimization. Our design involves manipulation of visual tokens at test time, replaying low-entropy activations from the input context of base LMMs to regulating the RLVR counterparts. Activation Replay triggers better reasoning across diverse scenarios, including mathematics, o3-like visual agents, and video reasoning. We further show that Activation Replay boosts Pass@K and mitigates narrower reasoning coverage of RLVR. Our design is compared against alternative choices, such as replaying high-entropy activations instead of low-entropy ones, or direct cross-model intervention instead of manipulating input tokens, demonstrating the superiority of our implementation. Codes will be made publicly available.