TTRV：面向视觉语言模型的测试时强化学习
TTRV: Test-Time Reinforcement Learning for Vision Language Models
把 GRPO 式的强化学习从训练阶段搬到推理阶段：对每个测试样本多次采样，用基座模型输出频率构造奖励，并加入经验分布熵正则控制多样性，从而在无任何标注的情况下在线提升 VLM 的识别与 VQA 表现，可直接复用于任意现成 LVLM 的测试时增强。
原文依据 we propose TTRV to enhance vision-language understanding by adapting the model on-the-fly at inference time, without the need for any labeled data
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论现有强化学习中提取奖励信号的方法通常依赖标注数据和专门的训练划分，这一设定与人类直接从环境中学习的方式形成对比。本文提出 TTRV，通过在推理时对模型进行即时适配来增强视觉语言理解，且无需任何标注数据。具体而言，我们在每个测试样本上多次推理，并基于基座模型输出的频率设计奖励，从而对组相对策略优化（GRPO）框架进行增强。此外，我们还提出同时奖励模型获得输出经验分布的低熵，以此控制模型输出的多样性。我们的方法在目标识别和视觉问答（VQA）上均取得一致提升，最高分别提升 52.4% 和 29.8%，在 16 个数据集上平均提升 24.6% 和 10.0%。值得注意的是，在图像识别任务上，将 TTRV 应用于 Intern-VL-8B 后在 8 个基准上平均超越 GPT-4o 达 2.3%，同时在 VQA 上仍保持很强的竞争力，表明测试时强化学习能够匹配甚至超越最强的专有模型。最后，我们发现了视觉语言模型测试时强化学习的许多有趣性质：例如，即使在数据极度受限的场景下，仅在一个随机选取的无标注测试样本上进行适配，TTRV 在识别任务上仍能带来最高 5.5% 的显著提升。
英文原文摘要 1534 字符
Existing methods for extracting reward signals in Reinforcement Learning typically rely on labeled data and dedicated training splits, a setup that contrasts with how humans learn directly from their environment.In this work, we propose TTRV to enhance vision-language understanding by adapting the model on-the-fly at inference time, without the need for any labeled data.Concretely, we enhance the Group Relative Policy Optimization (GRPO) framework by designing rewards based on the frequency of the base model's output, while inferring on each test sample multiple times.Further, we also propose to control the diversity of the model's output by simultaneously rewarding the model for obtaining low entropy of the output empirical distribution.Our approach delivers consistent gains across both object recognition and visual question answering (VQA), with improvements of up to 52.4% and 29.8%, respectively, and average boosts of 24.6% and 10.0% across 16 datasets. Remarkably, on image recognition, TTRV applied to Intern-VL-8B surpasses GPT-4o by an average of 2.3% over 8 benchmarks, while remaining highly competitive on VQA, demonstrating that test-time reinforcement learning can match or exceed the strongest proprietary models. Finally, we find many interesting properties of test-time RL for VLMs: for example, even in extremely data-constrained scenarios, where adaptation is performed on a single randomly chosen unlabeled test example, \method still yields non-trivial improvements of up to 5.5% in recognition tasks.