推理调色板：通过潜变量情境化为（视觉）语言模型的可控探索调制推理
Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
为视觉语言模型提供一种在生成前对推理策略做潜变量采样的机制：从问答对均值池化嵌入经 VAE 采样潜变量，解码为可学习词元前缀拼接到提示前，可在 RL 训练中按需注入不同推理模式以提升探索多样性，可直接迁移到多模态推理与 RLHF 流程。
原文依据 a sampled latent is decoded into learnable token prefixes and prepended to the input prompt, modulating the model's internal reasoning trajectory
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论探索能力同时影响大（视觉）语言模型的推理时表现与强化学习（RL）训练，因为随机采样往往产生冗余的推理路径，缺乏高层多样性。本文提出推理调色板（Reasoning Palette），一种新的潜变量调制框架，为模型赋予一个随机潜变量以进行策略性情境化，在词元生成之前引导其内部规划。该潜变量情境由问答对的均值池化嵌入经变分自编码器（VAE）推断得到，每个采样得到的潜变量都可能编码一种不同的推理情境。在推理阶段，采样得到的潜变量被解码为可学习的词元前缀并拼接到输入提示之前，从而调制模型的内部推理轨迹。这样，模型在输出生成之前就对推理策略进行内部采样，进而塑造整个回复序列的风格与结构。一个简短的监督微调（SFT）预热阶段使模型适应这种潜变量条件。在 RL 优化中，推理调色板通过支持按需注入多样化推理模式来促进结构化探索，显著提升探索效率与持续学习能力。在多个推理基准上的实验表明，本方法能够对（视觉）语言模型的策略行为实现可解释且可控的调控，从而相较标准 RL 方法取得一致的性能提升。
英文原文摘要 1574 字符
Exploration capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models, as stochastic sampling often yields redundant reasoning paths with little high-level diversity. This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a stochastic latent variable for strategic contextualization, guiding its internal planning prior to token generation. This latent context is inferred from the mean-pooled embedding of a question-answer pair via a variational autoencoder (VAE), where each sampled latent potentially encodes a distinct reasoning context. During inference, a sampled latent is decoded into learnable token prefixes and prepended to the input prompt, modulating the model's internal reasoning trajectory. In this way, the model performs internal sampling over reasoning strategies prior to output generation, which shapes the style and structure of the entire response sequence. A brief supervised fine-tuning (SFT) warm-up phase allows the model to adapt to this latent conditioning. Within RL optimization, Reasoning Palette facilitates structured exploration by enabling on-demand injection for diverse reasoning modes, significantly enhancing exploration efficiency and sustained learning capability. Experiments across multiple reasoning benchmarks demonstrate that our method enables interpretable and controllable control over the (vision-) language model's strategic behavior, thereby achieving consistent performance gains over standard RL methods.