ReLaX：面向大型推理模型的潜空间探索推理
ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
ReLaX 利用 Koopman 算子理论对 LRM 的隐藏状态动力学进行线性化建模，并提出动态谱离散度（DSD）指标来量化潜空间探索程度，为视觉-语言模型中的强化学习微调提供了一种超越词元级多样性的探索调控手段，可直接用于多模态推理任务的策略优化。
原文依据 we leverage Koopman operator theory to obtain a linearized representation of their hidden state dynamics
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于可验证奖励的强化学习（RLVR）近期在提升大型推理模型（LRM）的推理能力方面展现出显著潜力。然而，RLVR 往往驱使策略趋向过度确定性，导致探索低效和策略过早收敛。尽管促进词元级多样性已被证明有助于缓解熵坍缩，但我们认为，词元生成背后的潜变量动力学编码了更为丰富的计算结构，可用于引导策略优化实现更有效的探索-利用权衡。为了对 LRM 的潜变量动力学进行可处理的分析与干预，我们利用 Koopman 算子理论获得其隐藏状态动力学的线性化表示。这使我们能够引入动态谱离散度（DSD），一种量化模型潜变量动力学异质性的新指标，作为策略探索的直接指示器。基于这些基础，我们提出了潜空间探索推理（ReLaX）框架，该框架在策略优化过程中显式地纳入潜变量动力学以调控探索与利用。在广泛的多模态和纯文本推理基准上的综合实验表明，ReLaX 能够持续激励推理能力并优于现有的词元级方法。项目见 https://github.com/ZhangShimin1/ReLaX。
英文原文摘要 1452 字符
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated remarkable potential in enhancing the reasoning capability of Large Reasoning Models (LRMs). However, RLVR often drives the policy toward over-determinism, resulting in ineffective exploration and premature policy convergence. While promoting token-level diversity has shown promise in mitigating entropy collapse, we argue that the latent dynamics underlying token generation encode a far richer computational structure for steering policy optimization toward a more effective exploration-exploitation tradeoff. To enable tractable analysis and intervention of the latent dynamics of LRMs, we leverage Koopman operator theory to obtain a linearized representation of their hidden state dynamics. This enables us to introduce Dynamic Spectral Dispersion (DSD), a new metric to quantify the heterogeneity of the model's latent dynamics, serving as a direct indicator of policy exploration. Building upon these foundations, we propose Reasoning with Latent eXploration (ReLaX), a framework that explicitly incorporates latent dynamics to regulate exploration and exploitation during policy optimization. Comprehensive experiments across a wide range of multimodal and text-only reasoning benchmarks show that ReLaX consistently incentivizes reasoning capability and outperforms existing token-level methods. Our project is available at https://github.com/ZhangShimin1/ReLaX.