SRPO：面向视觉-语言-动作模型的自参照策略优化
SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
用世界模型的潜空间编码替代像素级比较来度量轨迹进度，为视觉驱动的机器人策略提供了一种无需人工奖励工程、可跨环境泛化的稠密奖励构造方式，可直接迁移到其他 VLA 后训练或视觉轨迹评估场景。
原文依据 we utilize the compressed, transferable encodings from a world model's latent space
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉-语言-动作（VLA）模型在机器人操作中表现优异，但受制于对专家演示的严重依赖，导致演示偏差并限制性能。强化学习（RL）是突破这些限制的关键后训练策略，然而当前的 VLA-RL 方法（包括基于组的优化方法）受困于严重的奖励稀疏性。依赖二值成功指标浪费了失败轨迹中的宝贵信息，导致训练效率低下。为解决这一问题，我们提出自参照策略优化（SRPO），一种新的 VLA-RL 框架。SRPO 无需外部演示或人工奖励工程，而是利用模型在当前训练批次内生成的自身成功轨迹作为自参照，从而为失败的尝试分配进度奖励。一项核心创新是使用潜变量世界表征来稳健地度量行为进度。我们不再依赖原始像素或需要领域特定的微调，而是利用世界模型潜空间中的压缩、可迁移编码。这些表征天然捕捉跨环境的进度模式，从而实现准确且泛化的轨迹比较。在 LIBERO 基准上的实证评估表明 SRPO 的高效性与有效性。从成功率为 48.9% 的监督基线出发，SRPO 仅用 200 步 RL 即达到 99.2% 的最优成功率，在无任何额外监督的情况下实现 103% 的相对提升。此外，SRPO 展现出显著的鲁棒性，在 LIBERO-Plus 基准上取得 167% 的性能提升。
英文原文摘要 1707 字符
Vision-Language-Action (VLA) models excel in robotic manipulation but are constrained by their heavy reliance on expert demonstrations, leading to demonstration bias and limiting performance. Reinforcement learning (RL) is a vital post-training strategy to overcome these limits, yet current VLA-RL methods, including group-based optimization approaches, are crippled by severe reward sparsity. Relying on binary success indicators wastes valuable information in failed trajectories, resulting in low training efficiency. To solve this, we propose Self-Referential Policy Optimization (SRPO), a novel VLA-RL framework. SRPO eliminates the need for external demonstrations or manual reward engineering by leveraging the model's own successful trajectories, generated within the current training batch, as a self-reference. This allows us to assign a progress-wise reward to failed attempts. A core innovation is the use of Latent World Representations to measure behavioral progress robustly. Instead of relying on raw pixels or requiring domain-specific fine-tuning, we utilize the compressed, transferable encodings from a world model's latent space. These representations naturally capture progress patterns across environments, enabling accurate, generalized trajectory comparison. Empirical evaluations on the LIBERO benchmark demonstrate SRPO's efficiency and effectiveness. Starting from a supervised baseline with 48.9% success, SRPO achieves a new state-of-the-art success rate of 99.2% in just 200 RL steps, representing a 103% relative improvement without any extra supervision. Furthermore, SRPO shows substantial robustness, achieving a 167% performance improvement on the LIBERO-Plus benchmark.