面向三维灵巧操作的结构化动作 Transformer
Structural Action Transformer for 3D Dexterous Manipulation
把动作块从时序序列重构为无序的逐关节轨迹序列，并用关节码本编码运动学先验，使 Transformer 能以变长序列原生处理不同自由度的机械手；这一「结构化动作表示 + 点云条件流匹配」的思路可迁移到任何需要跨本体泛化的视觉-动作策略学习。
原文依据 We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论通过模仿学习从异构数据集中获得人类水平的机器人灵巧性，受制于跨本体技能迁移的难题，对高自由度机械手尤为突出。现有方法往往依赖二维观测和以时序为中心的动作表示，难以捕捉三维空间关系，也无法处理本体异构性。本文提出结构化动作 Transformer（SAT），一种新的三维灵巧操作策略，以结构化为中心的视角挑战这一范式。我们将每个动作块重新表述为变长、无序的逐关节轨迹序列，而非时序序列。这种结构化表述使 Transformer 能够原生处理异构本体，将关节数量视为可变序列长度。为编码结构先验并消解歧义，我们引入本体关节码本，用于嵌入每个关节的功能角色与运动学属性。我们的模型通过连续时间流匹配目标，学习从三维点云生成这些轨迹。我们在大规模异构数据集上进行预训练，并在仿真与真实世界灵巧操作任务上微调，以验证所提方法。本方法一致优于所有基线，展现出更优的样本效率与有效的跨本体技能迁移。这种以结构化为中心的表示，为扩展高自由度、异构操作器的策略提供了新路径。
英文原文摘要 1529 字符
Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint's functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators.