学习具有 3D 空间先验的手术机器人操作
Learning Surgical Robotic Manipulation with 3D Spatial Priors
把立体内窥镜图像中隐含的 3D 几何线索通过微调的几何 Transformer 提取为 3D 潜变量表征，再用轻量连接器对齐到动作空间，为视觉研究者提供了一条「免显式重建、端到端注入 3D 先验」的范式，可直接迁移到其他需要空间感知的视觉运动策略。
原文依据 finetune a powerful geometric transformer to extract robust 3D latent representations from stereo endoscopes images
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论实现 3D 空间感知对于手术机器人操作至关重要，因为这类操作要求精确而精细的动作。现有方法要么在操作之前先显式重建手术场景，要么通过增加腕部相机来补充默认立体内窥镜的多视角特征。然而，这两种范式都存在明显局限：前者由于多阶段特性容易导致误差累积，并阻碍端到端优化；后者在临床实践中很少被采用，因为腕部相机会干扰手术机械臂的运动。在本工作中，我们提出空间手术 Transformer（SST），这是一种端到端视觉运动策略，通过直接挖掘内窥镜图像中蕴含的 3D 空间线索，赋予手术机器人 3D 空间感知能力。首先，我们构建了 Surgical3D，这是一个大规模照片级真实感数据集，包含 30K 对具有精确 3D 几何的立体内窥镜图像，以缓解手术场景中 3D 数据稀缺的问题。基于 Surgical3D，我们微调了一个强大的几何 Transformer，从立体内窥镜图像中提取稳健的 3D 潜变量表征。随后，这些表征通过一个轻量的多级空间特征连接器（MSFC）与机器人的动作空间无缝对齐，整个过程都在以内窥镜为中心的坐标系中完成。大量真实机器人实验表明，SST 在打结和离体器官解剖等复杂手术任务上取得了最优性能，并展现出强大的空间泛化能力，这标志着向实际临床部署迈出了重要一步。数据集和代码将公开发布。
英文原文摘要 1695 字符
Achieving 3D spatial awareness is crucial for surgical robotic manipulation, where precise and delicate operations are required. Existing methods either explicitly reconstruct the surgical scene prior to manipulation, or enhance multi-view features by adding wrist-mounted cameras to supplement the default stereo endoscopes. However, both paradigms suffer from notable limitations: the former easily leads to error accumulation and prevents end-to-end optimization due to its multi-stage nature, while the latter is rarely adopted in clinical practice since wrist-mounted cameras can interfere with the motion of surgical robot arms. In this work, we introduce the Spatial Surgical Transformer (SST), an end-to-end visuomotor policy that empowers surgical robots with 3D spatial awareness by directly exploring 3D spatial cues embedded in endoscopic images. First, we build Surgical3D, a large-scale photorealistic dataset containing 30K stereo endoscopic image pairs with accurate 3D geometry, addressing the scarcity of 3D data in surgical scenes. Based on Surgical3D, we finetune a powerful geometric transformer to extract robust 3D latent representations from stereo endoscopes images. These representations are then seamlessly aligned with the robot's action space via a lightweight multi-level spatial feature connector (MSFC), all within an endoscope-centric coordinate frame. Extensive real-robot experiments demonstrate that SST achieves state-of-the-art performance and strong spatial generalization on complex surgical tasks such as knot tying and ex-vivo organ dissection, representing a significant step toward practical clinical deployment. The dataset and code will be released.