Action-Sketcher：通过视觉草图从推理到动作的机器人操作
Action-Sketcher: From Reasoning to Action via Visual Sketches for Robotic Manipulation
提出可渲染点、框、箭头和关系类型的视觉草图作为中间表示，将语言意图外化为场景几何，为视觉-语言-动作模型提供可验证的推理-动作桥梁，可复用于需要空间消歧的机器人操作任务。
原文依据 we first introduce Visual Sketch, an implausible visual intermediate that renders points, boxes, arrows, and typed relations in the robot's current views
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论长程机器人操作对于实际部署日益重要，需要在复杂布局中进行空间消歧，并在动态交互下保持时序韧性。然而，现有的端到端和分层视觉-语言-动作（VLA）策略通常依赖纯文本线索，同时将规划意图保持为隐式，这削弱了在杂乱或欠指定场景中的指代视觉定位，阻碍了对具有闭环交互的长程目标进行有效任务分解，并通过掩盖动作选择背后的理由限制了因果解释。为了解决这些问题，我们首先引入视觉草图（Visual Sketch），一种不合理的视觉中间表示，在机器人当前视图中渲染点、框、箭头和类型化关系，以外化空间意图，将语言与场景几何连接起来，并在高层推理与低层控制之间提供人类可验证的桥梁。基于视觉草图，我们提出Action-Sketcher，一个VLA框架，以循环的“看→想→画→做”工作流运行，由自适应令牌门控策略协调推理触发、草图修订和动作发出，从而支持反应式修正和人类交互，同时保持实时动作预测。为了支持可扩展的训练和评估，我们整理了一个多样化的语料库，包含交错的图像、文本、视觉草图监督和动作序列，并使用多阶段课程配方训练Action-Sketcher，该配方结合了用于模态统一的交错序列对齐、用于精确语言视觉定位的语言到草图一致性，以及用草图到动作强化增强的模仿学习以提高鲁棒性。在杂乱场景和多物体任务上的大量实验，在仿真和真实世界任务中，显示出长程成功率的提高、对动态场景变化的更强鲁棒性，以及通过可编辑草图和逐步计划增强的可解释性。
英文原文摘要 1962 字符
Long-horizon robotic manipulation is increasingly important for real-world deployment, requiring spatial disambiguation in complex layouts and temporal resilience under dynamic interaction. However, existing end-to-end and hierarchical Vision-Language-Action (VLA) policies often rely on text-only cues while keeping plan intent latent, which undermines referential grounding in cluttered or underspecified scenes, impedes effective task decomposition of long-horizon goals with close-loop interaction, and limits causal explanation by obscuring the rationale behind action choices. To address these issues, we first introduce Visual Sketch, an implausible visual intermediate that renders points, boxes, arrows, and typed relations in the robot's current views to externalize spatial intent, connect language to scene geometry, and provide a human-verifiable bridge between high-level reasoning and low-level control. Building on Visual Sketch, we present Action-Sketcher, a VLA framework that operates in a cyclic See -> Think -> Sketch -> Act workflow coordinated by adaptive token-gated strategy for reasoning triggers, sketch revision, and action issuance, thereby supporting reactive corrections and human interaction while preserving real-time action prediction. To enable scalable training and evaluation, we curate diverse corpus with interleaved images, text, Visual Sketch supervision, and action sequences, and train Action-Sketcher with a multi-stage curriculum recipe that combines interleaved sequence alignment for modality unification, language-to-sketch consistency for precise linguistic grounding, and imitation learning augmented with sketch-to-action reinforcement for robustness. Extensive experiments on cluttered scenes and multi-object tasks, in simulation and on real-world tasks, show improved long-horizon success, stronger robustness to dynamic scene changes, and enhanced interpretability via editable sketches and step-wise plans.