重新思考基于视觉语言模型的机器人操作的中间表示
Rethinking Intermediate Representation for VLM-based Robot Manipulation
把中间表示拆成「操作词表 + 语法」的思路，以及用上下文学习做开放词表部件级分割（杯柄、壶口）的范式，可直接迁移到需要细粒度部件定位的具身操作与视觉定位任务中。
原文依据 we design a novel open-vocabulary segmentation paradigm with an in-context learning strategy to precisely localize fine-grained object parts for manipulation
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言模型（VLM）现已成为实现稳健机器人操作的重要组成部分。然而，用它把人类指令转化为可解析为动作的中间表示，往往需要在 VLM 可理解性与泛化性之间做出权衡。受上下文无关文法结构的启发，我们设计了名为 SEAM 的语义装配（Semantic Assembly）表示，通过将中间表示分解为词表与文法两部分。这样做使我们得到一个由语义丰富的操作构成的简洁词表，以及一个对 VLM 友好的文法，用于处理多样的未见任务。此外，我们设计了一种新颖的开放词表分割范式，采用上下文学习策略，以在所有最优的并行工作中最短的推理时间，有效地精确局部化用于操作的细粒度物体部件（例如杯柄、茶壶口）。随后，我们为动作泛化性与 VLM 可理解性制定了新的度量指标，用以评估主流表示，并展示了 SEAM 在这两方面都具有强劲表现。大量真实世界实验进一步表明，SEAM 在不同设置与任务下均达到最优性能。
英文原文摘要 1169 字符
Vision-Language Model (VLM) is now an important component to enable robust robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate representation often needs a tradeoff between VLM-comprehensibility and generalizability. Inspired by context-free grammar structure, we design the Semantic Assembly representation named SEAM, by decomposing the intermediate representation into vocabulary and grammar. Doing so leads us to a concise vocabulary of semantically-rich operations and a VLM-friendly grammar for handling diverse unseen tasks. Also, we design a novel open-vocabulary segmentation paradigm with an in-context learning strategy to precisely localize fine-grained object parts for manipulation (e.g., cup handle, teapot opening) effectively with the shortest inference time over all state-of-the-art parallel works. We then formulate new metrics for action-generalizability and VLM-comprehensibility to evaluate mainstream representations, demonstrating the strong performance of SEAM on both aspects. Extensive real-world experiments further manifest the SOTA performance of SEAM under varying settings and tasks.