通过视觉符号诊断、纠正并学习操作失败
Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
用显式视觉符号（如箭头、标记）作为中间表示来标注和生成纠正动作，为 VLA 失败恢复提供了一种可复用的视觉提示与数据构造范式；ViFailback-Bench 的 11 项细粒度 VQA 任务也可直接用于评测 VLM 的具身失败诊断能力。
原文依据 Our framework utilizes explicit visual symbols to enhance annotation efficiency.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视觉语言动作（VLA）模型近期在机器人操作方面取得了显著进展，但在失败诊断与从失败中学习方面仍存在局限。此外，现有的失败数据集大多是在仿真中以程序化方式生成的，这限制了它们向真实世界的泛化能力。鉴于此，我们提出 ViFailback，一个旨在诊断机器人操作失败并提供文本与视觉纠正指导的框架。该框架利用显式视觉符号来提升标注效率。我们进一步发布了 ViFailback 数据集，这是一个大规模集合，包含 58,128 个视觉问答（VQA）对及其对应的 5,202 条真实世界操作轨迹。基于该数据集，我们建立了 ViFailback-Bench，一个由 11 项细粒度 VQA 任务组成的基准，用于评估视觉语言模型（VLM）的失败诊断与纠正能力，其中包含面向封闭式评估的 ViFailback-Bench Lite 和面向开放式评估的 ViFailback-Bench Hard。为证明该框架的有效性，我们构建了 ViFailback-8B VLM，它不仅在 ViFailback-Bench 上取得了显著的总体性能提升，还能生成用于纠正动作指导的视觉符号。最后，通过将 ViFailback-8B 与一个 VLA 模型集成，我们开展了真实世界机器人实验，证明其能够辅助 VLA 模型从失败中恢复。项目主页：https://x1nyuzhou.github.io/vifailback.github.io/
英文原文摘要 1532 字符
Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,128 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project page: https://x1nyuzhou.github.io/vifailback.github.io/