定位、结构化与渲染：连接3D与2D视觉-语言-动作模型用于机器人操作
Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation
将可微渲染作为连接3D几何推理与2D视觉语义的桥梁，通过生成可微图像并让2D VLA的梯度回传到3D表示，为视觉研究者提供了一种融合显式3D空间关系与图像级语义的通用手段，可借鉴到需要空间推理的视觉任务中。
原文依据 We introduce DiffRender-VLA, a differentiable rendering-based framework that bridges 3D and 2D Vision-Language-Action models through gradient-consistent visual mediation.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论复杂3D环境中的机器人操作需要将空间推理与直观视觉感知统一起来，而当前的视觉-语言-动作范式分别处理这两种能力。3D VLA擅长几何与物理推理，但缺乏直观的图像级理解和密集视觉语义；相反，2D VLA（即使使用深度图像）提供丰富的视觉直觉和语义连续性，但缺少显式的空间全局定位。我们提出DiffRender-VLA，一个基于可微渲染的框架，通过梯度一致的视觉中介连接3D和2D视觉-语言-动作模型。它通过以下方式生成可微图像：用世界对齐的立方体标记定位下一个末端执行器目标，可微地结构化周围几何体（其颜色编码与标记的空间关系），并渲染优化后的自适应视角以揭示目标-环境空间关系。这些可微图像作为视觉桥梁，嵌入空间语义，同时允许来自2D VLA的梯度反向传播到3D表示中，从而将几何推理与视觉感知耦合起来。这一闭环可微回路统一了推理与感知，在遮挡、杂乱和复杂空间操作任务下显著提升性能，相比最优方法平均提升+12.1%。代码见 https://github.com/zyl123456aB/DIFFVLA。
英文原文摘要 1488 字符
Robotic manipulation in complex 3D environments requires unifying spatial reasoning with intuitive visual perception, which is a capability that current Vision-Language-Action paradigms address separately. While 3D VLAs excel in geometric and physical reasoning, they lack intuitive, image-level understanding and dense visual semantics; conversely, 2D VLAs (even with depth image) provide rich visual intuition and semantic continuity but miss explicit spatial global grounding. We introduce DiffRender-VLA, a differentiable rendering-based framework that bridges 3D and 2D Vision-Language-Action models through gradient-consistent visual mediation. It generates differentiable images by localizing the next end-effector target with a world-aligned cube marker, differentiably structuring surrounding geometry whose color encodes spatial relations to the marker, and rendering adaptive viewpoints optimized to reveal the target-environment spatial relationships. These differentiable images serve as visual bridges, embedding spatial semantics while allowing gradients from 2D VLAs to backpropagate into 3D representations, thereby coupling geometric reasoning with visual perception. This closed differentiable loop unifies reasoning and perception, substantially improving performance under occlusion, clutter, and complex spatial manipulation tasks, achieving average improvements of +12.1% over state-of-the-art methods. Codes are available at https://github.com/zyl123456aB/DIFFVLA.