描述任何事物在任何地点任何时刻
Describe Anything Anywhere At Any Moment
提出基于优化的前端，将局部描述模型（如 DAM）批量处理以加速推理，并构建分层 4D 场景图作为全局时空一致记忆，为视觉语言导航与具身智能提供可查询的 4D 表示。
原文依据 DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从增强现实到大规模环境中的机器人自主性，计算机视觉与机器人应用需要同时捕获用于精确语言定位的几何结构和语义细节的时空记忆框架。现有方法面临权衡：生成丰富的开放词表描述时，若需将这些描述定位到 3D 中，则以牺牲实时性能为代价。为解决这些挑战，我们提出描述任何事物在任何地点任何时刻（DAAAM），一种用于大规模实时 4D 场景理解的新型时空记忆框架。DAAAM 引入一种新颖的基于优化的前端，利用批处理从局部描述模型（如描述任何事物模型 DAM）推断详细语义描述，将推理速度提升一个数量级以支持在线部署。它利用这种语义理解构建分层 4D 场景图（SG），作为有效的全局空间与时间一致记忆表示。DAAAM 在保持实时性能的同时，构建具有详细几何定位描述的 4D 场景图。我们展示 DAAAM 的 4D 场景图与工具调用智能体良好接口，用于推理与推演。我们在 NaVQA 基准上的时空问答（SQA）复杂任务中全面评估 DAAAM，并展示其在 SG3D 基准上顺序任务视觉定位的泛化能力。我们进一步构建扩展的 OC-NaVQA 基准用于大规模长时间评估。DAAAM 在两个任务上均取得最优结果，相比最具竞争力的基线，OC-NaVQA 问题准确率提升 53.6%，位置误差降低 21.9%，时序误差降低 21.6%，SG3D 任务视觉定位准确率提升 27.8%。我们开源数据与代码。
英文原文摘要 1876 字符
Computer vision and robotics applications ranging from augmented reality to robot autonomy in large-scale environments require spatio-temporal memory frameworks that capture both geometric structure for accurate language-grounding and semantic detail. Existing methods face a tradeoff, where producing rich open-vocabulary descriptions comes at the expense of real-time performance when these descriptions have to be grounded in 3D. To address these challenges, we propose Describe Anything, Anywhere, at Any Moment (DAAAM), a novel spatio-temporal memory framework for large-scale and real-time 4D scene understanding. DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models, such as the Describe Anything Model (DAM), using batch processing to speed up inference by an order of magnitude for online deployment. It leverages such semantic understanding to build a hierarchical 4D scene graph (SG), which acts as an effective globally spatially and temporally consistent memory representation. DAAAM constructs 4D SGs with detailed, geometrically grounded descriptions while maintaining real-time performance. We show that DAAAM's 4D SG interfaces well with a tool-calling agent for inference and reasoning. We thoroughly evaluate DAAAM in the complex task of spatio-temporal question answering (SQA) on the NaVQA benchmark and show its generalization capabilities for sequential task grounding on the SG3D benchmark. We further curate an extended OC-NaVQA benchmark for large-scale and long-time evaluations. DAAAM achieves state-of-the-art results in both tasks, improving OC-NaVQA question accuracy by 53.6%, reducing position errors by 21.9% and temporal errors by 21.6%, and improving SG3D task grounding accuracy by 27.8% over the most competitive baselines. We release our data and code open-source.