带用户反馈的交互式情景记忆
Interactive Episodic Memory with User Feedback
把用户反馈作为条件注入到长视频时序定位中，并用轻量对齐模块（FALM）免去昂贵的序列优化，可直接迁移到视频检索、时刻定位等需要多轮交互修正的任务。
原文依据 a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在基于自然语言查询的情景记忆（EM-NLQ）中，用户可能提出一个问题（例如“我把杯子放在哪里了？”），该问题需要在一段以用户视角拍摄的长第一人称视频中进行搜索，以找到回答该问题的时刻。然而，查询可能含糊或不完整，从而导致错误回答。现有方法忽略了这一关键方面，以单样本方式处理 EM-NLQ，限制了其在真实场景中的适用性。在本工作中，我们填补这一空白，提出带问题与反馈的情景记忆任务（EM-QnF）。在该任务中，用户可以对模型的初始预测提供反馈或补充更多信息（例如“在这之前。我要找的是那个大蓝杯子，不是白色的那个”），从而帮助模型以交互方式逐步修正预测。为此，我们收集了用于基于反馈的交互的数据集，并提出一种轻量训练方案，避免昂贵的序列优化。我们还引入即插即用的反馈对齐模块（FALM），使现有 EM-NLQ 模型能够有效融合用户反馈。我们的方法在三个具有挑战性的基准上显著优于当前最优方法，并且在保持高效的同时优于或媲美商用大型视觉语言模型。使用人工生成的反馈进行的评测表明，该方法能够很好地泛化到真实场景。
英文原文摘要 1389 字符
In episodic memory with natural language queries (EM-NLQ), a user may ask a question (e.g., "Where did I place the mug?") that requires searching a long egocentric video, captured from the user's perspective, to find the moment that answers it. However, queries can be ambiguous or incomplete, leading to incorrect responses. Current methods ignore this key aspect and address EM-NLQ in a one-shot setup, limiting their applicability in real-world scenarios. In this work, we address this gap and introduce the Episodic Memory with Questions and Feedback task (EM-QnF). Here, the user can provide feedback on the model's initial prediction or add more information (e.g., "Before this. I'm looking for the big blue mug not the white one"), helping the model refine its predictions interactively. To this end, we collect datasets for feedback-based interaction and propose a lightweight training scheme that avoids expensive sequential optimization. We also introduce a plug-and-play Feedback ALignment Module (FALM) that enables existing EM-NLQ models to incorporate user feedback effectively. Our approach significantly improves over the state of the art on three challenging benchmarks and is better than or competitive with commercial large vision-language models while remaining efficient. Evaluation with human-generated feedback shows that it generalizes well to real-world scenarios.