各司其职:面向稠密视频描述的角色专属查询与重叠抑制损失
Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning
把定位与描述拆成角色专属查询并用对比对齐保持语义一致,再以查询间时序重叠惩罚抑制冗余,这套「解耦查询+重叠抑制损失」可直接迁移到任何 query-based 视频定位/描述或时序动作检测框架中缓解多任务干扰。
原文依据 we propose utilizing role-specific queries that separate localization and captioning into independent components
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论稠密视频描述(DVC)是一项具有挑战性的多模态任务,需要在视频中时序定位多个事件并用自然语言对其进行描述。尽管基于查询的框架能够以端到端方式同时处理定位与描述,但其对共享查询的依赖往往导致两个任务之间出现显著的多任务干扰,并在定位中产生时序冗余。本文提出使用角色专属查询,将定位与描述分离为相互独立的组件,使各自仅学习自身的角色。随后采用对比对齐来强制对应输出之间的语义一致性,从而确保分离后的查询在行为上保持连贯。此外,我们设计了一种新颖的抑制机制,对查询之间的相互时序重叠施加惩罚以应对时序冗余,监督模型学习互不重叠的独立事件区域,从而实现更精确的定位。另外,我们引入一个轻量模块来捕获核心事件概念,通过概念级表示进一步增强描述的语义丰富度。我们在主要 DVC 基准 YouCook2 与 ActivityNet Captions 上的大量实验验证了本方法的有效性。
英文原文摘要 1324 字符
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.