面向组合视频检索的组合式变换推理
Compositional Transformation Reasoning for Composed Video Retrieval
把组合式图文查询拆成实体、动作、场景三个推理维度,并用多模态大模型做成对候选推理,这套零样本流程可直接迁移到视频-文本检索与视频时序变化理解任务中,避免依赖噪声三元组训练。
原文依据 We propose Multi-objective Reasoning (MoRe), a zero-shot framework based on MLLMs for multi-objective candidate selection and fine-grained transformation reasoning.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论组合视频检索旨在给定一段参考视频和一段描述期望变化的文本修改,检索出目标视频。其核心挑战在于建模组合式多模态变换,即实体、动作和场景如何在视频与语言两种模态之间、针对细粒度文本编辑发生演变。现有方法通过在大规模视频-文本-视频三元组上训练,或通过生成稠密文本描述来捕捉细微视觉差异,以此应对该问题。然而,这些监督式方法往往依赖带噪的网络规模数据和数据集特定的对应关系,导致过拟合,并在多样或细粒度场景中泛化能力有限,同时也无法有效建模组合式与时序变换。我们提出多目标推理(MoRe),一个基于多模态大语言模型的零样本框架,用于多目标候选选择与细粒度变换推理。该方法将组合式变换分解为三个互补的推理维度,即实体、动作和场景,并执行成对候选推理,以显式捕捉随时间的语义演变。此外,我们引入一个面向召回的多目标候选选择模块,在变换推理之前联合平衡视觉、文本和多模态相似度,以识别高质量检索目标。在 EgoCVR 和 WebVid-CoVR 上的实验表明,该方法在零样本设定下优于最优方法,R@1 分别提升 +5.8 和 +10.8。
英文原文摘要 1622 字符
Composed Video Retrieval aims to retrieve a target video given a reference video and a textual modification describing the desired change. The core challenge lies in modeling compositional multimodal transformations, i.e., how entities, actions, and scenes evolve across video and language modalities in response to fine-grained textual edits. Existing methods address this issue by training on large-scale video-text-video triplets or by generating dense textual descriptions to capture subtle visual differences. However, these supervised approaches often rely on noisy web-scale data and dataset-specific correspondences, leading to overfitting and limited generalization in diverse or fine-grained scenarios, while also failing to effectively model compositional and temporal transformations. We propose Multi-objective Reasoning (MoRe), a zero-shot framework based on MLLMs for multi-objective candidate selection and fine-grained transformation reasoning. Our method decomposes the compositional transformation into three complementary reasoning dimensions, i.e., entity, action, and scene, and performs pairwise candidate reasoning to explicitly capture semantic evolution over time. Furthermore, we introduce a recall-oriented multi-objective candidate selection module that identifies high-quality retrieval targets by jointly balancing visual, textual, and multimodal similarities before transformation reasoning. Experiments on EgoCVR and WebVid-CoVR demonstrate the effectiveness of our method over state-of-the-art approaches under the zero-shot setting, with R@1 improvements of +5.8 and +10.8, respectively.