CaptionFormer：面向时空目标的统一分割、跟踪与描述
CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
用现成 VLM 为视频中时空定位的实体自动生成描述，把 LVIS/LV-VIS 扩成带描述的预训练数据，再用一个端到端模型联合做检测、分割、跟踪与描述；对做视频-语言任务的人，这是一条用合成字幕绕开人工标注瓶颈的可复用数据构造与预训练路线。
原文依据 we propose to generate captions about spatio-temporally localized entities leveraging a state-of-the-art VLM
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论稠密视频目标描述（DVOC）是一项联合检测、跟踪并描述视频中目标轨迹的任务，要求具备理解时空细节并用自然语言加以描述的能力。由于任务复杂且人工标注成本高昂，此前的方法采用数据有限的训练策略，可能导致性能欠佳。为规避这一问题，我们提出利用最优的 VLM 为时空定位的实体生成描述，并用合成描述扩展 LVIS 与 LV-VIS 数据集（LVISCap 与 LV-VISCap）。此外，我们提出一个端到端模型 CaptionFormer，能够联合检测、分割、跟踪并描述目标轨迹。在 LVISCap 与 LV-VISCap 上预训练后，CaptionFormer 在 VidSTG、VLN 与 BenSMOT 三个现有基准上取得了最优的 DVOC 结果。
英文原文摘要 952 字符
Dense Video Object Captioning (DVOC) is the task of jointly detecting, tracking, and captioning object trajectories in a video, requiring the ability to understand spatio-temporal details and describe them in natural language.Due to the complexity of the task and the high cost associated with manual annotation, previous approaches resort to training strategies with limited data, potentially leading to suboptimal performance.To circumvent this issue, we propose to generate captions about spatio-temporally localized entities leveraging a state-of-the-art VLM, and extend the LVIS and LV-VIS datasets with our synthetic captions (LVISCap and LV-VISCap). Moreover, we introduce an end-to-end model, CaptionFormer, capable of jointly detecting, segmenting, tracking and captioning object trajectories.With pretraining on LVISCap and LV-VISCap, CaptionFormer achieves state-of-the-art DVOC results on three existing benchmarks, VidSTG, VLN and BenSMOT.