面向视频常识描述的自批判蒸馏网络
Self-Critical Distillation Network for Video-based Commonsense Captioning
论文把「视频→内容描述→常识」的推理链改造成带奖励函数的自批判学习加级联解码器蒸馏，可复用于任何需要让语言输出更依赖视觉证据、并在多个相关类别间共享推理的视觉-语言生成任务。
原文依据 we introduce self-critical learning and design a reward function to allow the model to refine its output
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论基于视频的常识描述旨在为视频内容生成描述，同时针对其中蕴含的事件提供多条常识。现有方法依赖构建「视频→内容描述→常识」的推理链，这会生成缺乏视觉依据的常识，并忽视不同类别常识之间的关联。首先，现有推理链导致模型在生成常识时过度依赖内容描述，从而产生视觉相关性有限的泛化性输出。其次，该推理链采用多个相互独立的解码器来生成常识，无法利用不同类别常识之间的关联。为解决上述局限，我们提出一种新的自批判蒸馏网络（SCD-Net），通过增强视觉推理并建立类别间常识关联来优化该推理链。具体而言，一方面我们引入自批判学习并设计奖励函数，使模型能够精炼自身输出；该机制激励模型最大化利用视觉信息，从而提升模型的视觉理解能力。另一方面，我们提出联合推理蒸馏框架，促进不同常识类别之间的相互推断。在该框架中，我们结合级联解码器与知识蒸馏策略，在保持测试公平性的同时促进类别间的常识知识迁移。我们在大规模 Video-to-Commonsense 数据集上的实验表明，我们的方法优于最优方法。代码即将发布。
英文原文摘要 1740 字符
Video-based commonsense captioning aims to generate captions for the video content while providing multiple commonsense about the underlying events. Existing approaches rely on constructing a "video -> content caption -> commonsense" reasoning chain, which generates visually ungrounded commonsense and neglects inter-category commonsense correlations. Firstly, the existing reasoning chain induces the model's excessive reliance on content caption when generating commonsense, resulting in generic outputs with limited visual relevance. Secondly, the reasoning chain adopts multiple isolated decoders for commonsense generation, which fails to leverage the correlations between different categories of commonsense. To address these limitations, we introduce a novel self-critical distillation network (SCD-Net), which optimizes the reasoning chain by enhancing visual reasoning and establishing inter-category commonsense correlations. Specifically, on the one hand, we introduce self-critical learning and design a reward function to allow the model to refine its output. This mechanism incentivizes the model to maximize the utilization of visual information, thus improving the model's capacity for visual comprehension. On the other hand, we propose a joint reasoning distillation framework that fosters mutual inference among diverse commonsense categories. In this framework, we incorporate the cascaded decoder and knowledge distillation strategy to facilitate inter-category commonsense knowledge transfer while maintaining the fairness of the testing. Our experiments on the large-scale Video-to-Commonsense dataset demonstrate that our approach performs favorably against state-of-the-art methods. The code will be released soon.