异步时序建模与双智能体框架用于流式密集视频描述
Asynchronous Temporal Modeling with Two-Agent Framework for Streaming Dense Video Captioning
提出双智能体异步框架，用轻量SMM检测事件边界、LMM生成描述，避免阈值依赖，为流式视频理解提供可复用的事件触发与描述生成范式。
原文依据 We introduce Takusen, an asynchronous temporal modeling two-agent framework comprising a Small Multimodal Model (SMM) as an Oracle agent and an LMM as a Listener agent.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论流式密集视频描述需要实时处理连续视觉输入，同时精确判断何时描述以及描述什么。当前方法主要集中于设计复杂的外部记忆机制，未能利用多模态大模型（LMM）固有的长上下文能力。此外，现有采用基于阈值触发描述的方法面临严重的阈值门控差异（TGD）问题，这是一种由数据不平衡引起的训练-推理不匹配，模型主要预测静默标记，导致所需阈值在不同视频间差异巨大且有效范围极窄。我们提出Takusen，一个异步时序建模的双智能体框架，包含作为Oracle智能体的小型多模态模型（SMM）和作为Listener智能体的LMM。Oracle智能体以加速速率处理稀疏视频输入以检测事件边界，而Listener智能体处理密集输入，在Oracle信号提示时生成准确描述。该架构通过从根本上改变静默/生成决策方式，消除了阈值依赖，解决了TGD问题。为增强对边界预测不稳定的鲁棒性，我们将均匀分布的固定解码点与Oracle预测的边界相结合。在ActivityNet Captions和YouCook2数据集上的实验表明，Takusen以更简单高效的设计实现了最优性能，平衡了时序敏感性与描述准确性。
英文原文摘要 1563 字符
Streaming dense video captioning requires real-time processing of continuous visual input while determining precisely when and what to caption. Current approaches primarily focus on designing complex external memory mechanisms, failing to leverage Large Multimodal Models' (LMMs) inherent long-context capabilities. Moreover, existing methods employing threshold-based caption triggering face a severe Threshold-Gated Discrepancy (TGD) problem, a training-inference mismatch arising from data imbalance, where models predominantly predict silence tokens, requiring thresholds that vary drastically across videos with extremely narrow effective ranges. We introduce Takusen, an asynchronous temporal modeling two-agent framework comprising a Small Multimodal Model (SMM) as an Oracle agent and an LMM as a Listener agent. The Oracle agent processes sparse video inputs at an accelerated rate to detect event boundaries, while the Listener agent processes dense inputs to generate accurate captions when prompted by the Oracle's signals. This architecture eliminates threshold dependencies by fundamentally changing how silence/generation decisions are made, resolving the TGD problem. To enhance robustness against boundary prediction instabilities, we integrate uniformly distributed fixed decoding points with Oracle-predicted boundaries. Experiments on ActivityNet Captions and YouCook2 datasets demonstrate that Takusen achieves state-of-the-art performance with a simpler and more efficient design that balances temporal sensitivity with descriptive accuracy.