StreamRAG：以检索增强提升实时视频理解
StreamRAG: Enhancing Real-Time Video Understanding with Retrieval Augmentation
把 RAG 从离线视频搬到流式视频，其 Stream Event Segmentation 的实时边界检测和 Token-Reusing Accelerator 的跨帧计算复用，可直接用于任何需要低延迟视频字幕/检索的视觉语言流水线。
原文依据 a Stream Event Segmentation (SES) module that performs real-time boundary detection to chunk the stream into meaningful units
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论检索增强生成（RAG）从离线视频分析向在线、流式场景的转变，带来了一系列关键且尚未被探索的挑战。这些挑战包括：需要对连续视频进行即时语义分割、低延迟处理与高质量知识抽取之间的内在矛盾，以及对查询特定时序推理的需求。我们提出 StreamRAG，一个旨在克服上述障碍的新框架。StreamRAG 建立在三个核心技术支柱之上：（1）流事件分割（SES）模块，执行实时边界检测，将视频流切分为有意义的单元；（2）Token 复用加速器，利用连续帧之间的计算重叠大幅降低字幕生成延迟；（3）动态检索门控，根据查询的时序敏感性与上下文相似度调节检索范围与策略。实证评估证实，StreamRAG 确立了新的最先进水平，在流式视频理解中以极低延迟实现更优准确率。
英文原文摘要 1108 字符
The transition of Retrieval-Augmented Generation (RAG) from offline video analysis to online, streaming scenarios presents a set of critical, unexplored challenges. These include the need for on-the-fly semantic segmentation of continuous video, the inherent tension between low-latency processing and high-quality knowledge extraction, and the demand for query-specific temporal reasoning. We propose StreamRAG, a novel framework designed to overcome these hurdles. StreamRAG is built upon three core technical pillars: (1) a Stream Event Segmentation (SES) module that performs real-time boundary detection to chunk the stream into meaningful units; (2) a Token-Reusing Accelerator that drastically cuts down captioning latency by leveraging computational overlap between consecutive frames; and (3) a Dynamic Retrieval Gate that modulates the retrieval scope and strategy based on the query's temporal sensitivity and contextual similarity. Empirical evaluation confirms that StreamRAG establishes a new state-of-the-art, delivering superior accuracy with minimal latency in streaming video comprehension.