SegMo：面向高效 VLM 推理的内容感知稀疏化与局部凝聚分段并行协同设计
SegMo: Co-Designing Content-Aware Sparsity and Locally-Cohesive Segment Parallelism for Efficient VLM Inference
对做视频多模态推理的人，这篇给出了一个可直接复用的思路：利用 VideoLLM 注意力在场景内的局部凝聚性，先按查询相关性做场景级筛选、再按时序冗余做场景内剪枝，得到非均匀计算负载，并在场景边界切分做分段并行，用轻量全局上下文注入替代全局注意力的通信开销。
原文依据 Driven by the empirical discovery of Local Cohesion in VideoLLM attention
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论视频大语言模型（VideoLLM）面临一个根本性的性能瓶颈：视频输入固有的词元爆炸。由此产生的 O(N^2) 预填充开销使得常规 Transformer 推理在大规模下代价高昂。已有尝试陷入精度与延迟的两难：朴素稀疏化有丢失关键时空上下文的风险，而朴素并行化则引入大量通信与内存开销。为突破这一僵局，我们提出 SegMo，一个基于算法-系统协同设计的统一框架，联合优化「计算什么」（稀疏化）与「如何计算」（并行化）。受 VideoLLM 注意力中局部凝聚性这一经验发现的驱动，SegMo 集成两个关键组件：（1）内容感知稀疏化（CAS）：一种轻量级分层算法，先利用查询相关性进行场景级评估，再利用时序冗余进行场景内静态冗余剪枝，从而生成精确的非均匀计算负载，保证精度。（2）局部凝聚分段并行（LSP）：一种新范式，利用注意力局部性在场景边界处切分视频，并用轻量的全局上下文注入机制替代全局注意力带来的巨大通信与内存开销。SegMo 在 LVBench、LongVideoBench 和 Video-MME 上得到验证。我们的 CAS 模块将精度最高提升 12.00%。与 LSP 集成后，完整系统（CAS + LSP）实现最高 3.55 倍的预填充加速，同时仍保持最高 8.31% 的显著精度增益。
英文原文摘要 1615 字符
Video Large Language Models (VideoLLMs) face a fundamental performance bottleneck: the token explosion intrinsic to video inputs. The resulting O(N^2) prefill cost makes conventional Transformer inference prohibitively expensive at scale. Existing attempts fall into a hard accuracy-latency dilemma: naive sparsification risks losing essential temporal-spatial context, whereas naive parallelization introduces substantial communication and memory overhead. To overcome this impasse, we propose SegMo, a unified framework based on algorithm-system co-design that jointly optimizes what to compute (sparsification) and how to compute it (parallelism). Driven by the empirical discovery of Local Cohesion in VideoLLM attention, SegMo integrates two key components: (1) Content-Aware Sparsification (CAS): A lightweight, hierarchical algorithm that first employs Query Relevance for scene-level assessment, and then uses Temporal Redundancy for intra-scene static redundancy pruning, to generate a precise, non-uniform computation load, ensuring accuracy. (2) Locally-Cohesive Segment Parallelism (LSP): A novel paradigm that leverages attention locality to partition the video at scene boundaries, using a lightweight Global Context Injection mechanism to replace the massive communication and memory overheads of global attention. SegMo was validated across LVBench, LongVideoBench, and Video-MME. Our CAS module improved accuracy by up to 12.00%. When integrated with LSP, the full system (CAS + LSP) achieved a peak prefill acceleration of 3.55x, while still maintaining a significant accuracy gain of up to 8.31%.