基于循环动态子模型的高效视频目标分割与跟踪
Efficient Video Object Segmentation and Tracking with Recurrent Dynamic Submodel
对需要在视频分割/跟踪中部署 SAM2 这类大模型的人，RDS 给出了一条可复用的路线：用轻量路由器基于上一帧分割掩码和当前帧特征逐帧选择子模型块，把时序信息直接注入动态剪枝决策，从而把理论计算削减转化为真实加速；I-LoRA 只微调关键块的做法也可迁移到其他大模型的高效适配。
原文依据 a dynamic architecture that adaptively selects submodel blocks for each frame
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论以 SAM2 为代表的大型视觉基础模型在视频目标分割与跟踪（VOST）任务上取得了显著性能。然而，其有效性受到巨大计算开销的制约。模型剪枝是应对该问题的常用策略，但传统的静态、与输入无关的剪枝方法难以有效应对视频数据多样而复杂的特性。一个有前景的替代方案是动态网络，但它们往往难以将理论上的计算量削减转化为实际加速。此外，静态与动态方法通常只关注单帧的视觉特征，而忽略了帧间的时序相关性，从而限制了其处理复杂视频流时的性能。为解决上述挑战，我们提出循环动态子模型（RDS），一种为每一帧自适应选择子模型块的动态架构。具体而言，它包含一个轻量的预测感知路由器（PAR），同时利用上一帧的分割掩码与当前帧的视觉特征来做出路由决策，使子模型能够显式捕捉视频数据的时序特性。此外，为降低动态子模型适配的成本，我们引入重要性感知低秩适配（I-LoRA），仅在最关键的子模型块上调整参数。大量实验表明，我们的方法在性能与速度之间取得了最优权衡，且所需的可训练参数与训练数据极少。
英文原文摘要 1554 字符
Large vision foundation models, such as SAM2, have achieved remarkable performance in video object segmentation and tracking (VOST). However, their effectiveness is hindered by significant computational overhead. While model pruning is a widely used strategy to address this issue, traditional static and input-agnostic pruning approaches fall short in managing the diverse and complex nature of video data effectively. A promising alternative is dynamic networks, yet they often struggle to translate theoretical computational reductions into actual acceleration. Furthermore, both static and dynamic approaches typically focus on visual features of individual frames while neglecting the temporal correlations between them, limiting their performance in handling complex video streams.To address these challenges, we propose Recurrent Dynamic Submodel (RDS), a dynamic architecture that adaptively selects submodel blocks for each frame. Specifically, it has a lightweight Prediction-aware Router (PAR), which leverages both the segmentation mask from the previous frame and the visual features of the current frame to make routing decisions, enabling the submodel to explicitly capture the temporal nature of video data. Additionally, to reduce the cost of adapting the dynamic submodel, we introduce an Importance-aware LoRA (I-LoRA), tuning parameters only in the most critical blocks. Extensive experiments show that our approach achieves an optimal trade-off between performance and speed, requiring minimal trainable parameters and training data.