视觉语言模型 · CVPR2026 ·2026.06.03

BoostSLT:通过即插即用的扩散式语义增强器提升手语翻译

BoostSLT: Boosting Sign Language Translation via a Plug-and-Play Diffusion-Based Semantic Enhancer

Changzhou Han、Wanlun Ma、Xi Tang、Kun Hu、Sheng Wen、Yang Xiang

与你有关的部分

把扩散模型当作段落级语义重建器,对片段级翻译结果做拼接与精炼,这个「先分段再扩散融合」的即插即用思路可直接迁移到视频字幕、长视频描述等长序列视觉-语言生成任务。

原文依据 a Diffusion-based Semantic Reconstruction (DSR) module that stitches and refines fragment-level translations into globally fluent paragraphs
可迁移到 Sign Language TranslationVideo CaptioningLong-form Video Understanding

摘要译文

由模型翻译 · 术语统一 · 请对照原文核对关键结论

手语翻译(SLT)将连续手语视频转换为口语文本,但现有模型无论基于手语词(gloss)还是无手语词,在处理长输入或篇章级输入时都表现不佳。TwoStreamNetwork 与 CV-SLT 等近期架构已使短句准确率接近饱和,但在长句和多句段落上性能下降。在新闻、访谈或日常对话等真实场景中,手语者自然会产出带有复杂上下文依赖的较长手语序列。此外,精确识别手语词边界仍是一大障碍,而基于手语词的方法虽然往往更优,却带来高昂的标注成本。因此,社区需要一种既能缓解对手语词依赖、又能保持翻译质量的方案。我们提出 BoostSLT,一个上下文感知框架,可在无手语词监督下增强长手语序列的语义一致性。BoostSLT 无需显式手语词分割,而是引入能量感知时序分割(EAT-Seg)模块,动态地将视频划分为语义连贯的片段,随后由基于扩散的语义重建(DSR)模块将片段级翻译拼接并精炼为全局流畅的段落。该框架即插即用且与模型无关,可跨语言无缝集成到现有的基于手语词或无手语词的流程中。在 PHOENIX-2014T、CSL-Daily 和 Auslan-Daily 上的实验显示 BLEU 与 ROUGE-L 一致提升,证实扩散驱动的语义重建有效弥合了长格式 SLT 中局部准确性与全局连贯性之间的差距。代码见 github.com/K1sna/BoostSLT。

英文原文摘要 1700 字符

Sign Language Translation (SLT) converts continuous sign videos into spoken language text, yet current models, whether gloss-based or gloss-free, struggle with long or discourse-level inputs. Recent architectures such as TwoStreamNetwork and CV-SLT have nearly saturated short-sentence accuracy, but their performance degrades on long sentences and multi-sentence paragraphs. In real scenarios such as news, interviews or daily conversations, signers naturally produce extended signing sequences with complex contextual dependencies. Moreover, identifying precise gloss boundaries remains a key obstacle, while gloss-based methods, though often superior, incur heavy annotation costs. The community therefore needs a solution that mitigates gloss dependency while preserving translation quality. We present BoostSLT, a context-aware framework for enhancing semantic consistency over long sign sequences without gloss supervision. Instead of requiring explicit gloss segmentation, BoostSLT introduces an Energy-Aware Temporal Segmentation (EAT-Seg) module that dynamically partitions videos into semantically coherent fragments, followed by a Diffusion-based Semantic Reconstruction (DSR) module that stitches and refines fragment-level translations into globally fluent paragraphs. The framework is plug-and-play and model-agnostic, seamlessly integrating with existing gloss-based or gloss-free pipelines across languages. Experiments on PHOENIX-2014T, CSL-Daily, and Auslan-Daily show consistent BLEU and ROUGE-L gains, confirming that diffusion-driven semantic reconstruction effectively bridges local accuracy and global coherence in long-form SLT. Code is available at github.com/K1sna/BoostSLT.

标签

会议 CVPR2026
页码 28828-28837
Bibkey
来源 CVF Open Access