锚定与重标定注意力实现语义连贯的中间帧生成
Anchoring and Rescaling Attention for Semantically Coherent Inbetweening
把关键帧与文本作为语义/时序引导注入自注意力(Keyframe-anchored Attention Bias),并用重标定的时序 RoPE 强化帧间一致性,这套无需训练的注意力调制手段可直接迁移到视频插帧、长视频生成与关键帧约束的扩散采样中。
原文依据 we give semantic and temporal guidance from the keyframes and text onto each intermediate frame through Keyframe-anchored Attention Bias
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论生成式中间帧合成(GI)旨在首末关键帧之间合成逼真的中间帧,而不仅仅是插值。随着序列变得更稀疏、运动幅度更大,以往的 GI 模型难以处理帧间不一致、节奏不稳定与语义错位的问题。由于 GI 涉及固定的端点以及众多可能的路径,该任务需要从关键帧与文本中获取额外引导,以指定预期的路径。为此,我们通过关键帧锚定注意力偏置,将来自关键帧与文本的语义引导和时序引导施加到每个中间帧上。我们还利用重标定时序 RoPE 更好地保证帧间一致性,使自注意力能够更忠实地关注关键帧。TGI-Bench 是首个专为文本条件 GI 评测设计的基准,支持面向特定挑战的评测,以分析 GI 模型。无需额外训练,我们的方法在多种挑战下的短序列与长序列上均取得了最优的帧间一致性、语义保真度与节奏稳定性。
英文原文摘要 1074 字符
Generative inbetweening (GI) seeks to synthesize realistic intermediate frames between the first and last keyframes beyond mere interpolation. As sequences become sparser and motions larger, previous GI models struggle with inconsistent frames with unstable pacing and semantic misalignment. Since GI involves fixed endpoints and numerous plausible paths, this task requires additional guidance gained from the keyframes and text to specify the intended path. Thus, we give semantic and temporal guidance from the keyframes and text onto each intermediate frame through Keyframe-anchored Attention Bias. We also better enforce frame consistency with Rescaled Temporal RoPE, which allows self-attention to attend to keyframes more faithfully. TGI-Bench, the first benchmark specifically designed for text-conditioned GI evaluation, enables challenge-targeted evaluation to analyze GI models. Without additional training, our method achieves state-of-the-art frame consistency, semantic fidelity, and pace stability for both short and long sequences across diverse challenges.