SMRABooth：面向定制视频生成的主体与运动表征对齐
SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation
利用自监督编码器和光流编码器提供对象级主体外观与运动表征，并通过稀疏低秩适配注入实现主体与运动的解耦控制，为视频定制生成中的主体保真与运动一致性提供可复用方案。
原文依据 we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject appearance and motion representations.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论定制视频生成旨在生成既忠实保留参考图像中主体外观、又保持参考视频中时序一致运动的视频。现有方法由于缺乏对主体和运动的对象级引导，难以同时保证主体外观相似性和运动模式一致性。为此，我们提出SMRABooth，利用自监督编码器和光流编码器提供对象级的主体外观和运动表征。这些表征在低秩适配微调过程中与模型对齐。我们的方法包含三个核心阶段：（1）我们通过自监督编码器利用主体表征来引导主体对齐，使模型能够捕捉主体的整体结构并增强高层语义一致性。（2）我们利用光流编码器的运动表征来捕捉结构连贯且独立于外观的对象级运动轨迹。（3）我们提出一种主体-运动关联解耦策略，在空间位置和时间上应用稀疏低秩适配注入，有效减少主体与运动低秩适配之间的干扰。大量实验表明，SMRABooth在主体和运动定制方面表现出色，能够保持一致的主体外观和运动模式，证明了其在可控文本到视频生成中的有效性。
英文原文摘要 1457 字符
Customized video generation aims to produce videos that faithfully preserve the subject's appearance from reference images while maintaining temporally consistent motion from reference videos. Existing methods struggle to ensure both subject appearance similarity and motion pattern consistency due to the lack of object-level guidance for subject and motion. To address this, we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject appearance and motion representations. These representations are aligned with the model during the LoRA fine-tuning process. Our approach is structured in three core stages: (1) We exploit subject representations via a self-supervised encoder to guide subject alignment, enabling the model to capture overall structure of subject and enhance high-level semantic consistency. (2) We utilize motion representations from an optical flow encoder to capture structurally coherent and object-level motion trajectories independent of appearance. (3) We propose a subject-motion association decoupling strategy that applies sparse LoRAs injection across both locations and timing, effectively reducing interference between subject and motion LoRAs. Extensive experiments show that SMRABooth excels in subject and motion customization, maintaining consistent subject appearance and motion patterns, proving its effectiveness in controllable text-to-video generation.