P-Flow:以提示词驱动视觉特效生成
P-Flow: Prompting Visual Effects Generation
把视觉语言模型当作奖励/评判器,在测试时迭代优化文本提示词,从而免训练地为视频生成注入高层动态特效;这套「VLM 反馈 + 提示词迭代」的框架可直接迁移到风格化、物理现象模拟等难以用轨迹显式控制的可控生成任务。
原文依据 P-Flow performs test-time prompt optimization, refining prompts based on the discrepancy between the visual effects of the reference video and the generated output.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论近期视频生成模型的进展显著提升了其遵循文本提示词的能力。然而,动态视觉特效的定制化仍未被充分探索——这类特效被定义为随时间演化、由外观驱动的视觉现象,例如物体压碎或爆炸。此前的运动定制或控制工作主要关注主体或相机的低层运动,这些运动可以用运动轨迹等显式控制信号加以引导。相比之下,动态视觉特效涉及更高层的语义,更自然地适合通过文本提示词进行控制。然而,人类要编写出能准确指定这些特效的单一提示词既困难又耗时,因为这需要复杂的时序推理以及随时间反复迭代修正。为应对这一挑战,我们提出 P-Flow,一种新颖的免训练框架,可在不修改底层模型的情况下定制视频生成中的动态视觉特效。借助视觉语言模型的语义与时序推理能力,P-Flow 执行测试时提示词优化,依据参考视频与生成输出之间视觉特效的差异来精炼提示词。通过迭代精炼,提示词逐步演化,从而在新场景中更好地诱导出所需的动态特效。实验表明,P-Flow 实现了高保真且多样的视觉特效定制,并在文本到视频与图像到视频生成任务上优于其他模型。代码见 https://github.com/showlab/P-Flow。
英文原文摘要 1589 字符
Recent advancements in video generation models have significantly improved their ability to follow text prompts. However, the customization of dynamic visual effects, defined as temporally evolving and appearance-driven visual phenomena like object crushing or explosion, remains underexplored. Prior works on motion customization or control mainly focus on low-level motions of the subject or camera, which can be guided using explicit control signals such as motion trajectories. In contrast, dynamic visual effects involve higher-level semantics that are more naturally suited for control via text prompts. However, it is hard and time-consuming for humans to craft a single prompt that accurately specifies these effects, as they require complex temporal reasoning and iterative refinement over time. To address this challenge, we propose P-Flow, a novel training-free framework for customizing dynamic visual effects in video generation without modifying the underlying model. By leveraging the semantic and temporal reasoning capabilities of vision-language models, P-Flow performs test-time prompt optimization, refining prompts based on the discrepancy between the visual effects of the reference video and the generated output. Through iterative refinement, the prompts evolve to better induce the desired dynamic effect in novel scenes. Experiments demonstrate that P-Flow achieves high-fidelity and diverse visual effect customization and outperforms other models on both text-to-video and image-to-video generation tasks. Code is available at https://github.com/showlab/P-Flow.