PHAC:可提示的人体非模态补全
PHAC: Promptable Human Amodal Completion
将点级提示(关节、边界框)经 ControlNet 注入预训练扩散模型,并仅微调交叉注意力块以保持生成先验,这一「轻量适配 + 多类型提示编码」范式可直接迁移到其他需要用户精细控制的条件生成任务。
原文依据 we fine-tune only the cross-attention blocks to obtain strong prompt alignment without degrading the underlying generative prior
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论条件图像生成方法正日益应用于以人为中心的应用,然而现有的人体非模态补全(HAC)模型对补全内容提供的用户控制十分有限。给定一张被遮挡的人体图像,它们会幻构出不可见区域并保持可见区域不变,但无法可靠地纳入用户指定的约束,例如期望的姿态或空间范围。因此,用户往往只能反复采样模型以获得满意的输出。姿态引导的人体图像合成(PGPIS)方法允许显式的姿态条件,但常常无法保持实例特有的可见外观,且即便建立在强扩散模型先验之上,也倾向于偏向训练分布。为解决这些局限,我们提出可提示的人体非模态补全(PHAC),这是一项新任务:在满足可见外观约束与多种用户提示的同时补全被遮挡的人体图像。用户提供简单的基于点的提示,例如为目标姿态补充的关节,或为期望区域提供的边界框;这些提示由针对每种提示类型专门化的 ControlNet 模块编码。这些模块将提示信号注入预训练扩散模型,并且我们仅微调交叉注意力块,从而在不损害底层生成先验的情况下获得强提示对齐。为进一步保持可见内容,我们提出一个基于图像修复的细化模块,它从带有轻微噪声的粗补全结果出发,忠实保持可见区域,并确保遮挡边界处的无缝融合。在标准 HAC 与 PGPIS 基准上的大量实验表明,与现有的非模态补全和姿态引导合成方法相比,我们的方法能生成物理上更合理、质量更高的补全结果,且提示对齐显著改善。
英文原文摘要 1893 字符
Conditional image generation methods are increasingly used in human-centric applications, yet existing human amodal completion (HAC) models offer users limited control over the completed content. Given an occluded person image, they hallucinate invisible regions while preserving visible ones, but cannot reliably incorporate user-specified constraints such as a desired pose or spatial extent. As a result, users often resort to repeatedly sampling the model to obtain a satisfactory output. Pose-guided person image synthesis (PGPIS) methods allow explicit pose conditioning, but frequently fail to preserve the instance-specific visible appearance and tend to be biased toward the training distribution, even when built on strong diffusion model priors. To address these limitations, we introduce promptable human amodal completion (PHAC), a new task that completes occluded human images while satisfying both visible appearance constraints and multiple user prompts. Users provide simple point-based prompts, such as additional joints for the target pose or bounding boxes for desired regions; these prompts are encoded using ControlNet modules specialized for each prompt type. These modules inject the prompt signals into a pre-trained diffusion model, and we fine-tune only the cross-attention blocks to obtain strong prompt alignment without degrading the underlying generative prior. To further preserve visible content, we propose an inpainting-based refinement module that starts from a slightly noised coarse completion, faithfully preserves the visible regions, and ensures seamless blending at occlusion boundaries. Extensive experiments on standard HAC and PGPIS benchmarks show that our approach produces more physically plausible, higher-quality completions with significantly improved prompt alignment compared to existing amodal completion and pose-guided synthesis methods.