面向高分辨率与解耦的基于参考的草图上色
Towards High-resolution and Disentangled Reference-based Sketch Colorization
以 SDXL 为骨干、草图作空间引导、RGB 参考图作颜色参考的参考式草图上色框架，其拆分交叉注意力加空间掩码分别处理前景/背景、以及用 tagger 网络提取属性级控制信号的思路，可直接迁移到其他需要区域解耦与细粒度参考控制的图像生成/编辑任务。
原文依据 A split cross-attention mechanism is coupled with spatial masks to separately colorize the foreground and background regions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论草图上色模型已被广泛研究，用于自动化和辅助动画帧与数字插画的创作。然而，当前方法在高分辨率合成与细节的精确可控性方面仍难以满足工业级应用的要求。为进一步提升合成质量与可控性，我们提出一种基于强大 SDXL 骨干网络的图像参考式草图上色方法，以草图作为空间引导、以 RGB 图像作为颜色参考。拆分交叉注意力机制与空间掩码相结合，分别对前景和背景区域上色，以避免空间纠缠。我们在海量动漫风格图像数据集上训练了一个 tagger 网络，用于从参考图像中提取属性级信息，并将其集成到流程中，为合成提供精确的控制信号。然而，SDXL 骨干网络分辨率的提升与注意力层数的增加，以及 tagger 网络提供的精确参考信息，会在上色过程中造成严重的纠缠。因此，我们结合前景编码器与背景编码器以实现解耦并获得更好的合成质量。此外，我们收集了一个高质量的、带标注且成对的草图上色数据集用于微调。所提方法是首个实现高分辨率、高质量且具备精确控制的草图上色方法，在定量与定性验证以及质量与可控性方面的用户研究中均明显优于现有方法。消融实验揭示了各组成部分的影响。代码与数据集将在论文被接收后公开。
英文原文摘要 1715 字符
Sketch colorization models have been widely studied to automate and assist in the creation of animation frames and digital illustrations. However, current methods are still not satisfactory for industrial standard applications in high-resolution synthesis and precise controllability of details. To further enhance the synthesis quality and controllability, we propose an image-referenced sketch colorization method based on the powerful SDXL backbone and leverage sketches as spatial guidance and RGB images as color references. A split cross-attention mechanism is coupled with spatial masks to separately colorize the foreground and background regions to avoid spatial entanglement. A tagger network trained on a massive anime-style image dataset is employed to extract attribution-level information from reference images and integrated into the pipeline to provide precise control signals for synthesis. However, the increased resolution and number of attention layers in the SDXL backbone and precise reference information from the tagger network cause severe entanglement during colorization. We consequently combine a foreground encoder and a background encoder for disentanglement and better synthesis quality. Furthermore, a high-quality annotated and paired sketch colorization dataset is collected for fine-tuning. The proposed method is the first to achieve high resolution high quality sketch colorization with precise control, and obvious outperforms existing methods in quantitative and qualitative validations, as well as user studies in both quality and controllability. Ablation study reveals the influence of each component. Code and dataset will be made publicly available upon paper acceptance.