让免训练扩散分割器随生成能力扩展
Making Training-Free Diffusion Segmentors Scale with the Generative Power
针对免训练扩散分割器中交叉注意力图与全局语义不一致、文本token分数不平衡的问题，提出自动聚合与逐像素重缩放两种技术，使分割性能随生成模型能力提升而提升，为利用更强扩散模型做判别任务提供了可复用方案。
原文依据 we propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论作为强大的生成模型，文本到图像扩散模型近期已被探索用于判别任务。一系列研究致力于将预训练的扩散模型适配到语义分割，且无需任何额外训练，从而形成免训练扩散分割器。这些方法通常依赖模型注意力层中的交叉注意力图，并假设这些图能捕获图像像素与文本标记之间的语义关系。理想情况下，此类方法应受益于更强大的扩散模型，即更强的生成能力应带来更好的分割效果。然而，我们观察到现有方法往往无法相应地扩展。为理解这一问题，我们识别出两个潜在差距：（i）交叉注意力在多个头和层上计算，但这些单独的注意力图与统一的全局表示之间存在差异。（ii）即使存在全局图，由于不同文本标记之间的分数不平衡，它也不能直接转化为用于分割的准确语义关联。为弥合这些差距，我们提出两种技术：自动聚合和逐像素重缩放，二者共同使免训练分割能更好地利用生成能力。我们在标准语义分割基准上评估了我们的方法，并进一步将其集成到一种生成技术中，证明了性能提升和广泛适用性。代码见 https://github.com/Darkbblue/goca。
英文原文摘要 1518 字符
As powerful generative models, text-to-image diffusion models have recently been explored for discriminative tasks. A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training, leading to training-free diffusion segmentors. These methods typically rely on cross-attention maps from the model's attention layers, which are assumed to capture semantic relationships between image pixels and text tokens. Ideally, such approaches should benefit from more powerful diffusion models, i.e., stronger generative capability should lead to better segmentation. However, we observe that existing methods often fail to scale accordingly. To understand this issue, we identify two underlying gaps: (i) Cross-attention is computed across multiple heads and layers, but there exists a discrepancy between these individual attention maps and a unified global representation. (ii) Even when a global map is available, it does not directly translate to accurate semantic correlation for segmentation, due to score imbalances among different text tokens. To bridge these gaps, we propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability. We evaluate our approach on standard semantic segmentation benchmarks and further integrate it into a generative technique, demonstrating both improved performance and broad applicability. Codes are at https://github.com/Darkbblue/goca.