潜扩散模型反演需理解潜空间
Latent Diffusion Inversion Requires Understanding the Latent Space
对视觉研究者而言，本文揭示了潜扩散模型中自编码器几何（解码器拉回度量）对记忆/过拟合的非均匀影响，并提供了一种按维度贡献排序的通用分析工具，可用于诊断生成模型的隐私泄露风险或指导潜空间正则化设计。
原文依据 the diffusion model exhibits non-uniform memorization across latent codes, tending to overfit samples located in high-distortion regions of the decoder pullback metric
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论从生成模型中恢复训练数据（“模型反演”）已在数据域中作为记忆/过拟合现象在扩散模型上得到广泛研究。潜扩散模型（LDM）在编码器/解码器对产生的潜变量编码上运行，对先前的反演方法具有鲁棒性。在本工作中，我们描述了两个关键发现：（1）扩散模型在不同潜变量编码上表现出非均匀的记忆，倾向于过拟合位于解码器拉回度量高失真区域的样本；（2）即使在单个潜变量编码内，记忆贡献在不同表示维度上也不相等。我们提出的方法根据潜变量维度对解码器拉回度量的贡献进行排序，进而识别出对记忆有贡献的维度。对于基于分数的成员推理（模型反演的一个子任务），我们发现移除记忆较少的维度能在所有测试方法和数据集上提升性能，在包括CIFAR-10、CelebA、ImageNet-1K、Pokemon、MS-COCO和Flickr在内的多个数据集上，平均AUROC提升1-4%，TPR@1%FPR显著提升（1-32%）。我们的结果强调了自编码器几何对LDM记忆的被忽视的影响，并为分析基于扩散的生成模型中的隐私风险提供了新视角。
英文原文摘要 1361 字符
The recovery of training data from generative models ("model inversion") has been extensively studied for diffusion models in the data domain as a memorization/overfitting phenomenon. Latent diffusion models (LDMs), which operate on the latent codes from encoder/decoder pairs, have been robust to prior inversion methods. In this work we describe two key findings: (1) the diffusion model exhibits non-uniform memorization across latent codes, tending to overfit samples located in high-distortion regions of the decoder pullback metric; (2) even within a single latent code, memorization contributions are unequal across representation dimensions. Our proposed method to ranks latent dimensions by their contribution to the decoder pullback metric, which in turn identifies dimensions that contribute to memorization. For score-based membership inference, a sub-task of model inversion, we find that removing less-memorizing dimensions improves performance on all tested methods and datasets, with average AUROC gains of 1-4% and substantial increases in TPR@1%FPR (1-32%) across diverse datasets including CIFAR-10, CelebA, ImageNet-1K, Pokemon, MS-COCO, and Flickr. Our results highlight the overlooked influence of the auto-encoder geometry on LDM memorization and provide a new perspective for analyzing privacy risks in diffusion-based generative models.