一个图像块描述所有：统一零样本描述框架
One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework
将图像描述从全局图像级转向以图像块为原子单元，利用DINO等骨干网络产生的密集视觉特征，无需区域级监督即可实现任意区域（包括非连续区域）的零样本描述，为开放世界区域理解提供了可复用的范式。
原文依据 we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论零样本描述器是近期提出的模型，利用共享空间的视觉-语言表示，无需依赖成对的图像-文本数据即可为图像生成描述。为了描述一张图像，它们通过对文本对齐的图像特征进行文本解码来推进，但局限于全局表示和整图描述。我们提出一个统一的零样本描述框架，从以图像为中心转向以图像块为中心的范式，使得无需区域级监督即可描述任意区域。我们不依赖全局图像表示，而是将单个图像块视为原子描述单元，并聚合它们来描述任意区域，从单个图像块到非连续区域乃至整幅图像。我们分析了使当前潜变量描述器在我们新提出的框架中工作的关键要素。实验表明，产生有意义的密集视觉特征的骨干网络（如DINO）是在多个基于区域的描述任务中实现最优性能的关键。与其他基线和最优竞争者相比，我们的模型在零样本密集描述和区域集描述上取得更好的性能。我们还引入了一个新的轨迹描述任务，进一步展示了逐块语义表示在灵活描述生成中的有效性。数据和代码见 https://paciosoft.com/Patch-ioner/ 。
英文原文摘要 1451 字符
Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense captioning and region-set captioning. We also introduce a new trace captioning task that further demonstrates the effectiveness of patch-wise semantic representations for flexible caption generation. Data and code are available at https://paciosoft.com/Patch-ioner/ .