基于注意力序列的个性化图像描述
Personalized Image Descriptions from Attention Sequences
把人类观看图像时的注意力序列作为个性化信号，通过辅助注意力预测任务学习主体嵌入，并用轻量适配器接入冻结的 VLM——这为视觉语言模型引入「感知行为」这一新条件维度提供了可复用范式。
原文依据 A lightweight adapter aligns these embeddings with a frozen vision-language model, enabling few-shot personalization without retraining.
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论人们对同一张图像的观看方式各不相同：他们关注不同的区域、物体和细节，关注顺序各异，并以不同的语言风格加以描述。这导致图像描述存在相当大的差异。然而，现有的个性化图像描述模型仅关注语言风格，尚无工作利用个体的观看模式。我们通过将个性化观看行为显式建模为描述生成的核心因素来填补这一空白。我们的方法 DEPER（DEscription-PERception persona encoder，描述—感知人格编码器）在辅助注意力预测任务的引导下，学习一个同时捕捉语言风格与观看行为的主体嵌入。一个轻量适配器将这些嵌入与冻结的视觉语言模型对齐，从而无需重新训练即可实现少样本个性化。在涵盖多样观看任务以及短描述与详细描述的四组数据集上，DEPER 取得平均 24% 的提升，表明对个性化注意力的建模能够产生更符合人类习惯且更高质量的描述。我们主张，理解人们如何看有助于预测他们如何说；对人类感知多样性的建模可以提升多模态系统的性能与人类对齐程度。代码见：https://github.com/cvlab-stonybrook/Personalized-Image-Description
英文原文摘要 1339 字符
People can view the same image differently: they focus on different regions, objects, and details in varying orders and describe them in distinct linguistic styles. This leads to substantial variability in image descriptions. However, existing models for personalized image description focus on linguistic style alone, with no prior work leveraging individual viewing patterns. We address this gap by explicitly modeling personalized viewing behavior as a core factor in description generation. Our method, DEPER (DEscription-PERception persona encoder), learns a subject embedding that captures both linguistic style and viewing behavior, guided by an auxiliary attention-prediction task. A lightweight adapter aligns these embeddings with a frozen vision-language model, enabling few-shot personalization without retraining. Across four datasets spanning diverse viewing tasks and both short and detailed descriptions, DEPER achieves a 24% average improvement, showing that modeling personalized attention produces more human-aligned and high-quality descriptions. We posit that understanding how people see helps predict what they say; modeling human diversity in perception can improve both performance and human alignment in multi-modal systems. Code is available at: https://github.com/cvlab-stonybrook/Personalized-Image-Description