一帧世界：将文化混合理解为视觉语言模型的新挑战
World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models
为 LVLM 研究提供了一个可复用的文化混合 VQA 基准与四种子任务设定，并验证了用扩散生成的多文化混合数据做监督微调可显著降低模型对背景的依赖、提升跨上下文预测一致性，可作为评估和缓解 LVLM 背景偏置的参考范式。
原文依据 we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论在全球化世界中，来自不同起源的文化元素经常同时出现在同一视觉场景中。我们将这类情形称为文化混合场景，但大型视觉语言模型（LVLM）如何感知它们仍缺乏充分探索。我们将文化混合作为 LVLM 面临的一项关键挑战加以研究，并考察当前模型在多个地区的文化物品同时出现时的行为。为系统分析这些行为，我们构建了 CultureMix，一个食物视觉问答（VQA）基准，包含 23k 张由扩散模型生成、经人工核验的文化混合图像，覆盖四个子任务：（1）仅食物，（2）食物+食物，（3）食物+背景，（4）食物+食物+背景。在评估 10 个 LVLM 后，我们发现它们在混合场景中一致地无法保留各个文化身份。模型表现出强烈的背景依赖：当在仅食物基线上加入文化背景时，准确率下降 14%，且对相同食物在不同上下文中给出不一致的预测。为应对这些局限，我们探索了三种鲁棒性策略。我们发现，使用多样化的文化混合数据集进行监督微调可显著提升模型一致性并降低背景敏感性。我们呼吁加强对文化混合场景的关注，这是迈向开发能在文化多样的真实环境中可靠运行的 LVLM 的关键一步。
英文原文摘要 1417 字符
In a globalized world, cultural elements from diverse origins frequently appear together within a single visual scene. We refer to these as culture mixing scenarios, yet how Large Vision-Language Models (LVLMs) perceive them remains underexplored. We investigate culture mixing as a critical challenge for LVLMs and examine how current models behave when cultural items from multiple regions appear together. To systematically analyze these behaviors, we construct CultureMix, a food Visual Question Answering (VQA) benchmark with 23k diffusion-generated, human-verified culture mixing images across four subtasks: (1) food-only, (2) food+food, (3) food+background, and (4) food+food+background. Evaluating 10 LVLMs, we find consistent failures to preserve individual cultural identities in mixed settings. Models show strong background reliance, with accuracy dropping 14% when cultural backgrounds are added to food-only baselines, and they produce inconsistent predictions for identical foods across different contexts. To address these limitations, we explore three robustness strategies. We find supervised fine-tuning using a diverse culture mixing dataset substantially improve model consistency and reduce background sensitivity. We call for increased attention to culture mixing scenarios as a critical step toward developing LVLMs capable of operating reliably in culturally diverse real-world environments.