Benchmarking Vision Language Models for Cultural Understanding (2024)

Abstract: Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM’s geo-diverse cultural understanding. We curate a diverse collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly weaker capabilities for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.

## introduction recent multimodal vision-language models (vlms) @xcite @xcite @xcite have shown impressive performance in tasks such as image-to-text generation, visual question answering, and image captioning, inter alia. these tasks predominantly focus on general scene understanding capabilities such as recognizing attributes, objects, and actions in scenes containing objects in their common context @xcite . however, given the advancing capabilities of vlms, we believe the time is now ripe to hold vlms to higher standards. indeed, to support increasingly global digital interactions, vlms must also be capable of understanding the cultural values @xcite such as beliefs, rituals, and traditions, for a variety of cultures in the world. in order to adequately assess whether the current state-of-the-art vlms -including proprietary models such as gpt-4o (openai, 2023) and gemini @xcite -encode cultural knowledge, we need systematic benchmarks. however, evaluating cultural understanding is a challenging task since culture is a multifaceted concept consisting of both tangible (e.g., clothing, and food) as well as intangible elements (e.g., ritual practices). current benchmarks in this domain, including marvl @xcite and gd-vcr @xcite , while offering foundational insights, have critical shortcomings. marvl primarily focuses on visual reasoning tasks (e.g., counting, spatial reasoning) on top of images sourced from various cultures, and lacks probing cultural common sense -the knowledge base shared by the members of a cultural group (see § 3). while gd-vcr does consider commonsense to a degree, it primarily considers movie scenes, which do not encompass the broader spectrum of everyday cultural contexts. in response to the above challenges, we propose culturalvqa, a novel benchmark specifically designed to assess cultural understanding of vlms. culturalvqa is based on visual question answering (vqa), requiring models to integrate both visual and textual information, which permits the formulation of diverse questions, thereby enabling the evaluation of a model's understanding of cultural nuances. the culturalvqa benchmark extends the language-only candle dataset @xcite , which provides a comprehensive collection of cultural commonsense knowledge assertions. we expand this dataset by automatically collecting images that depict the cultural concept described by the assertions. on top of these images, we collect questions and answers by employing annotators from different cultures who would be familiar with the different cultural concepts depicted in the images. see fig. 2 for some examples of questions and answers. our benchmark con-sists of 2,378 questions collected on top of 2,328 unique images with 1-5 answers per question (total 7,206 answers) from 11 countries. 1 we also present several analyses to better understand the nature of questions and answers in our benchmark. further, we systematically evaluate several state-of-the-art vlms on culturalvqa. our evaluation reveals a distinct performance gap between proprietary and open-source models, with open-source models significantly underperforming in comparison (e.g., there is a 29.78% gap between the highest-performing closed-source model and its open-source counterpart in the country for which the models perform the worst, ethiopia). additionally, we observe a significant disparity in model performance across countries. for instance, the highest-performing proprietary model, gpt-4o, achieves 67% and 72% accuracy on north american cultural concepts while only between 43% and 56% accuracy on concepts from africa. vlms also show varying degrees of proficiency across cultural facets, with closed-source vlms performing better on questions about rituals and traditions while scoring worse on those related to clothing, food, and drink. we develop culturalvqa as a comprehensive evaluation set for gauging vlms' progress in understanding diverse cultures and highlighting areas where vlms lack cultural understanding. we hope that our benchmark will contribute to accelerating the advancements of vlms in their cultural understanding, as illustrated in fig. 1 . cultural understanding is closely related to geodiverse understanding. for instance, the dollar street dataset (gaviria rojas et al., 2022) includes 38,479 images of everyday household items from homes around the world, while the gldv2 dataset (weyand et al., 2020) contains 5 million images and 200k distinct instance labels of natural and human-made landmarks, but both only test recognition capabilities as opposed to cultural understanding. burda-lassen et al. (2024) introduce mosaic-1.5k, a culture-specific captioning dataset that includes images from various regions. bhatia et al. (2024) propose globalrg, which aims to evaluate retrieval and grounding capabilities in vlms across 15 and 50 countries respectively. another related line of work focuses on multilingual understanding. for instance, bugliarello et al. (2022) culturalvqa uniquely emphasizes open-ended evaluation, includes culturally diverse images (i.e., images from multiple cultures), and its questions probe for cultural understanding by design. the combination of these characteristics sets cul-turalvqa apart from other datasets that either lack culturally diverse images (maxm), or use restricted evaluation formats such as multiple-choice (cvqa, gdvcr) or true/false (marvl). ## culturalvqa: dataset creation cultural taxonomy culture is a multifaceted concept that describes the way of life of a collective group of people, distinguishing them from other groups with different cultures @xcite . in this paper, we use the concept of a country as a proxy for a cultural group (adilazuarda et al., 2024) foot_1 . our work assumes common ground within a cultural group by probing culturally relevant concepts that are collectively understood, as well as shared cultural common sense employed in reasoning @xcite . for instance, lavash -a traditional persian bread (see fig. 2 ) -is an example of a culturally relevant concept, while the common prac-tice of waltzing at weddings exemplifies the cultural common sense among germans. building on these definitions, we introduce a benchmark that evaluates both the tangible aspects of culture through culturally relevant concepts, such as food, drink, and clothing, as well as the intangible facets via shared common sense embedded in rituals and traditions. 4 we frame this evaluation as a vqa task assessing models' cultural understanding. starting with a pool of countries, we collect images and use culturally knowledgeable annotators to frame questions. finally, we collect the ground truth answers. ## dataset analysis this section provides a detailed analysis of cul-turalvqas' composition and characteristics including analysis of images, questions, answers, and cultural concepts contained in it. images culturalvqa comprises 2,328 unique images. in fig. 2 we show representative samples. we choose images to ensure significant cultural representation across 11 different countries. the distribution of unique image count per country is detailed in fig. 3 (a) . questions we collect 2,378 questions in total. in fig. 3 (a), we present the number of unique questions per country. the questions have an average length of 10.98 words (see fig. 3 (b ) for country-wise breakdown). most frequent question types include 'what'(51.3%), 'which'(11.2%), 'in' (5.6%), and 'why' (3.4%) questions. for example, 'what' questions often relate to identifying cultural entities like saree or dirndl (traditional indian and german dresses, respectively) in the clothing cate-gory, or festivals like spring festival (celebrated in china) among rituals. 'where' questions inquire about locations significant to specific foods, such as the origins of quebec chicken. finally, we analyze whether the collected questions contain stereotypes and found that they are largely absent (see app. c). answers culturalvqa consists of 7,206 manually curated answers in total. 6 the average answer length is 1.73 words (see fig. 3 (b ) for country wise breakdown). we assess whether answers predominantly feature terms from local languages. to this end, we verified how many answers have corresponding english wikipedia titles; for 80% of the answers at least one of the answer words is contained in at least one wikipedia title. thus our benchmark is still suitable for english vlms. cultural concepts according to the pie chart in fig. 4 , food-related questions are most prevalent, accounting for 37.3% of the dataset, followed closely by traditions and rituals, which represent 26.1% and 18% respectively. thus, roughly 44% of the questions in our dataset probe for cultural understanding of the intangible aspects of culture (rituals and traditions). the word clouds generated from the collected answers in fig. 4 illustrate the diversity of expressions, such as hamam (turkey) and meskel (ethiopia) for rituals and traditions, and feijoada (brazil), fufu (nigeria), and vada (india) for food, indicating a geographically diverse culinary and cultural scope. while the clothing category is the least prevalent in the dataset, it shows the highest variety in terms of collected answers. the drink category is notably one of the smallest, both in terms of the size and number of unique answers. ## evaluating vlms on culturalvqa evaluation metric evaluating open-ended vqa is challenging. traditionally, string matching has been used but it is known to underestimate model performance. based on findings from @xcite , which demonstrate the effectiveness of reference-based llm evaluation for open-ended vqa tasks, we adopt lave, their proposed metric, as our evaluation metric with gpt-4 as the llm (see app. l for the llm prompt used). we validated the effectiveness of lave for our use case by computing correlation with human judgments. lave judgment agrees with human judgment 79% of the times for gpt-4, 73% of the times for gem-ini, and 76% of the times for intern-vl. vlms used for evaluation we benchmark several state-of-the-art vlms on the proposed cul-turalvqa dataset, ranging from closed-source models like gpt-4 (gpt-4o), claude (claude 3.5) and gemini pro (gemini-pro-vision 1.0) to a wide variety of open-source models, ranging from 7 to 25 billion parameter count: blip2 @xcite , instructblip @xcite , mblip @xcite pal-ligemma @xcite llava1.5 @xcite , llava_next @xcite , idefics2 @xcite , and intern-vl 1.5 @xcite . see app. d for a detailed discussion of the selected models. ## conclusions in this paper, we introduce culturalvqa, a novel vqa benchmark for assessing vlms on their cultural understanding. by curating a diverse collection of images from 11 countries across 5 continents and collecting 2,378 hand-crafted questions and 7,206 answers about cultural concepts presented in these images, written by annotators, we ensured a broad representation of cultural concepts pertinent to diverse cultural groups. benchmarking state-of-the-art models on cul-turalvqa reveals notable disparities in their performance across regions. models perform much better on north american cultures compared to african-islamic ones. further, we find a stark performance disparity between closed-and opensource models, with a 29.78% gap between the highest-performing closed-source and open-source models for the lowest-performing country. vlms also show varying proficiency across cultural facets, excelling in questions about clothing, rituals, and traditions but struggling with food and drink. our results underscore the current limitations of vlms in achieving uniform cultural comprehension and pinpoint specific areas that require improvement.