From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models (2024)

Abstract: Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models’ cultural inclusivity. Still, they have limited coverage of cultures and do not adequately assess cultural diversity across universal and culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures – underscoring the necessity for enhancing multicultural understanding in vision-language models.

## introduction vision-language models (vlms) have shown emergent capabilities through large-scale training that have made them gain popularity in recent years. vlms show promising results across various vision and language tasks, from image captioning to visual question answering and cross-modal retrieval and grounding. a key component contributing to their strong performance across the board is the scale of their pre-training datasets. however, these large-scale datasets tend to predominantly contain images from western cultures @xcite . the underrepresentation of certain cultures in the data translates into performance disparities across cultures. @xcite . several benchmarks and datasets have been proposed to test the cultural inclusivity of vlms. these include testing the models' performance on questions pertaining to images from certain cultures @xcite , on their ability to adapt images from one culture to another @xcite , or on stereotypical depiction of various cultures @xcite . nonetheless, existing benchmarks address a limited set of cultures (5-7), leaving a substantial representational gap. moreover, current benchmarks leave out a crucial aspect: assessing the cultural diversity in the representation of universal concepts. to address this gap, we present the globalrg benchmark. globalrg consists of two tasks which are based on popular vision and language tasks: image-text retrieval, which is used, for exam-ple, in search engines; and visual grounding, which can test models' vision-language alignment and is crucial for instruction-following applications such as in robotics. figure 1 presents the multicultural versions of these two tasks. the first task, retrieval across universals, covers images from 50 countries across 10 regions. it assesses the ability of vlms to retrieve culturallydiverse images pertaining to textual prompts of universal concepts such as "breakfast" and "wedding". in addition to the standard precision@k metric, which verifies that the retrieved images correctly depict the target concept, we also propose a new metric, diversity@k, that measures the cultural-diversity among the retrieved images, allowing us to identify models' bias towards specific countries or regions. in the second task, cultural visual grounding, we cover 15 countries across 8 regions and evaluate models' ability to ground culture-specific concepts (e.g., "molinillo", mexican whisk) within an image. extensive evaluation on 7 models for the retrieval task and 5 models for the grounding task reveals discrepancies across cultures, reassessing findings by prior work (e.g., @xcite . we further analyze whether vlms exhibit biases towards certain cultures. in the grounding task, the performance on north america and europe is substantially higher than on east asia and south east asia. this preference is inconsistent across universals in the retrieval task, e.g., a model may retrieve european images of funerals but african images of farming. a closer look reveals that when models retrieve seemingly diverse images, they often share western elements, such as eggs for breakfast, white dresses at weddings. globalrg highlights the lack of cultural awareness in vlms. by identifying and addressing these gaps, we can work towards developing models that perform equally well on inputs pertaining to concepts and images from diverse cultures. ## related work the geo-diversity problem. existing largescale vision and language datasets are imbalanced in their representation of different regions, overrepresenting the west @xcite . as a result, models trained on these datasets may exhibit discrepancies in performance when introduced with inputs concerning various demographic and geographic factors (e.g. @xcite . for instance, image generation models-when asked to generate images of universal concepts such as "house", tend to depict the concept as it appears in the us or india, cultures that are more prominently featured in the training data @xcite . to serve users from diverse cultures fairly, it is imperative to collect large-scale datasets from diverse data sources @xcite . two recent geo-diverse image datasets that are popular for training geo-diverse vlms, dollar street @xcite and geode (ramaswamy et al., 2024), focus on common household items, lacking coverage of more abstract and culture-specific concepts. finally, to make crosscultural data collection more feasible, researchers proposed to apply domain adaptation @xcite and active learning @xcite based on visual similarity. ## task 1: retrieval across universals image-text retrieval is a fundamental task for evaluating vlms, where the objective is to retrieve relevant images based on textual queries. existing image-text retrieval is typically evaluated using precision. beyond measuring the correctness of the retrieved images, this metric overlooks a significant aspect of retrieval systems: cultural diversity. we thus propose an additional evaluation metric to measure the cultural diversity of the retrieved images (sec 3.2). we evaluate an extensive number of vlms on the retrieval task (sec 3.3) and report the results in sec 3.4. ## task 2: cultural visual grounding visual grounding is essential for human-ai interactions, enabling users to reference regions using spatial cues and models to respond with precise visual answers, such as bounding boxes. existing grounding datasets such as refcoco and its variants @xcite , flickr entities (plummer et al., 2015), visual genome @xcite , and grit @xcite tend to focus on generic concepts and their images lack cultural contexts. to address this limitation, we propose the task of cultural visual grounding, to evaluate the ability of vlms to identify culture-specific concepts. we describe our dataset collection (sec 4.1), the task and evaluation metric (sec 4.2). we evaluate various models on our task (sec 4.3), and report the performance in sec 4.4. ## conclusion in this work, we introduced globalrg, a challenging benchmark designed to assess the multicultural understanding of vlms. globalrg includes two tasks: retrieving culturally diverse images representing universal concepts and visually grounding culture-specific concepts. our extensive experiments across a variety of vlms uncover specific cultural biases. for instance, in the retrieval task, models consistently show varying diversity scores, with a noticeable preference for western-style elements. likewise, vlms demonstrate higher accuracy in grounding images from western cultures, which underscores the imbalances in their training data. these results reveal an urgent need for more culturally diverse, large-scale datasets and for developing training objectives that explicitly consider a wide range of cultural contexts. addressing these challenges will pave the way for more inclusive and equitable models that better capture global cultural diversity and mitigate biases in downstream applications.