CROPE : Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts (2025)

Abstract: As Vision and Language models (VLMs) become accessible across the globe, it is important that they demonstrate cultural knowledge. In his paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models.

## introduction recent vision and language models (vlms) @xcite @xcite have shown impressive performance across a variety of benchmarks @xcite . at the same time, frontier vlms @xcite have become widely accessible, making it crucial that these models can grasp the nuances of different cultures. models lacking cultural awareness can affect global cultural diversity, as they can potentially contribute to content reinforcing beliefs, habits, or perspectives that are primarily encountered within a particular community. in this work, we focus on culturespecific concepts and curate a dataset to answer the following research questions: how well do modern vlms perform in recognizing culture-specific concepts, and can they adapt to these concepts by leveraging multimodal contextual information? prior studies show that large language models (llms) @xcite , as well as clip-style vision encoders @xcite are biased towards western cultures @xcite . given the paradigm of developing vlms by combining together pre-trained vision encoders @xcite , and llms @xcite , recent work @xcite has highlighted that these biases transfer to multimodal models. towards more culturally inclusive vlms, we develop a culture-specific probing evaluation (crope) dataset. similar to previous vl probing datasets @xcite @xcite , we formulate the task as binary questions, which probe for the presence of a concept in the image, as shown in figure 1 . to stress-test a model's knowledge, we construct hard negative questions in which the concept in question and the concept in the image are visually or functionally similar. although language and culture interact @xcite , we limit our dataset to english, disentangling cultural from linguistic knowledge. following previous work @xcite , we designed crope to evaluate two types of knowledge: (1) parametric-knowledge encoded in the model weights, and (2) contextual-external knowledge (e.g., a wikipedia summary and corresponding image) given to the model to describe the culture-specific concept. we experiment with several state-of-the-art open-source and open-weights vlms with four different conditions where we vary the amount of contextual information (see figure 1 ). our findings illustrate that with no context at all, models exhibit a considerable performance drop relative to common concepts @xcite that are prevalent in most established training data for developing vlms. surprisingly, when provided with contextual knowledge, the performance of most models deteriorates even more. we analyze this behavior by inspecting the performance on an easy version of crope, where the target concept and the concept in the question belong to separate categories (e.g., food and beverage vs animals). in this case, most models show a performance improvement when provided with the textual information indicating that models struggle to differentiate between hard negative concepts. finally, we conduct a human evaluation that highlights which type of context (wikipedia summary, image, or both) is beneficial for humans when completing the same task. we find that the information provided by the text and the image modality is complementary for humans, which suggests that the observed model performance results from a lack of multimodal context understanding. ## crope dataset the objective of crope is to serve as a challenging evaluation set that probes the capabilities of modern vlms to recognize and adapt to culture-specific concepts. figure 2 outlines the steps of the dataset development: 1) concept selection from different cultures, 2) negative candidate selection via model-based sampling, and 3) human annotation that verifies the correctness of each example. ## experimental setup models we experiment with a variety of opensource and open-weights generative models up to 11b parameters (see ## conclusion in this work, we introduce crope, an evaluation benchmark for probing the parametric and contextual knowledge of vlms on culture-specific concepts. our results identify a significant performance disparity of state-of-the-art open vlms on concepts that appear commonly in vl datasets and crope. we also explore whether vlms can adapt to culture-specific concepts with multimodal contextual information and find that most models fail to utilize this context. we show that this is not necessarily the case when the models are required to compare semantically distant concepts, which indicates that current vlms struggle to reason about nuanced differences. conversely, our findings suggest that humans unfamiliar with the concepts in question benefit from multimodal information. discussion our investigation raises the question: are modern vlms truly capable of learning new concepts in-context? early work @xcite showed promise for vlms capable of fast-mapping, which refers to learning to bind new concepts to images with limited contextual information @xcite . the authors speculate that the binding capacity of a model can be improved with richer visual or textual support. our analysis shows that current vlms have not yet met this expectation, as they exhibit, at best, marginal improvements with relevant context. thus, processing arbitrary interleaved image-text formats remains a challenge. finally, we do not take the position that models should solely rely on non-parametric knowledge to perform well on tasks that require cultural understanding. given that benchmarks often become quickly saturated @xcite and the growing interest in more pluralistic representation in vlms, we anticipate future model iterations to 'solve' the task in a zero-shot manner. nevertheless, crope provides the opportunity to stress-test the current knowledge of culture-specific concepts embedded into the model weights, as well as the utilization of contextual knowledge. we find that there is plenty of room for improvement across both evaluation axes, and hope that crope can contribute towards the development of more capable and culturally inclusive vlms.