GIMMICK : Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking (2025)

Abstract: Large Vision-Language Models (LVLMs) have recently gained attention due to their distinctive performance and broad applicability. While it has been previously shown that their efficacy in usage scenarios involving non-Western contexts falls short, existing studies are limited in scope, covering just a narrow range of cultures, focusing exclusively on a small number of cultural aspects, or evaluating a limited selection of models on a single task only. Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive multimodal benchmark designed to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions. GIMMICK comprises six tasks built upon three new datasets that span 728 unique cultural events or facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary and 26 open-weight models of all sizes. We systematically examine (1) regional cultural biases, (2) the influence of model size, (3) input modalities, and (4) external cues. Our analyses reveal strong biases toward Western cultures across models and tasks and highlight strong correlations between model size and performance, as well as the effectiveness of multimodal input and external geographic cues. We further find that models have more knowledge of tangible than intangible aspects (e.g., food vs. rituals) and that they excel in recognizing broad cultural origins but struggle with a more nuanced understanding.

## introduction recently, proprietary as well as open-weight large vision-language models (lvlms) @xcite @xcite have attracted marked attention due to their broad applicability across various domains. several large-scale holistic benchmarks (duan et al., 2024; yue et al., 2024; fu et al., 1 http://github.com/floschne/gimmick 2023) demonstrate lvlms' remarkable performances in a wide range of multimodal tasks. however, most benchmarks concentrate on westerncentric english tasks, and multilingual benchmarks @xcite reveal a significant deterioration in performance on non-english tasks. while multilingualism is essential for globally equitable ai, multiculturalism @xcite is equally crucial for models to reflect and respect the diverse cultural backgrounds of users worldwide. in this context, it has been shown that current llms @xcite and lvlms suffer in tasks involving knowledge from non-western cultures. however, the scope of existing multimodal cultural studies is still severely limited: existing research often focuses only on specific concepts like food or dance @xcite , covers a limited number of cultures @xcite , evaluates only a small selection of lvlm models @xcite , or tests only a single combination of input modalities. to address these gaps, we introduce gimmick, a comprehensive evaluation framework assessing 31 state-of-the-art models, ranging from proprietary lvlms to open-weight llms and lvlms of all sizes-from 500m to 78b parameters-across multiple model families. it comprises six tasks built on three novel datasets that contain 728 unique cultural events or facets (cefs) from 144 countries in six global macro-regions and target both high-level and nuanced cultural knowledge through multimodal and unimodal tasks. our vqa tasks span a total of 57 cultural aspects (see §b.2) ultimately, gimmick enables us to answer four research questions: (rq1) are there regional biases in llms' and lvlms' cultural knowledge, and if so, which? for the most complex tasks, we observe consis-figure 1 : an overview of the gimmick benchmark and its tasks. tent cultural regional biases (up to 14.72pp difference between instances targeting western europe & north america vs. subsaharian africa; §5.1)even for the largest models. for less complex tasks, these differences flatten out. ## results and analyses in this section, we present a series of in-depth analyses based on the outcomes of our benchmark. we show aggregated results: open-weight models are grouped and averaged by parameter size, and proprietary models are averaged together (see ## conclusion we introduce gimmick, a comprehensive benchmark to assess various aspects of cultural knowledge of current lvlms and llms and introduce six tasks built upon three novel datasets, which span 728 unique cultural events or facets (cefs) from 144 countries grouped into six global macro-regions. through extensive analyses, we study general cultural biases and the influence of model size, input modalities, and external cues. our results consistently reveal a prominent bias toward western cultures across all models. interestingly, when only coarse cultural knowledge is required-such as regional originsmodels performed remarkably better. across all tasks, significant correlations between a model's performance and its size are evident, with a substantial gap between proprietary and open-weight models. our analyses show that while models grasp broad cultural categories, they struggle with nuanced understanding. this suggests that gimmick poses a challenging benchmark and highlights the need for further advances in modeling broad cultural awareness.