M5 – A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks (2024)

Abstract: Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and 41 languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.

## introduction since the release of chatgpt, natural language processing has experienced a significant surge in interest and research, with a particular focus on llms finetuned to follow human instructions. besides proprietary models like gpt-4 @xcite , claude @xcite , or gemini @xcite , there are also successful open-source variants such as llama @xcite , phi @xcite , or @xcite . while llms offor xflickrco and xm3600, we report bertscore f1. for the other datasets, the accuracy metric is reported. in this work, we focus on multimodal variants of llms, large multimodal models (lmms), such as gpt 4v @xcite , gemini pro v @xcite , or the popular open-source model, llava @xcite . llms are not textonly but are also capable of processing images in addition to text. most open-source lmms comprise three major components: an llm, a vision-encoder model, and a mapping network that projects image embeddings into the text embedding space. with this architecture, where an llm serves as the core, we argue that lmms inherently suffer from the same issue as llms: they generally perform much worse in non-english languages. however, existing benchmarks are either text-only @xcite or multimodal but monolingual @xcite , thus unable to prove multi-cultural: cultural diversity in images taken across the globe multi-task: five different vision-language tasks multiple models: 18 different lmms, 10 model families, s to xl sizes this hypothesis. in other words, current research lacks multimodal multilingual benchmarks to examine lmms' multilingual capabilities. in this work, we fill this gap by introducing the m5 benchmark, taking a significant step towards identifying and measuring the performance disparities of current lmms between various languages. figure 2 and figure 1 present a high-level summary of our benchmark. moreover, we introduce two new evaluation datasets, including a novel vision-language task. both datasets focus on african and asian cultures, which are underrepresented or even nonexistent in previous benchmarks. our exhaustive analyses additionally investigate the influence of different factors on the performance, such as the models' size or language fidelity. ## the m5 benchmark this section describes the setup of the m5 benchmark introduced by this work. details about the experimental setup, including prompts and hyperparameters, are reported in appendix a. ## novel m5 datasets in addition to the existing datasets introduced in the previous section, we crafted two novel multimodal and multilingual evaluation datasets. the principal motivation behind this is to fill the gap in existing vision-language datasets concerning the lack of underrepresented languages, tasks, and cultural diversity. moreover, we aim to enable further examination of lmms and their performance on non-english and non-western data with a particular focus on african and asian regions. more details, statistics, and examples are reported in appendix b. ## general results discussion this section discusses the models' performance on the datasets considered in our benchmark. ## aggregated result analyses 6.1 performance per language figure 5 shows the average performances aggregated by language foot_5 or language taxonomy classes @xcite . these taxonomy classes indicated how well a respective language is represented and considered within the research field of nlp based on papers published at cl conferences. high-resource languages such as english or german are in class 5, whereas low-resource languages such as berber are in class 0. for details about the languages and their taxonomy classes, please refer to as can be observed from figure 5a and figure 5b , the models perform best in english, followed by other european languages across all datasets. our newly presented m5-vlod dataset is an exception, where the average performance for all languages is around the random baseline, indicating the challenge it implies. as expected, the models consistently perform worse on lowresource languages than on high-resource languages on all datasets. this is also displayed in figure 5c , where it can be observed that the average performance decreases with the language taxonomy class. note that this is not precisely true for xflickrco and xvnli because the average on class-5 languages is lowered by outliers, as indicated by the large error bars. in contrast, the models performed comparably well in only one class 3 or 4 language, respectively. figure 5 : models' performances on all datasets aggregated by language or language taxonomy classes. ## conclusion we introduced m5, a diverse benchmark in which we evaluated 18 large multimodal models (lmms) with varying sizes across five visiolinguistic tasks in eight datasets comprising 41 unique languages. further, we presented two novel datasets -m5-vgr and m5-vlod -which focus on underrepresented languages and depict culturally diverse scenes. with m5-vlod, we introduce a new visio-linguistic outlier detection task in which only proprietary models achieve reasonable scores. our experiments revealed that model size does not always correlate with better performance, especially in non-english languages, underscoring the importance of diverse, multilingual training data and robust architectures. performance disparities were prominent between high-resource languages like english and low-resource languages across all datasets and models, highlighting ongoing challenges in achieving globally equitable multilingual ai. with m5, we aim to impel the development of more inclusive models suitable for diverse languages and cultures.