MEGAVERSE : Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks (2024)

Abstract: There has been a surge in LLM evaluation research to understand LLM capabilities and limitations. However, much of this research has been confined to English, leaving LLM building and evaluation for non-English languages relatively unexplored. Several new LLMs have been introduced recently, necessitating their evaluation on non-English languages. This study aims to perform a thorough evaluation of the non-English capabilities of SoTA LLMs (GPT-3.5-Turbo, GPT-4, PaLM2, Gemini-Pro, Mistral, Llama2, and Gemma) by comparing them on the same set of multilingual datasets. Our benchmark comprises 22 datasets covering 83 languages, including low-resource African languages. We also include two multimodal datasets in the benchmark and compare the performance of LLaVA models, GPT-4-Vision and Gemini-Pro-Vision. Our experiments show that larger models such as GPT-4, Gemini-Pro and PaLM2 outperform smaller models on various tasks, notably on low-resource languages, with GPT-4 outperforming PaLM2 and Gemini-Pro on more datasets. We also perform a study on data contamination and find that several models are likely to be contaminated with multilingual evaluation benchmarks, necessitating approaches to detect and handle contamination while assessing the multilingual performance of LLMs.

## introduction large language models (llms) have surpassed the performance of previous generation of language models on several tasks and benchmarks, sometimes even approaching or exceeding human performance @xcite . however, the root cause of the observed capabilities in these models is not always apparent, whether stemming from augmented model capabilities or other factors like contamination in test datasets and the absence of datasets that genuinely measure the capabilities of these models @xcite . thus, evaluation of large language models has become an important field of study. most of the work on evaluating llms via benchmarking @xcite , qualitative tests for specific capabilities @xcite or human evaluation have focused solely on english. however, studies have shown that there is a large gap between the capabilities of llms in english and other languages @xcite . evaluation of llms in languages other than english is challenging due to a variety of factors, including the lack of benchmarks covering a large number of languages from diverse language families and the lack of multilingual benchmarks covering tasks such as reasoning, chat, and dialogue. therefore, it is crucial to prioritize multilingual evaluation to enhance the development of more effective multilingual models. neglecting this critical aspect may result in a significant population being left behind and may widen the digital divide @xcite . our prior work on evaluating multilingual capabilities of llms, mega @xcite , yielded the following observations: gpt-4 (ope-nai, 2023a) comes close to the performance of sota fine-tuned language models such as tulrv6 @xcite . gpt models perform worse on languages that are written in non-latin scripts, and on low-resource languages. other llms such as bloomz @xcite usually perform worse than gpt-4. however, several newer models are comparable to gpt-4 in performance on english, and it is essential to study their multilingual performance as well. moreover, there is a rising interest in large multimodal models (lmms), and the convergence of multimodal and multilingual llms remains an understudied area @xcite . our contributions are as follows: ## discussion in this work, we benchmark 22 datasets covering 83 languages across several models -gpt-3.5-turbo, gpt-4, palm2, gemini-pro, gemma, llama2, mistral as well as multimodal models. we find similar trends across most datasets we study -larger commercial models such as gpt-4 and gemini-pro outperform smaller models like gemma, llama and mistral models, particularly on low-resource languages. this suggests that multilingual performance is a challenge for smaller models, and directions such as language-specific models, language family-based models and fine-tuning should be explored for better multilingual performance. gpt-4, palm2 and gemini-pro excel on different datasets, with gpt-4 showing superior performance overall on multilingual datasets compared to both palm2 and gemini-pro. gpt-4-vision outperforms llava and gemini-pro-vision on the multimodal datasets we study. tokenizer fertility is correlated with language model performance @xcite . we plot the fertility analysis of all the tokenizers (figure : 14) for the models that we studied in this work. we noticed that on average, latin script languages such as spanish, english had lower fertility as compared to languages that are morphologically complex languages like telugu, malay and malayalam having high fertility amongst all the tokenizers. dataset contamination is a critical issue that affects english and non-english language benchmarking studies. our contamination analysis on open source and commercial models shows that almost all models are contaminated with datasets included in megaverse. new multilingual evaluation datasets are difficult to create due to resource and funding constraints, hence, care should be taken to make sure that they are not included in the training data of llms. to achieve this, we need to enhance our ability to identify instances of contamination, as well as implement measures to avoid future contamination. ## limitations our work is subject to the following limitations: