MEGA : Multilingual Evaluation of Generative AI (2023)

Abstract: Generative AI models have shown impressive performance on many Natural Language Processing tasks such as language understanding, reasoning, and language generation. An important question being asked by the AI community today is about the capabilities and limits of these models, and it is clear that evaluating generative AI is very challenging. Most studies on generative LLMs have been restricted to English and it is unclear how capable these models are at understanding and generating text in other languages. We present the first comprehensive benchmarking of generative LLMs - MEGA, which evaluates models on standard NLP benchmarks, covering 16 NLP datasets across 70 typologically diverse languages. We compare the performance of generative LLMs including Chat-GPT and GPT-4 to State of the Art (SOTA) non-autoregressive models on these tasks to determine how well generative models perform compared to the previous generation of LLMs. We present a thorough analysis of the performance of models across languages and tasks and discuss challenges in improving the performance of generative LLMs on low-resource languages. We create a framework for evaluating generative LLMs in the multilingual setting and provide directions for future progress in the field.

## introduction large large models (llms) such as chatgpt and gpt-4 have created a lot of interest in the ai community and beyond, due to the step jump in their capabilities, such as maintaining context over conversations, fluency of generation, and reasoning. many users have reported having tested these systems on languages other than english, with varying results, and recent demos of these models @xcite have been shown in multiple (albeit highresource) languages. recently, the gpt-4 model @xcite was evaluated on the mmlu multiple choice questions benchmark by automatically translating it into 26 languages, and the results for some low-resource languages in the latin script were found to be quite promising. the multilingual capabilities of these models can be traced to their pre-training data, where even the predominantly english large-scale corpora contain hundreds of millions of non-engish tokens @xcite . for gpt-3 unlabeled pre-training data has been documented to contain 119 languages @xcite , where roughly 93% of the tokens are in english foot_0 . other llms like bloom @xcite and palm @xcite have a better multilingual representation with 60% and 18% non-english data respectively for pre-training. while these models have been trained on multiple languages with varying distributions in the pre-training data, it is not clear how well they perform relative to each other across diverse tasks and languages due to a lack of comprehensive analysis across all models with the same experimental setup. recently, there has been a lot of interest in evaluating the different capabilities of llms, with comprehensive studies like helm @xcite that evaluate these models on a wide variety of capabilities. however, such studies are largely performed on english language data and there is a lack of such large-scale evaluation of llms for their multilingual capabilities. given the current pace at which new language technologies are being developed that use llms, the importance of such an evaluation cannot be understated as the cases of inequalities in the performance of previousgeneration models across languages have been welldocumented @xcite . in our work, we present the first large-scale multilingual evaluation of generative ai models (mega), spanning 16 different datasets, 70 topologically diverse languages, and four llms i.e. gpt-3.5 models text-davinci-003 and gpt-3.5-turbo, gpt-4 (gpt-4-32k) and bloomz @xcite . we also compare these models with the models fine-tuned on these datasets like tulrv6 @xcite and @xcite , which are sota on different multilingual benchmarks. through our evaluation, we aim to answer three research questions. (1), how well do llms fare on multilingual benchmarks compared to fine-tuned sota models? (2), what languages do these models perform well in, and can we explain the trends in performance for these models across languages? our study highlights that there is a significant disparity between the performance of llms in english vs non-english languages, especially low-resource languages with non-latin scripts for which fine-tuned models perform significantly better. while gpt-4 bridges this gap to some extent, the discrepancy still exists. further, we find that for these languages it is often difficult to do better than simply machine translating the input in a target language to english and then sending it to the llm for prediction (translate-test). we also discuss how different prompt-design choices like prompt-tuning, use of explanations, and number of few-shot examples impact multilingual performance. finally, we perform some initial analysis to the test the possibility of test data contamination in llms that we evaluate and discuss its implications on our findings. our work provides a blueprint for strategies that can be used for building systems using generative ai for multilingual users. we also release our code foot_1 for the community to scale up the multilingual evaluation of generative models. ## mega in this section, we discuss different components of our benchmarking exercise to measure the multilingual capabilities of llms. we start by discussing different nlp tasks and datasets that we evaluate these models on, along with their linguistic diversity. we provide an overview of the models we evaluate, baselines for comparison, and describe our evaluation scheme and prompting strategies. ## results and analysis in this section, we analyze the results of our benchmarking exercise across tasks and languages. broadly, we cover the comparison between the effectivness of various prompting strategies §3.1 followed by the performance comparison of gpt-3.5 and gpt-4 models with appropriate baselines §3.2. we conclude with an examination of the factors that affects the performance of these models §3.3. ## challenges in multilingual evaluation in this section, we examine some of the challenges and consequent limitations of a large-scale multilingual evaluation like ours. ## conclusion in this work, we conduct an evaluation across different prompting strategies, models, tasks, and languages to investigate the multilingual capabilities of llms. we also investigate underlying properties like tokenizer quality and size of pretraining data to explain the trends in performance that we observe. our investigation shows the consistent performance gap between high-resource, latin script, and underresourced languages in addition to highlighting the efficacy, yet limited sufficiency of methods like translate-test prompting. through our evaluation, we present evidence of the need to prioritize automatic benchmarking and human evaluation across as many languages as possible. we hope that this work spurs research in meeting this goal.