Evaluating the Elementary Multilingual Capabilities of Large Language Models with M ulti Q (2024)

Abstract: Large language models (LLMs) need to serve everyone, including a global majority of non-English speakers. However, most LLMs today, and open LLMs in particular, are often intended for use in just English (e.g. Llama2, Mistral) or a small handful of high-resource languages (e.g. Mixtral, Qwen). Recent research shows that, despite limits in their intended use, people prompt LLMs in many different languages.Therefore, in this paper, we investigate the basic multilingual capabilities of state-of-the-art open LLMs beyond their intended use.For this purpose, we introduce MultiQ, a new silver standard benchmark for basic open-ended question answering with 27.4k test questions across a typologically diverse set of 137 languages. With MultiQ, we evaluate language fidelity, i.e. whether models respond in the prompted language, and question answering accuracy. All LLMs we test respond faithfully and/or accurately for at least some languages beyond their intended use. Most models are more accurate when they respond faithfully. However, differences across models are large, and there is a long tail of languages where models are neither accurate nor faithful. We explore differences in tokenization as a potential explanation for our findings, identifying possible correlations that warrant further investigation.

## introduction languages other than english remain underrepresented and underserved by state-of-the-art language technologies, posing a barrier to equal and inclusive ai @xcite . while proprietary large language models (llms) like @xcite may answer questions and follow instructions in many different languages, even the best and most popular open llms are much more restricted in their language coverage: llama2-chat @xcite , for example, ## experiments and results using multiq, we can now answer our two main research questions regarding the multilingual language fidelity and multilingual qa accuracy of current chat-optimized open llms. we first separately assess fidelity and accuracy, and then evaluate their relationship. ## tokenization and multilinguality our results show significant variations in terms of language fidelity, qa accuracy, and the relationship between them, across the models we test with multiq. this prompts us to investigate what factors may explain these differences. we focus on the differences between mistral and llama2 7b, which are particularly surprising given that both models are intended for use in english only. prior research highlights the significant roles of tokenization and training data in multilingual capability @xcite @xcite . since there is little to no public information on the training data of the models that we test, we focus on their tokenizers. background: byte pair encoding both llama2 and mistral employ a tokenizer that uses byte pair encoding @xcite , streamlining sequence encoding by minimizing token count through identifying common subwords. this allows for frequent sequences to be represented through subword tokens, whereas rare or unseen sequences make the model default to individual characters or, failing that, ascii code tokens. unique tokens in multiq first, we evaluate how different tokenization strategies impact prompt tokenization in multiq. we observe a significant deviation in the number of unique tokens used to represent all prompts in multiq: mistral uses 9,933 unique tokens, contrasting with llama2's 10,676. this suggests that llama2 may be less efficient at segmenting the typologically diverse mul-tiq dataset into a smaller number of subwords. tokenization strategies for each model, we group languages into three categories depending on the model's tokenization strategy for the respective language. we develop a heuristic that allows us to classify languages into three tokenization categories: "ascii", "character" or "subword". we do so by compiling the model's 20 most commonly used tokens for representing each language in mul-tiq, removing noise tokens (e.g. sentence start) from the list, and then quantifying the prevalence of ascii tokens and characters based on the tokenid ranges they usually occupy. languages with over 70% of tokens in the ascii or character categories were classified accordingly. for llama2, out of 137 languages, we find 89 subword, 36 character and 12 ascii languages. mistral has 37 character and 9 ascii languages. the models differ in the tokenization of some symbolic languages such as chinese (zh, see figure 6 ): while mistral mostly uses individual character tokens, llama2 more frequently resorts to ascii tokens due to its limited chinese token vocabulary, which limits its ability to effectively tokenize chinese language prompts. tokenization vs. qa accuracy we evaluate the models' average qa accuracy across tokenization categories, finding a clear hierarchy (see ## related work we discuss the related literature with respect to (i) large multilingual benchmarks, (ii) explicitly multilingual models, and (iii) multilingual studies of monolingual models. multilingual benchmarks existing multilingual benchmarks primarily target the performance of fully or partially supervised models on collections of standard nlp tasks, like xnli @xcite . popular examples are the xtreme benchmarks (e.g. @xcite , and xglue @xcite . similarly, researchers also presented benchmarks designed for languages spoken in particular regions, e.g. indicxtreme @xcite for indic languages, masakhan-ner @xcite , and tata @xcite covering african languages, as well as nusax (winata et al., 2023) for indonesian languages. recently, ahuja et al. (2023) proposed mega for evaluating multilingual generative models, which they use to evaluate llms like gpt-4 on a set of standard tasks. in a similar vein, asai et al. (2023) presented buffet and benchmark llms for fewshot transfer. another line of work focuses on multilingual qa datasets for reading comprehension, such xquad artetxe et al. ( 2020 ), tydiqa @xcite and the belebele benchmark @xcite with up to 122 diverse languages. concurrent work proposes a new multilingual instruction tuning dataset called aya @xcite , which also covers a range of openended questions in 114 languages. by comparison, multiq contains parallel questions in a larger set of 137 languages covering 95.4% of grambank features, which demonstrates its typological diversity and allows the analysis of multilingual llm behavior at the margins of language coverage. furthermore, multiq's carefully selected short and simple questions target basic llm knowledge to test only their multilingual capabilities and not ancillary factors such as complex reasoning. ## conclusion we introduced multiq, a new silver standard benchmark for open-ended question answering that covers 137 typologically diverse languages. with multiq, we evaluated the basic multilingual capabilities of six current, chat-optimized open llms, which are restricted in their intended use to just one or a small handful of languages. our analysis focused on two key dimensions of multilingual capability -language fidelity and qa accuracy -and how they relate to each other. we found that all llms we test respond faithfully and/or accurately for at least some languages beyond their intended use. most models are more accurate when they respond faithfully. however, we found that differences across models are large, and that there is a long tail of languages where models are neither accurate nor faithful. finally, we indentified differences in tokenization as a potential explanation for our results. overall, we hope that our findings can motivate further research into improving the multilingual capabilities of open llms, especially for diverse and under-represented languages, so that language technologies can benefit everyone, regardless of which language they speak.