The Less the Merrier? Investigating Language Representation in Multilingual Models (2023)

Abstract: Multilingual Language Models offer a way to incorporate multiple languages in one model and utilize cross-language transfer learning to improve performance for different Natural Language Processing (NLP) tasks. Despite progress in multilingual models, not all languages are supported as well, particularly in low-resource settings. In this work, we investigate the linguistic representation of different languages in multilingual models. We start by asking the question which languages are supported in popular multilingual models and which languages are left behind. Then, for included languages, we look at models’ learned representations based on language family and dialect and try to understand how models’ learned representations for (1) seen and (2) unseen languages vary across different language groups. In addition, we test and analyze performance on downstream tasks such as text generation and Named Entity Recognition. We observe from our experiments that community-centered models—models that focus on languages of a given family or geographical location and are built by communities who speak them—perform better at distinguishing between languages in the same family for low-resource languages. Our paper contributes to the literature in understanding multilingual models and their shortcomings and offers insights on potential ways to improve them.

## introduction while we have seen improvements in state-of-theart performance in various nlp tasks by multilingual models @xcite , there is a disparity in which languages are actively studied. the field of nlp has largely been anglocentric, with a large portion of the world's languages, particularly low-resource languages, not being covered in the literature @xcite @xcite . for languages that have been included in popular multilingual models, performance is not the same for all languages @xcite . low-resource languages suffer in performance even when they are included in multilingual models. several works @xcite @xcite have proposed methods for improving performance for low-resource languages. previous work @xcite presents an argument that languages might benefit from being included in multilingual models as the models learn language-independent feature representations, while it concludes that the question of the benefit of multilingual training for a given language remains open. previous works also show that multilingual models might suffer from "negative interference" (zirui wang, 2019) in both high-resource @xcite and lowresource (zirui wang, 2020) settings. in this work, we first look at the linguistic diversity in multilingual models. we then investigate the embeddings for different languages in multilingual models and show how the representations affect downstream performance in language identification. for our analysis, we used three autoregressive and five autoencoder models. first, we looked at 2d visualizations of learned representations for all models. then, we evaluated autoregressive models' performance on text generation and the autoencoder models by language classification based on learned representations and performance on named entity recognition (ner). we base our analysis on (1) language family, (2) dialects, and (3) writing scripts, with a focus on low-resource language settings. ## related works in efforts to include more of the world's languages, previous works have built multilingual language models over the years. while models with larger numbers and more diverse sets of languages have shown commendable performance in several nlp tasks @xcite , we have also seen community-centered models improve upon task performance @xcite . previous work @xcite hypothesizes that models might suffer from "curse of multilinguality", which describes how for a given model size, increasing the number of languages does not improve performance for individual languages after a certain point. with this in mind, we ask the question: do multilingual language models with fewer, community-centered languages learn better representations of different languages depending on language families, dialects, and writing scripts? previous works have analyzed how pre-trained language models learn representations for different languages using probing tasks @xcite @xcite as well as investigating the geometry of sub-spaces @xcite . one previous work @xcite focuses on understanding multilingual language models' overall embedding space geometry and identifies axes for language-sensitive and language-neutral features. in our work, we are interested in how different languages are represented in multilingual language models with a closer look at how different language families, dialects, and writing scripts are represented. ## models and data we chose models from two categories for our experiments: autoencoder and autoregressive models. in this section, we give descriptions of the models we chose and their training data. ## language diversity before starting our experiments to understand how multilingual models learn representations for different languages, we first looked at which languages are included and which languages are excluded from mainstream nlp research. from the models discussed in section 3, we selected xlm-r model type model name size languages data autoencoders xlm-r 270m 100 filtered commoncrawl afrolm 270m 23 jw300, bible, news bert-multilingual 110m 104 bookscorpus and english wikipedia arabert 110m 1 arabic news, arabic corpus, osian: the open source international arabic news corpus indicbert 12m 12 ai4bharat's monolingual corpus autoregressive gpt-3 175b 119 filtered commoncrawl, webtext (liu and curran, 2006), e-books, english wikipedia llama 65b 20 english commoncrawl, c4, github, wikipedia, gutenberg and books3, arxiv, stack exchange bloom 560m 59 roots (bigscience, 2022b) and llama from generic multilingual models and afrolm from community-centered models to evaluate their linguistic diversity across countries. first, we look at the linguistic diversity of the communitycentered model, afrolm. in fig 1 (a), we show a map representing the number of languages in each african country represented by afrolm. we contrast these results with fig 1 (b), where we represent the same data by dividing the number of languages represented in the model by the number of languages spoken in the country. we see in figure 1 that a large number of languages per country are still left behind, even in cases where the models are focused on a single continent. next, we look at xlm-r which has been a popular choice in multilingual studies @xcite . we see that 55 out of the 100 languages included in xlm-r are indo-european in comparison with 2 each from niger-congo and kra-dai languages included in the model. in fig. 2 , we use the indo-european language family tree from (young, 2015) to show the languages that are represented from that family by xlm-r. for llama, ## conclusion in this work, we investigated linguistic diversity, the learned representation of different languages, and the downstream task performance of multilingual models. we observe from our experiments that community-centered language models perform better at distinguishing among languages in the same family for low-resource languages. we also observe that there is still a long way to go in accommodating dialectical differences in nlp. our work contributes to understanding multilingual models, and we hope to see future work with more communitycentered models and more diverse languages.