How Vocabulary Sharing Facilitates Multilingualism in LL a MA ? (2024)

Abstract: Large Language Models (LLMs), often show strong performance on English tasks, while exhibiting limitations on other languages. What is an LLM’s multilingual capability when it is trained only on certain languages? The underlying mechanism remains unclear. This study endeavors to examine the multilingual capability of LLMs from the vocabulary sharing perspective by conducting an exhaustive analysis across 101 languages. Through the investigation of the performance gap before and after embedding fine-tuning, we discovered four distinct quadrants. By delving into each quadrant we provide actionable and efficient guidelines for tuning these languages. Extensive experiments reveal that existing LLMs possess multilingual capabilities that surpass our expectations, and we can significantly improve the multilingual performance of LLMs based on these attributes of each quadrant .

## introduction large language models (llm), such as gpt @xcite , palm @xcite , and llama @xcite , are trained on massive amounts of text data. while these models show strong capabilities on english tasks, their performance in other languages is often limited @xcite . significant research effort has been dedicated to enhancing multilingual capabilities by using methods such as continued training with abundant monolingual data @xcite , or employing instruction-tuning foot_1 techniques @xcite . despite the encouraging results, the underlying mechanism of llm's multilingual capability remains mysterious. multilingual capability @xcite refers to how effectively models that have been fine-tuned in one source language can be applied to tasks in other languages and achieve decent performance. this ability has been extensively studied in machine translation @xcite @xcite @xcite and multilingual pre-trained models @xcite @xcite . however, it has not been investigated for english-centric llms, given that the pre-training data is predominantly in english. we aim to address this issue by focusing on the multilingual foundation of pre-trained llms and providing some guidance to help other people train llms more efficiently for non-english languages. generally, multilingual capabilities are built on two key foundations: the volume of multilingual data used during the pre-training stage @xcite @xcite , and the vocabulary @xcite @xcite . in this work, we focus on the latter: vocabulary. to investigate the multilingual foundation provided by the vocabulary of an existing llm, we only fine-tune the embedding layer and keep the rest of the parameters frozen, denoted as embed ft. this approach requires fewer adjustments to the model parameters than full fine-tuning, and unlike lora @xcite , it doesn't require any additional model structure. in our experiments, we focus on the llama as a case study, but the analysis method can be applied to other llms. to examine the multilingual capabilities of llms without loss of generality, we applied embed ft to a 10k @xmath0: lego-mt @xcite , wikimatrix @xcite and newscommentary @xcite , and ted @xcite . we evaluated the bilingual performance (refers to the performance of the fine-tuning languages) and multilingual performance (refers to the performance of other languages) of each model to determine if there was a significant positive or negative change compared to the original model. from the results, all languages can be categorized into four distinct quadrants. the multilingual capability quadrant of the ted dataset, illustrated in figure 1 , includes four quadrants: the reciprocal quadrant, the altruistic quadrant, the stagnant quadrant, and the selfish quadrant. the full definition of each quadrant is in section 3. the selfish quadrant refers to scenarios where the fine-tuned model only improves on the fine-tuning language directions but not other languages. it is considered a default quadrant, as languages that fall into the selfish quadrant exhibit behavior that aligns intuitively with the effects of bilingual fine-tuning. certain languages such as bulgarian fall into the reciprocal quadrant, where training with bilingual data (e.g. @xmath1. the majority of these languages in this quadrant are from the indo-european family, benefiting from the pretraining data and vocabulary sharing. for these languages, we find that there is no need to fine-tune all parameters, which could lead to overfitting to a specific language. we recommend fine-tuning only the embedding layer, which yields bilingual perfor-mance on par with full fine-tuning while preserving the model's multilingual capabilities. remarkably, certain languages exhibit altruistic characteristics. when we use these languages as training data, their primary effect is to enhance multilingual performance. upon further analysis, we discovered that the decline in bilingual performance is primarily due to a change in error types: from those that are easy to score to those that are more challenging. the improvement in multilingual performance, on the other hand, stems from vocabulary sharing. for such languages, employing a small dataset for full fine-tuning can be more effective for multilingual capabilities. indeed, there are certain languages located in the stagnant quadrant that are quite stubborn. this means that using data from these languages doesn't improve bilingual performance or bring about multilingual benefits. regardless of parametereffective tuning strategies (lora) or extensive fine-tuning on large datasets, the results are still disappointing. interestingly, even expanding the vocabulary for full fine-tuning doesn't lead to better results. then, we find that existing llms often over-tokenized these languages, which reduces the density of information they carry. by simply removing the common prefix of tokenized representation, we have seen an average improvement of 2.5 spbleu points. our main contributions are: ## background multilingual large language model large language models (llms; @xcite @xcite @xcite have shown demonstrated performance in english, but the performance in other languages is limited. to address this limitation, researchers have proposed multilingual language models (mllms) that can handle multiple languages simultaneously. the first line of research proposes to learn a shared representation space for multiple languages by first pre-training on multilingual data and then fine-tuning for specific tasks or languages. representative works include mbert @xcite , xlm (lample and conneau, 2019), xlmr @xcite , bloom @xcite , xglm @xcite , and polylm @xcite ). another line of research adopted existing monolingual llms to multilingual using techniques such as prompt engineering @xcite , instruction tuning @xcite @xcite , or continue training @xcite . the multilingual foundation of llm the robust multilingual capabilities of llm are founded on: the presence of diverse multilingual data @xcite @xcite and vocabulary @xcite @xcite . the size of multilingual data is a critical factor in the multilingual capabilities of llm. llama @xcite ) is pre-trained on a vast scale, with over 1.6 trillion tokens, of which less than 9% is multilingual data, foot_2 spanning 20 different languages. llama2 @xcite further enhances the proportion of multilingual data to approximately 11% and increases the number of languages to around 26. polylm @xcite is trained on 640 billion tokens and supports 18 of the most commonly spoken languages. bloom @xcite is trained with data from 46 natural languages. the existing language data in the pre-training phase provides llm with a robust foundation for multilingual capabilities. another key factor is vocabulary construction. a common approach to constructing vocabulary involves tokenizing text into subwords: including byte-level byte-pair-encoding (bbpe), byte-pair-encoding (bpe), sentencepiece (sp) @xcite @xcite , which are units smaller than words that can encapsulate morphological variations. nevertheless, in a multilingual context encompassing a diverse range of scripts, the base vocabulary comprising subwords can become exceedingly large, leading to inefficiency and sparsity. further information on bbpe is in appendix b. ## inherent multilingual capabilities in this section, we begin by exploring the inherent multilingual capabilities of llms and give some fascinating observations detailed in section 3.1. drawing on these insights, we then proceed to conduct an in-depth examination of the multilingual capability of llm in section 3.2. ## enhancing multilingual capability this section conducts a comprehensive analysis of the properties and training strategies of each in the realm of bilingual performance, both strategies prove equally effective. however, when it comes to multilingual performance, the embed ft strategy stands out for its adaptability across various languages, while the full ft strategy tends to over-specialize the model to a single language. the numerical results for each language pair can be found in appendix g. ## conclusion in this study, we performed a comprehensive analysis of 101 languages, categorizing them based on shared characteristics into four distinct quadrants: reciprocal, altruistic, selfish, and stagnant quadrants. upon examining each quadrant in-depth, we identified the primary reasons for the placement of languages within their respective quadrants and provided some practical guidance for training. however, the primary focus of this study is the analysis of persistent language characteristics within each quadrant. a thorough investigation into the conditions that trigger language migration across various phenomena is a subject for our future research.