LEIA : Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation (2024)

Abstract: Adapting English-based large language models (LLMs) to other languages has become increasingly popular due to the efficiency and potential of cross-lingual transfer. However, existing language adaptation methods often overlook the benefits of cross-lingual supervision. In this study, we introduce LEIA, a language adaptation tuning method that utilizes Wikipedia entity names aligned across languages. This method involves augmenting the target language corpus with English entity names and training the model using left-to-right language modeling. We assess LEIA on diverse question answering datasets using 7B-parameter LLMs, demonstrating significant performance gains across various non-English languages.

## introduction while large language models (llms) are emerging as foundational technology @xcite , their data hungriness restricts their application to a few resource-rich languages, with english being the most dominant among them @xcite . a promising strategy to broaden their scope is language adaptation tuning @xcite , where an already-pretrained llm is further trained on a corpus of a language of interest. the underlying motivation is that the model can leverage the knowledge acquired during pretraining to the target language. however, this typical approach overlooks the potential benefits of incorporating cross-lingual supervision. although language models can learn cross-lingual knowledge from a mix of monolingual corpora @xcite , knowledge sharing between languages is limited, and significant performance gaps still exist between english and non-english languages @xcite @xcite . in this work, we propose a language adaptation tuning method, leia (lightweight entity-based inter-language adaptation), that explicitly exploits cross-lingual supervision. we focus on wikipedia as a source of target language corpus, as it offers high-quality text data in a wide range of languages and the text contains hyperlinks to entities (i.e., wikipedia articles) that are aligned across different languages via inter-language links. in our tuning phase, we insert an english entity name beside the corresponding entity in the text (see figure 1 ), and train the model using the left-to-right language modeling objective. this simple modification enables the model to extract and apply its english knowledge about the entities within the target language text during training, which we hypothesize to facilitate cross-lingual knowledge transfer. we assess the effectiveness of leia through experiments using 7b-parameter llms, llama 2 @xcite and swallow @xcite , and a diverse set of question answering datasets. the results demonstrate that through our fine-tuning, llms benefit from knowledge transfer from english and significantly outperform the base models and those fine-tuned without leia. ## method our method involves fine-tuning on a pretrained llm using an augmented corpus derived from the target language edition of wikipedia. specifically, for each hyperlink in the wikipedia corpus, we insert the english name of the referred entity next to the hyperlink (figure 1 ). the english name is enclosed within @xmath0, allowing the model to identify its boundaries. the english name of an entity is extracted from the title of the corresponding english wikipedia page, which is identified using the interlanguage links. we ignore any hyperlinks pointing to entities not present in the english wikipedia. 2 further details are available in appendix a. we fine-tune a pretrained llm using the language modeling objective. we train the model on the corpus of a target language and evaluate it using datasets in the same language. the aforementioned special tokens are added to the vocabulary. to prevent the model from generating these special tokens during inference, we block loss propagation when predicting these tokens during training. ## experiments with llama 2 we start with experiments using the llama 2 7b model @xcite . since it is primarily trained for english, it possesses a substantial amount of english knowledge that could be transferred to other languages. furthermore, its training corpus, containing approximately 38b non-english language tokens @xcite , fosters competitive multilingual performance @xcite . this makes llama 2 a good candidate for investigating the effectiveness of our language adaptation method from english to other languages. ## experiments with swallow in this section, we examine if bilingual language models that already possess substantial knowledge not only in english but also in the target language can benefit from knowledge transfer from leia. we focus on japanese, a language with a variety of benchmark datasets, and experiment with the state-of-the-art english-japanese llm, swallow 7b @xcite ). 4 this model was developed through continual pretraining on llama 2 with vocabulary extension @xcite @xcite , using bilingual corpora consisting of 90b japanese tokens and 10b english tokens. we show that even after the adaptation with a massive target language corpus, leia can further boost the performance of the model. ## analysis qualitative analysis we present five random predictions of leia and llama2+ft ( §3) from the japanese x-codah dataset, where leia answered correctly but the llama2+ft failed in the results in ## related work language adaptation a common domain adaptation technique for language models is training on a domain-specific corpus @xcite , and when different languages are considered as different domains, it can be used for language adaptation. this strategy is shown to be effective in various models including encoder-decoder models @xcite , bidirectional language models @xcite @xcite , and auto-regressive language models @xcite . however, when adapting to a new language, knowledge transfer from one language to an-other can be insufficient due to discrepancies in the surface forms. to facilitate the sharing of internal knowledge across languages, our proposed method leverages cross-lingually aligned entity names. ## conclusion we introduced leia, a method to facilitate crosslingual knowledge transfer via fine-tuning language models using wikipedia text augmented with english entity names. we applied leia to the english llm, llama 2, and the english-japanese llm, swallow, demonstrating significant improvements on various non-english question answering tasks. future research will investigate leia's effectiveness using an arbitrary text corpus with entity annotations generated through entity linking, instead of the wikipedia corpus, and employing the augmented corpus during the pretraining of bi-or multi-lingual llms, rather than relying on posthoc fine-tuning.