Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer (2024)

Abstract: This paper explores cost-efficient methods to adapt pretrained Large Language Models (LLMs) to new lower-resource languages, with a specific focus on Estonian. Leveraging the Llama 2 model, we investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. Our results demonstrate that even a relatively small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances results on Estonian. Furthermore, we showcase cross-lingual knowledge transfer from high-quality English instructions to Estonian, resulting in improvements in commonsense reasoning and multi-turn conversation capabilities. Our best model, named Llammas, represents the first open-source instruction-following LLM for Estonian. Additionally, we publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in the direction of developing open-source LLMs for Estonian.

## introduction instruction-tuning is a method for aligning large language models (llms) with human preferences @xcite @xcite . however, the majority of instruction-tuning datasets and advancements focus on english. moreover, to benefit from instruction tuning, a strong foundation model is needed but due to the extensive training training data required, such models are available only for a few languages. to overcome the lack of a strong foundation model in the target language, one could try to elicit non-english abilities from english-centric llms through cross-lingual instruction-tuning. in this setup, instructions are given in both english and the target language, often including a translation task * equal contribution to directly stimulate the alignment @xcite @xcite . while empirical evidence indicates benefits from incorporating translation-following demonstrations into the training dataset, the best training strategy and its effectiveness with monolingual pretraining remain unclear. in this paper, we investigate these aspects in the context of creating an instruction-following model for estonian. we focus on a low-resource scenario where only a relatively small amount of monolingual data is available. by utilizing a novel general task instruction dataset, alpaca-est, we examine the impact of combining monolingual pretraining with cross-lingual instruction-tuning using both general and translation task instructions. our experiments with llama 2 @xcite demonstrate the benefits of translation task instructions when no monolingual data is available for additional pretraining. however, monolingual pretraining greatly diminishes the importance of the translation task. furthermore, we showcase that supplementing our instruction-tuning dataset consisting of alpaca @xcite and alpaca-est with high-quality english instructions and english conversations further enhances results on estonian through crosslingual knowledge transfer. this is reflected in improved commonsense reasoning and the ability to engage in multi-turn conversations despite no estonian conversations used during training. as a result, we present llammas -the first opensource instruction-following conversational llm for estonian that achieves competitive zero-shot performance on multiple tasks. et @xcite @xcite . for that purpose, both humanwritten and synthetic instructions generated with llms have been shown to work remarkably well @xcite . one of the prerequisites for instruction-tuning is the availability of a strong pretrained language model, which due to high training costs is the major limiting factor for many to contribute to the development of llms. fortunately, over the last year, a few foundation models @xcite @xcite have been publicly released which somewhat mitigates the issue. however, the models are mostly trained on english and perform poorly on other languages. a common method of acquiring instruction data is using strong proprietary models such as gpt-4 for generating instructions @xcite @xcite . however, @xcite have shown that models trained on these generated datasets learn to imitate the style of strong llms but not necessarily the factuality. ## experiments and results our experiments are divided into two main sections. in the first section, we pretrain llama-2-7b on different amounts of pretraining data and investigate the effect of it on cross-lingual instruction-tuning that is done with translation task and general task 6 https://github.com/tartunlp/estgec/tree/main/m2_scorer_est 7 sacrebleu signature: nrefs:1|case:mixed|eff:no|tok:13a| smooth:exp|version:2.3.1 8 sacrebleu signature: nrefs:1|case:mixed| eff:yes|nc:6|nw:2|space:no|version:2.3.1 instructions (alpacas). in the second section, we study the influence of supplementing alpacas with high-quality english instructions, translations, and conversations to the results on estonian. ## conclusion we successfully adapt llama 2 to estonian by creating llammas -an instruction-following model for estonian. additionally, we release alpaca-est, an alpaca-style general task instruction dataset for estonian. our work has shown competitive results for tasks such as question-answering, machine translation, and grammatical error correction in estonian while keeping solid results for english. we have also identified signs of cross-lingual transfer from english to estonian and investigated the effects of translation bitexts in the fine-tuning process. this work marks the first step towards open-source llms for estonian.