Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly (2024)

Abstract: Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address this, i.e., multilingual pretraining and multilingual instruction tuning. However, whether and how do such methods contribute to the cross-lingual knowledge alignment inside the models is unknown. In this paper, we propose CLiKA, a systematic framework to assess the cross-lingual knowledge alignment of LLMs in the Performance, Consistency and Conductivity levels, and explored the effect of multilingual pretraining and instruction tuning on the degree of alignment. Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed. Namely, continued pretraining improves the alignment of the target language at the cost of other languages, while mixed pretraining affect other languages less. Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity.

## introduction the language imbalance of modern nlp systems has long been discussed @xcite . many studies have shown that the performance of current llms on english tasks is much higher than non-english tasks @xcite . one possible explanation is that the knowledge required for completing tasks are learnt mainly from english text. so it could be better retrieved with english than with other languages. recent studies suggest that cross-lingual consistency may be a possible way to narrow the gap between languages @xcite . ideally, if the knowledge of a fact could be aligned to a "true" representation regardless of the language it is described with, it may be retrieved in any required language, helping the model to generalize across languages. in this paper, we refer to this internal mechanism as cross-lingual knowledge alignment. to improve llms' performance in non-english languages, two approaches are proposed. the first is multilingual pretraining, which add non-english data in the pretraining corpus. the second is multilingual instruction tuning, i.e., using tasks in different languages or translation-related tasks, to finetune a foundation model @xcite . although these methods do improve llms' non-english performance, whether they can bring real cross-lingual knowledge alignment is not well investigated. therefore, the aim of this study is to evaluate the effect of multilingual pretraining and instruction tuning on the cross-lingual knowledge alignment mechanism. however, the evaluation is challenging, because the improvement of performance may come from the improvement of language ability in a specific language or the improvement of knowledge alignment. it is hard to discriminate the effects of the two by performance as the single clue. furthermore, even if llms show higher consistency between two languages, there is still possibility that the knowledge in the two languages are learned correctly but separately. to meet this challenge, we propose to assess cross-lingual knowledge alignment systematically, by using 3 deepening levels of measurement: most previous evaluations of the multilingualism only focus on the pf level @xcite and the ct level @xcite , but the cd level is closer to the nature of knowledge alignment. the evaluation for the cd level is non-trivial, because the successful retrieval of a factual knowledge learned in another language depends not only on the alignment of the knowledge, but also on the basic language ability in the current language. for example, even if the alignment is correct, retrieving this knowledge in non-english languages such as japanese is harder than doing it in english, because the model's basic ability is not as good. in this regard, we propose a systematic framework, clika (standing for cross-lingual knowledge alignment), to reveal the effects of different multilingualism. clika considers all three levels of the alignment, with specific metrics for each level. clika includes three comparative settings: factual, basic, and fictional, to further discriminate the effects of language abilities and knowledge alignment in knowledge retrieval. we apply clika to popular llms, including bloom @xcite , llama @xcite , chatgpt and their multilingual variants @xcite . our results indicate that: to address this issue, some work uses supervised parallel data in the pretraining stage to enhance the model's multilingualism, e.g. xlm @xcite and bloom @xcite . such method is also used nowadays to train llms with better multilingualism, bringing models such as palm 2 (anil et al., 2023) on multiple languages; and chatglm @xcite and baichuan 2 @xcite ) mainly focusing on english and chinese. also, another popular practise is to fine-tune an english-centered foundation model with translation and instruction data in english and other languages @xcite , resulting in models with better translation ability and instruction-following ability in those languages, such as bigtranslate @xcite , bayling @xcite , x-llama/m-llama @xcite , and mfti @xcite . however, despite the performance gain on multilingual benchmarks, the effect of these training methods on cross-lingual knowledge alignment is still to be examined. ## methods because the cross-lingual knowledge retrieval is affected by both the basic language ability and the three levels of cross-lingual knowledge alignment, our clika framework adopts special testing datasets and metrics to evaluate them separately. ## conclusion in this paper, we evaluated the cross-lingual knowledge alignment of representative multilingual llms, and systematically assessed the effect of multilingual pretraining and instruction tuning on it, using the proposed clika framework. the first part of our results shows that the crosslingual knowledge alignment of current multilingual llms is unsatisfactory, and even though they show high cross-lingual consistency, it is more likely to come from overlapping training data, instead of knowledge conduction between languages. the second part of our results demonstrates the effect on basic language ability and knowledge alignment of adding multilingualism in pretraining and instruction tuning, which shows that mixed multilingual pretraining and multilingual instruction tuning is beneficial. however, our results also point out that neither of the two techniques can improve the knowledge conductivity of llms, meaning the cross-lingual alignment in current models are still shallow and requires novel strategies for improvement.