Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024)

Abstract: In recent years, Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence. However, training these models from scratch requires substantial computational resources and vast amounts of text data. In this paper, we explores an alternative approach to constructing a LLM for a new language by continually pre-training (CPT) from existing pre-trained LLMs, instead of using randomly initialized parameters. Based on parallel experiments on 40 model sizes ranging from 40M to 5B parameters, we find that 1) CPT converges faster and saves significant resources in a scalable manner. 2) CPT adheres to an extended scaling law derived from with a joint data-parameter scaling term. 3) The compute-optimal data-parameter allocation for CPT markedly differs based on our estimated scaling factors. 4) The effectiveness of transfer scale is influenced by training duration and linguistic properties, while robust to data replaying, a method that effectively mitigates catastrophic forgetting in CPT. We hope our findings provide deeper insights into the transferability of LLMs at scale for the research community.

## introduction in recent years, large language models (llms) pre-trained on web-scale corpora have achieved significant success in various language tasks @xcite @xcite . as the scale of pre-training increases, llms have exhibited remarkable abilities, particularly in transferring knowledge across different domains @xcite . training an llm from scratch is prohibitively expensive. to address this, some practitioners leverage transfer learning to adapt llms to new domains or tasks. this usually involves finetuning the models on a small dataset within the target domain. previous works have showcased multiple benefits of transfer learning in fine-tuning when the transfer gap is small, including faster convergence and better final performance @xcite . however, it remains unclear if these benefits hold when finetuning on massive data or across large distribution shifts (e.g., different languages). this becomes a crucial consideration if one aims to efficiently build an llm using transfer learning, especially when there is a sufficient amount of data available from different distributions. to fill this gap, we investigate training llms with transfer learning on large pre-training corpora. to be specific, we create llms for a new language by using pre-trained llms as initialization instead of starting from scratch. we refer to this approach as continual pre-training (cpt). the motivation for our work stems from the inherent ability of meta-knowledge to transfer across various languages @xcite @xcite . by leveraging this transferability, llms can use existing linguistic knowledge to enable more efficient training. in this paper, we conduct pre-training with parameter sizes ranging from 40m to 5b, spanning 40 different sizes, to systematically study the effect of cpt at different conditions and scales. specifically, we use english as the source language for the source model and chinese as the target language for cpt. we compare two different training strategies: our main focus lies in the comparative analysis between the two strategies, including their scaling behaviors, the robustness of scaling, and their corresponding impact factors. we also study the technique of data replaying @xcite to mitigate catastrophic forgetting in cpt. data replaying involves replaying a portion of the source language data during the training of the target language model. finally, to explain and model the scalings under different settings, we fit a new extended scaling law for cpt, derived from hoffmann et al. @xcite . our findings are outlined as follows: ## setup 2.1 training framework to compare the transfer effects in cpt versus pretraining from scratch, we train two sets of models with the same parameter sizes. additionally, another set of model checkpoints is trained in the source language to serve as the initialization for the continually pre-trained models. the training configurations for the three sets of models are shown in to simplify the experiments, we use identical training strategies for all three pre-training sets, including learning rate schedules, batch sizes and trained token counts. all models are pre-trained with a context length of 2048 and undergo training on tokens equivalent to 20 times the model size (e.g., a 5b model is trained on 100b tokens). although this is far from the extensive pre-training seen in recent practices @xcite , as outlined in @xcite , the 20x trained token count is sufficient to demonstrate the lossdata scaling relationship. our learning rate (lr) schedule features a cosine lr decay from a maximum lr of 2 × 10 -4 and an lr warm-up, which increases the lr to the maximum in the first 5% of the training session. we use different batch sizes for different parameter sizes, as shown in ## conclusion in this paper, we explored continual pre-training (cpt), analyzing its principles, influencing factors, and best practices. through training multiple llms with varying sizes, language distributions, and conditions, we derived an extended scaling law for cpt. our results quantitatively demonstrate that cpt achieves lower loss more quickly, saving 25% to 50% of training resources. however, cpt is particularly sensitive to factors such as language type, training duration, and catastrophic forgetting. based on these insights, we provide best practices for cpt, including optimal data-to-parameter allocation and replay ratios. these findings motivate future practitioners to apply cpt, offering deeper insights into factors like dataset distribution and training budgets.