Incremental pre-training from smaller language models (2024)

Abstract: Large language models have recently become a new learning paradigm and led to state-of-the-art performance across a range of tasks. As explosive open-source pre-trained models are available, it is worth investigating how to better utilize existing models. We propose a simple yet effective method, Incr-Pretrain, for incrementally pre-training language models from smaller well-trained source models. Different layer-wise transfer strategies were introduced for model augmentation including parameter copying, initial value padding, and model distillation. Experiments on multiple zero-shot learning tasks demonstrate satisfying inference performance upon transferring and promising training efficiency during continuing pre-training. Compared to training from scratch, Incr-Pretrain can save up to half the training time to get a similar testing loss.

## introduction large language models have led to state-of-theart accuracies across a range of tasks and have demonstrated strong performances with few-shot in-context learning @xcite @xcite . from gpt @xcite to switch-transformer @xcite , the number of parameters grows from 125 million to 1.6 trillion at an exponential rate. the study of @xcite shows that a large language model (up to 175 billion) can have strong context learning ability, and obtains comparable performances with state-of-the-art finetune style methods even without any parameter updating. an empirical scaling law @xcite shows that the larger models with wider and deeper architecture are significantly more sampleefficient on a relatively modest amount of data. furthermore, as the model size increases, there is still room for performance improvement. however, training large language models from scratch always costs huge computing resources and time. for instance, nvidia leveraged their selene supercomputer to perform scaling studies and used up to 3,072 a100 gpus for training the largest megatron @xcite model (1 trillion parameters). openai spent 355 gpu-years for training @xcite , and the total costs are more than ten million dollars. most existing model transfer methods aim at improving the performance of downstream tasks, e.g. transfer learning @xcite or speeding up the inference process, e.g. knowledge distillation @xcite , but studies for accelerating model pre-training from scratch remain limited. to our knowledge, no research on how to transfer a small pre-trained model to a large model has been done. we introduce incr-pretrain to augment a smaller source transformer model to a larger target model and make them have comparable performances, both upon transferring and after continuing pretraining. different layer-wise transfer strategies are introduced for model augmentation including parameter copying, padding and model distillation. specifically, we propose a kl-divergence-based approximation method to distill the layernorm layer to address a mathematically intractable issue during transferring. we tested our method's performance on zero-shot tasks of bert-base and gpt-2, and the results show that the augmented models obtain satisfying performances. when incrementally training a dialogue-gpt model on different scales, the training and testing losses can continue declining from the values before transferring. the total training time can be saved up to half compared with that training from scratch. to the best of our knowledge, this is the first parameter-based method for incrementally pretraining language models. our method can help reduce the heavy resource cost of training large language models from scratch and can be applied to almost any open-source pre-trained model in the transformers library @xcite . the proposed method is also compatible with mainstream parallel training techniques. we summarize our contributions as follows: 1) we prove that it is feasible to train a larger language model from smaller transformer models without training from scratch; 2) we propose a distillation-based method to transfer the layernorm parameters. ## method we present the implementation of incr-pretrain in the scenarios of both widening and deepening a transformer model. for widening the model, we use parameter copying and padding to transfer the embedding, attention and mlp layers and a distillation-based method to adjust the layer-norm's parameters to the new input distribution due to the changed input dimension problem. for deepening the model, we initialize the deeper layers with small parameter values, the noise of which could be overwritten by the residual connection setting and have less adverse impact on the entire model. the overall framework is shown in figure 1 . ## experiments we conducted extensive experiments on inference upon transferring and continuing pre-training. we tested bert on the cloze tasks and gpt on the next word prediction tasks, which are corresponding objectives at their stages of pre-training. to validate the time efficiency of using incr-pretrain, we continued to pre-train the target model and compared the loss curve with that of training from scratch. ## conclusion we propose a transfer strategy that can incrementally pre-train language models with acceptable performance decreases. the inference performances show that the target models are comparable with the source models. the continuing pre-training experiment demonstrates that the transfer method is computationally efficient compared to pre-training a large model from scratch. our future directions will be transferring with different parallel styles and exploring the influence of padding values.