P retrained Language Model Embryology: T he Birth of ALBERT (2020)

Abstract: While behaviors of pretrained language models (LMs) have been thoroughly examined, what happened during pretraining is rarely studied. We thus investigate the developmental process from a set of randomly initialized parameters to a totipotent language model, which we refer to as the embryology of a pretrained language model. Our results show that ALBERT learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. We also find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. These findings suggest that knowledge of a pretrained model varies during pretraining, and having more pretrain steps does not necessarily provide a model with more comprehensive knowledge. We provide source codes and pretrained models to reproduce our results at https://github.com/d223302/albert-embryology .

## introduction the world of nlp has gone through some tremendous revolution since the proposal of contextualized word embeddings. some big names are elmo @xcite , gpt @xcite , and bert @xcite , along with its variants @xcite @xcite . performance boosts on miscellaneous downstream tasks have been reported by finetuning these totipotent pretrained language models. with a view to better grasping what has been learned by these contextualized word embedding models, probing is generally applied to the pretrained models and the models finetuned from them. probing targets can range from linguistic knowledge, including semantic roles and syntactic structures @xcite @xcite , to world knowledge @xcite . while the previous work focuses on what knowledge has been learned after pretraining of transformer-based language models, few delve into their dynamics during pretraining. what happened during the training process of a deep neural network model has been widely studied, including @xcite , @xcite , @xcite , @xcite . some previous works also study the dynamics of the training process of an lstm language model @xcite , but the training dynamics of a large scale pretrained language models are not well-studied. in this work, we probe albert @xcite during its pretraining phase every n parameter update steps and study what it has learned and what it can achieve so far. we perform a series of experiments, detailed in the following sections, to investigate the development of predicting and reconstructing tokens (section 3), how linguistic and world knowledge evolve through time (section 4, section 6), and whether amassing those information serves as an assurance of good downstream task performances (section 5). we have the following findings based on al-bert: while we only include the detailed results of albert in the main text, we find that the results also generalize to the other two transformer-based language models, electra @xcite and bert, which are quite different from albert in the sense of pretext task and model architecture. we put the detailed results of electra and bert in the appendix. ## pretraining albert albert is a variant of bert with cross-layer parameters sharing and factorized embedding parameterization. the reason why we initially chose albert as our subject lies in its parameter efficiency, which becomes a significant issue when we need to store 1000 checkpoints during the pretraining process. to investigate what happened during the pretraining process of albert, we pretrained an albert-base model ourselves. to maximally reproduce the results in @xcite , we follow most of the training hyperparameters in the original work, only modifying some hyperparameters to fit in our limited computation resources foot_1 . we also follow @xcite , using english wikipedia as our pretraining data, and we use the project guttenberg dataset @xcite instead of bookcorpus. the total size of the corpus used in pretraining is 16gb. the pretraining was done on a single cloud tpu v3 and took eight days to finish 1m pretrain steps, costing around 700 usd. more details on pretraining are specified in appendix b.1. 3 learning to predict the masked tokens and reconstruct the input tokens 0 5 k 10 k 15 k 20 k 25 k 30 k pretrain step 0.0 0.2 0.4 0.6 0.8 1.0 accuracy (rescaled) conj. det. prep. adj. noun proper noun pron. adv. verb (a) token reconstruction 0 25 k 50 k 75 k 100 k pretrain step 0.0 0.2 0.4 0.6 0.8 accuracy (rescaled) (b) mask prediction figure 1: rescaled accuracy of token reconstruction and mask prediction during pretraining. we rescale the accuracy of each line by the accuracy when the model is fully pretrained, i.e., the accuracy after pretraining 1m steps. token reconstruction are evaluated every 1k pretrain steps, and mask prediction evaluated every 5k steps. the results of reconstruction are shown in figure 1(a ). albert first learns to reconstruct function words, e.g., determiners, prepositions, and then gradually learns to reconstruct content words in the order of verb, adverb, adjective, noun, and proper noun. we also found that different forms and tenses of a verb do not share the same learning schedule, with third-person singular present being the easiest to reconstruct and present participle being the hardest (shown in appendix c.2). the prediction results in figure 1 (b) reveal that learning mask prediction is generally more challenging than token reconstruction. albert learns to predict masked tokens with an order similar to token reconstruction, though much slower and less accurate. we find that bert also learns to perform mask prediction and token reconstruction in a similar fashion, with the results provided in appendix c.4. ## probing linguistic knowledge development during pretraining probing is widely used to understand what kind of information is encoded in embeddings of a language model. in short, probing experiments train a task-specific classifier to examine if token embeddings contain the knowledge required for the probing task. different language models may give different results on different probing tasks, and representations from different layers of a language model may also contain different linguistic information @xcite . our probing experiments are modified from the "edge probing" framework in @xcite . @xcite previously showed that probing models should be selective, so we use linear classifiers for probing. we select four probing tasks for our experiments: part of speech (pos) tagging, constituent (const) tagging, coreference (coref) resolution, and semantic role labeling (srl). the former two tasks probe syntactic knowledge hidden in token embeddings, and the last two tasks are designed to inspect the semantic knowledge provided by token embeddings. we use annotations provided in ontonotes 5.0 @xcite in our experiments. the probing results are shown in figure 2b . we observe that all four tasks show similar trends during pretraining, indicating that semantic knowledge and syntactic knowledge are developed simultaneously during pretraining. for syntactically related tasks, the performance of both pos tagging and constituent tagging boost very fast in the first 100k pretrain steps, and no further improvement can be seen throughout the remaining pretraining process, while performance fluctuates from time to time. we also observe an interesting phenomenon: the probed performances of srl peak at around 150k steps and slightly decay over the remaining pretraining process, suggesting that some information in particular layers related to probing has been dwindling while the albert model strives to advance its performance on the pretraining objective. the loss of the pretraining objective is also shown in figure 2a . scrutinizing the probing results of different layers (figure 3 and appendix d.3), we find that the behaviors among different layers are slightly different. while the layers closer to output layer perform worse than layers closer to input layer at the beginning of pretraining, their performances rise figure 4: attention patterns of head 11 across layer 1 (first row), 2 (second row), and 8 (third row) during pretraining. pretrain steps labeled atop the attention map. we averaged the attention maps of different input sentences to get the attention pattern of a single head. ## does expensive and lengthy pretraining guarantee exceptional results on downstream tasks? glue scores, whether the performance gain of downstream tasks is proportional to the resources spent on additional pretrain steps is unknown. this drives us to explore the downstream performance of the albert model before fully pretrained. we choose glue benchmark @xcite for downstream evaluation, while excluding wnli, following @xcite . we illustrate our results of the downstream performance of the albert model during pretraining in figure 5 . while the glue score gradually increases as pretraining proceeds, the performance after 250k does not pale in comparison with a fully pretrained model (80.23 v.s. 81.50). from figure 5b , we also observe that most glue tasks reach comparable results with their fully pretrained counterpart over 250k pretrain steps, except for mnli and qnli, indicating nli tasks do benefit from more pretrain steps when the training set is large. we also finetuned bert and electra models as pretraining proceeds, and we observe similar trends. the glue scores of the bert and elec-tra model rise drastically in the first 100k pretrain steps, and then the performance increments less slowly afterward. we put the detailed result of these two models in section e.4. we conclude that it may not be necessary to train an albert model until its pretraining loss converges to obtain exceptional downstream performance. the majority of its capability for downstream tasks has already been learned in the early stage of pretraining. note that our results do not contradict previous findings in @xcite , @xcite , and @xcite , all of which showing that downstream tasks do benefit from more pretrain steps; we show that the performance gain on downstream tasks in latter pretrain steps might be disproportional to the cost on more pretrain steps. the results are shown in figure 6 , in which we observe that world knowledge is indeed built up during pretraining, while performance fluctuates occasionally. from figure 6 , it is clear that while some types of knowledge stay static during pretraining, some vary drastically over time, and the result of a fully pretrained model (at 1m steps) may not contain the most amount of world knowledge. we infer that world knowledge of a model depends on the corpus it has seen recently, and it tends to forget some knowledge that it has seen long ago. these results imply that it may not be sufficient to draw a conclusion on albert's potential as a knowledge base merely based on the final pretrained one's behavior. we also provide qualitative results in appendix f.2. ## conclusion although finetuning from pretrained language models puts in phenomenal downstream performance, the reason is not fully uncovered. this work aims to unveil the mystery of the pretrained language model by looking into how it evolves. our findings show that the learning speeds for reconstructing and predicting tokens differ across pos. we find that the model acquires semantic and syntactic knowledge simultaneously at the early pretraining stage. we show that the model is already prepared for finetuning on downstream tasks at its early pretraining stage. our results also reveal that the model's world knowledge does not stay static even when pretraining loss converges. we hope our work can bring more insights into what makes a pretrained language model a pretrained language model.