Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models (2022)

Abstract: The emergent cross-lingual transfer seen in multilingual pretrained models has sparked significant interest in studying their behavior. However, because these analyses have focused on fully trained multilingual models, little is known about the dynamics of the multilingual pretraining process. We investigate when these models acquire their in-language and cross-lingual abilities by probing checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones. In contrast, the point in pretraining when the model learns to transfer cross-lingually differs across language pairs. Interestingly, we also observe that, across many languages and tasks, the final model layer exhibits significant performance degradation over time, while linguistic knowledge propagates to lower layers of the network. Taken together, these insights highlight the complexity of multilingual pretraining and the resulting varied behavior for different languages over time.

## introduction large-scale language models pretrained jointly on text from many different languages @xcite @xcite perform very well on various languages and on crosslingual transfer between them (e.g., @xcite . due to this success, there has been a great deal of interest in uncovering what these models learn from the multilingual pretraining signal ( §6). however, these works analyze a single model artifact: the final training checkpoint at which the model is considered to be converged. recent work has also studied monolingual models by expanding the analysis to multiple pretraining checkpoints to see how model knowledge changes across time @xcite . we analyze multilingual training checkpoints throughout the pretraining process in order to identify when multilingual models obtain their inlanguage and cross-lingual abilities. the case of multilingual language models is particularly interesting, as the model learns both to capture individual languages and to transfer between them just from unbalanced multitask language modeling for each language. specifically, we retrain a popular multilingual model, xlm-r @xcite , and run a suite of linguistic tasks covering 59 languages on checkpoints from across the pretraining process. 1 this suite evaluates different syntactic and semantic skills in both monolingual and cross-lingual transfer settings. while our analysis primarily focuses on the knowledge captured in model output representations over time, we also consider how the performance of internal layers changes during pretraining for a subset of tasks. our analysis uncovers several insights into multilingual knowledge acquisition. first, while the model acquires most in-language linguistic information early on, cross-lingual transfer is learned across the entire pretraining process. second, the order in which the model acquires linguistic information for each language is generally consistent with monolingual models: lower-level syntax is learned prior to higher-level syntax and then semantics. in comparison, the order in which the model learns to transfer linguistic information between specific languages can vary wildly. finally, we observe significant degradation of performance for many languages at the final layer of the last, converged model checkpoint. however, lower layers of the network often continue to improve later in pretraining and outperform the final layer, particularly for cross-lingual transfer. these observations indicate that there is not a single time step (or layer) in pretraining that performs the best across all languages and suggest that methods that better balance these tradeoffs could improve multilingual pretraining in the future. ## analyzing knowledge acquisition throughout multilingual pretraining our goal is to quantify when information is learned by multilingual models across pretraining. to this end, we reproduce a popular multilingual pretrained model, xlm-r -referred to as xlm-r replica -and retain several training checkpoints ( §2.1). a suite of linguistic tasks is then run on the various checkpoints ( §2.2). for a subset of these tasks, we also evaluate at which layer in the network information is captured during pretraining. since we want to identify what knowledge is gleaned from the pretraining signal, each task is evaluated without finetuning. the majority of our tasks are tested via probes, in which representations are taken from the final layer of the frozen checkpoint and used as input features to a linear model trained on the task of interest @xcite . additional evaluations we consider for the model include an intrinsic evaluation of model learning (bpc) and unsupervised word alignment of model representations. each of the tasks in our evaluation suite tests the extent to which a training checkpoint captures some form of linguistic information, or a specific aspect of linguistic knowledge, and they serve as a proxy for language understanding in the model. ## in-language learning throughout pretraining we first consider the in-language, or monolingual, performance of xlm-r replica on different types of linguistic information across pretraining. we find that in-language linguistics is learned (very) early in pretraining and is acquired in a consistent order, with lower-level syntactic information learned before more complex syntax and semantics. additionally, the final checkpoint of xlm-r replica often experiences performance degradation compared to the best checkpoint for a language, suggesting that the model is forgetting information for a number of languages by the end of pretraining. ## cross-lingual transfer throughout pretraining another question of interest is: when do multilingual models learn to transfer between languages? we find that cross-lingual transfer is acquired later in pretraining than monolingual linguistics and that the step at which xlm-r replica learns to transfer a specific language pair varies greatly. furthermore, though the order in which xlm-r replica learns to transfer different linguistic information across languages is on average consistent with in-language results, the order in which the model learns to transfer across specific language pairs for different tasks is much more inconsistent. ## layer-wise learning throughout pretraining in the experiments above we show that in many cases the final layer of xlm-r replica forgets information by the end of pretraining. motivated by this, we investigate whether this information is retained in a different part of the network by probing how information changes across layers during pretraining. we find a surprising trend in how the best-performing layer changes over time: the model acquires knowledge in higher layers early on, which then propagates to and improves in the lower layers later in pretraining. ## conclusion in this paper, we probe training checkpoints across time to analyze the training dynamics of the xlm-r pretraining process. we find that although the model learns in-language linguistic information early in training -similar to findings on monolingual models -cross-lingual transfer is obtained all throughout the pretraining process. furthermore, the order in which linguistic information is acquired by the model is generally consistent, with lower-level syntax acquired before semantics. however, we observe that for individual language pairs this order can vary wildly, and our statistical analyses demonstrate that model learning speed and overall performance on specific languages (and pairs) are difficult to predict from language-specific factors. we also observe that the final model artifact of xlm-r replica performs often significantly worse than earlier training checkpoints on many languages and tasks. however, layer-wise analysis of the model shows that linguistic information shifts lower in the network during pretraining, with lower layers eventually outperforming the final layer. altogether, these findings provide a better understanding of multilingual training dynamics that can inform future pretraining approaches.