Analyzing BERT Cross-lingual Transfer Capabilities in Continual Sequence Labeling (2022)

Abstract: Knowledge transfer between neural language models is a widely used technique that has proven to improve performance in a multitude of natural language tasks, in particular with the recent rise of large pre-trained language models like BERT. Similarly, high cross-lingual transfer has been shown to occur in multilingual language models. Hence, it is of great importance to better understand this phenomenon as well as its limits. While most studies about cross-lingual transfer focus on training on independent and identically distributed (i.e. i.i.d.) samples, in this paper we study cross-lingual transfer in a continual learning setting on two sequence labeling tasks: slot-filling and named entity recognition. We investigate this by training multilingual BERT on sequences of 9 languages, one language at a time, on the MultiATIS++ and MultiCoNER corpora. Our first findings are that forward transfer between languages is retained although forgetting is present. Additional experiments show that lost performance can be recovered with as little as a single training epoch even if forgetting was high, which can be explained by a progressive shift of model parameters towards a better multilingual initialization. We also find that commonly used metrics might be insufficient to assess continual learning performance.

## introduction state-of-the-art models for natural language processing (nlp) usually leverage deep neural networks. in particular, pre-trained transformerbased @xcite language models like bert @xcite have proven to perform very well on various nlp tasks, often achieving state-of-the-art results @xcite . these models are pre-trained in a self-supervised way on large text corpora and rely on knowledge transfer to solve downstream tasks, where the pre-trained model is fine-tuned on the target task. multilingual versions of these models have also been trained and demonstrate high cross-lingual transfer as well @xcite @xcite . given the interest in these models for cross-lingual transfer, it is of great importance to better understand this phenomenon as well as its limits. in this work, we analyse the cross-lingual transfer capabilities of multilingual bert and we work on sequence labeling, where each token of a sentence must be annotated with a specific label. this problem regroups various nlp tasks like named entity recognition (ner), part-of-speech (pos) tagging, text chunking and slot-filling. we focus our study on two of these tasks using two multilingual corpora foot_0 : multiatis++ for slot-filling @xcite and multiconer for ner @xcite . experimenting on different corpora allows us to identify which observations may generalize and which ones may be corpus specific. while most cross-lingual transfer studies about slot-filling or ner focus either on joint training or training on a source and a target language @xcite @xcite @xcite , our main contribution is a study with special focus on continual cross-lingual transfer, where the model performs one single task but is progressively adapted over a sequence of languages. we believe this experimental setup to be interesting not only as a novel way of studying crosslingual transfer but also because it is better suited to real case scenarios. indeed, adaptation to new data over time is a highly desirable feature of most nlp models: oftentimes, collecting data and anfor each language in the given order, we train the model on its training set, select the best epoch on the development set and then test on all test sets independently. in this paper, we first describe in section 2 and 3 the task, the corpora and the model we are working with. then in section 4 we define the different continual learning metrics that we use in our experiment. our study is guided by the following research questions, as presented in section 5: does cross-lingual transfer exist during continual training or does catastrophic forgetting prevent it? how much transfer can we expect relative to monolingual and multilingual i.i.d. training? in section 6 we perform an extensive analysis on multiatis++ in order to understand how transfer is affected by the training sequence. finally, in section 7 we investigate whether lost performance (due to forgetting) can be recovered and at what cost. ## model we use the multilingual bert @xcite base model, consisting of 12 multi-head attention layers with 12 heads and hidden size of 768 (177m parameters). this model was trained on large wikipedia dumps from 104 different languages using masked language modelling and next sentence prediction objectives. as we use the model for sequence labeling, we append a two-layer feed-forward classifier with hidden size 768 and relu (rectified linear unit) activation @xcite . the input of the classifier are the last layer word hidden states after applying dropout with @xmath0.1. following @xcite , we train the model on multiatis++ using the adam optimizer (kingma and ba, 2015) with a learning rate of 10 -5 and a batch size of 32 utterances for 50 epochs (unless stated otherwise), selecting the model with the highest slot f1 on the corresponding dev set. we train the model on multiconer the same way, except for the learning rate (optimized on dev and set to 5 × 10 -5 ) and the number of epochs, which is set to 15. we evaluate the model on all test i sets for every language i using the slot f1 calculated with the seqeval library @xcite . ## continual learning metrics cross-lingual transfer can be defined as the performance improvement of a model on a particular language based on knowledge of other languages. this can take several forms depending on the training structure. in an i.i.d. context, where all data are available from the start, we think of transfer in terms of joint training. if training on language i and j jointly (multilingual) yields better performance on j than training only on j (monolingual), then there is transfer from i to j. however, continual learning adds a different dimension. indeed, when training on a language sequence we can identify two types of transfer: forwards and backwards @xcite . forward transfer denotes the performance and learning efficiency improvement on a given language thanks to previously acquired knowledge of other languages. conversely, backward transfer denotes the performance improvement on a previously acquired language when learning a new one. more formally, and similarly to @xcite , given a sequence of l languages, we define the performance matrix @xmath1, where p ij is the performance of language i after learning language j. in this context, backward transfer of i is defined as: negative backward transfer is also called forgetting, as it denotes performance loss on previous languages. since p 11 is equivalent to monolingual performance mono 1 , we can define backward transfer of the first language after learning language j: conversely, we define forward transfer as: ## cross-lingual transfer does transfer exist during continual training or does catastrophic forgetting prevent it? before studying the continual learning scenario, we first measure transfer when training the model on all languages at once (i.e. joint transfer). then, having this frame of reference, we investigate transfer when training the model on each language sequentially (i.e. continual transfer). ## training sequence how is transfer affected by the training sequence? in order to better understand the effect of the training sequence on transfer, we first look at measures of forward transfer at each position relative to monolingual and multilingual. secondly, we study the impact of the training sequence length on backward transfer measured on the first language. this analysis is conducted only on multiatis++ due to time and computational constraints. in the figures of this section, the mean, median and percentiles do take into account eventual outlier languages, while the minimum and maximum do not. when considering forward transfer, figure 3a shows that apart from the first position (equal to monolingual), the model consistently benefits from transfer at any point in the sequence, as performance is higher than monolingual. interestingly, due to some outlier languages (generally hindi and turkish), we observe that the means are poor estimates of the distribution when measuring ft mono i . this is an indicator that commonly used continual transfer metrics might over-or underestimate real performance when transfer is not uniformly distributed among languages. indeed, these metrics usually consist of averages across the adaptation axis @xcite . in figure 3b , we also observe that performance gets closer to multilingual as the sequence advances, although it rarely outperforms it. as per backward transfer, figure 4 shows that performance of the first language is in general worse than monolingual for any given sequence length. in particular, we observe that performance loss is not strictly monotonic, which means that measuring forgetting between the beginning and the end of the sequence may not be sufficient to explain how the model forgets. note that a sequence of @xmath2. furthermore, as hinted by continual experiments from our results indicate that the preserved knowledge that facilitates the acquisition of a new language in multilingual bert for slot filling is not the same knowledge that preserves previous language performance. this might be explained by a progressive shift of model parameters towards a better multilingual initialization for the atis task that might however fail to retain the specificities of previous figure 3 : distributions of forward transfer on test i relative to monolingual and multilingual for different positions i in the sequence. we average over 54 sequences and 5 runs. note that forward transfer is 0 when performance is equal to (a) monolingual and (b) multilingual. outliers not shown for readability. for different sequence lengths j. we average across 54 sequences and 5 runs. note that bt 1j = 0 if performance is equal to monolingual. outliers not shown for readability. ## fast recovery can lost performance due to forgetting be recovered? given that forward transfer does not seem to be affected by forgetting, we investigate in this section whether performance lost as a result of forgetting can be recovered quickly after continual training. the ability to recover is especially interesting for multiconer where forgetting is pretty high, but we still conduct experiments on both corpora. to investigate if this is possible, we first set out to discover whether the model shifts towards a better multilingual initialization. hence we compare the multilingual performance of the initial model 0 (consisting of bert and a random classifier) against model l , the model at the end of training sequence (e.g. model 4 in figure 1 ). in particular, we train both mod-els on all languages jointly for different numbers of epochs and evaluate on each language. notice that model l comes from our continual p 1l experiments (see the comparison between model 0 multilingual and model l multilingual for both corpora shows two interesting results. on one hand, we observe that even one epoch of multilingual training for model l achieves better performance than the monolingual baseline (model 0 monolingual) and is even close to the multilingual topline (model 0 multilingual) foot_3 , both of which are trained on the maximum number of epochs (50 or 15). this means that model l is capable of achieving good multilingual performance with very little training, hence canceling the effect of forgetting. on the other hand, we see that model l multilingual performance is greatly superior to model 0 multilingual with a single training epoch. this is not surprising given that the classifier is initialized randomly in model 0 , but it shows that the model is capable of retaining knowledge from previous languages, although it is not clear whether that knowledge is preserved in the classifier or in bert. we dive deeper into this question by training model l with a random classifier in the same manner (see model l + rnd clf multi. in overall, these results lead us to think that for the sequence labeling task, continual training over the language sequence does indeed shift model parameters to a better multilingual initialization. as a result, we explore the possibility to leverage this phenomenon in order to quickly recover lost language specificities due to forgetting for both corpora. to do this, we train model l on the first language of the sequence a second time (i.e. as if it were an (l + 1) th language) and evaluate on the first language only. as shown in tables 4 and 5 , when comparing model l monolingual to model 0 monolingual (equal to first language performance p 11 ), we see that the performance of the first language can be recovered and improved upon with as little as a single training epoch 3 . these results are outstanding for multiconer considering the high forgetting that we previously observed. on mul-tiatis++, model l monolingual even achieves 50epoch model 0 multilingual performance in most cases after only one epoch, with the remaining languages still showing a big improvement. in particular, hindi and turkish improve an absolute 3.9% and 7.8% from model 0 monolingual respectively. note that for multiatis++ increasing the number of recovery epochs for the first language does not bring considerable improvements. the only exception to this observation is turkish, which might be explained by the small size of its training set. in multiconer however, performance still improves after 5 epochs, getting closer to the multilingual topline. surprisingly, model l monolingual is even on par with the multilingual topline for turkish and chinese. although the cost of adding a language remains o(n ), the ability to recover all languages raises costs to o(ln ), making it expensive to use in practice. the design of a strategy taking full advantage of these recovery capabilities to limit forgetting with lower cost is left for future work. ## discussion to summarize, we observe a high level of crosslingual transfer in the i.i.d. setting when learning the sequence labeling task on all languages jointly for both corpora. in a real low resource scenario where data and annotations are scarce, it may be difficult or even impossible to implement either a monolingual or multilingual adaptive approach, as time/space complexity is high and not all languages might be available at once. in a continual learning setting where languages are learned in sequence, these costs are the lowest and cross-lingual transfer is retained in the form of forward transfer. however, forgetting occurs for the first language of the sequence since performance consistently drops below monolingual. when looking at continual cross-lingual transfer across the entire sequence, we obtain two surprising results. first, commonly used continual transfer metrics may not be a reliable estimate of the performance distribution across languages when transfer is not evenly distributed. since even in other adaptation axes a considerable variability across datasets is to be expected, we believe a statistic like the median might be a better choice, as we believe it better represents expected performance at any given point. second, as the sequence progresses, forward transfer improves, while backward transfer diminishes. this might indicate that model parameters remain a good initialization for future languages but that previous language specificities might be lost. motivated by this hypothesis, we compare the model at the beginning and at the end of the training sequence. our results suggest that knowledge from past languages is mostly stored in bert (as opposed to the task-specific classifier) and that the model may indeed shift towards a better multilingual initialization, making it suitable to quickly recover the performance lost as a result of forgetting. we then measure the recovery capabilities of the model with respect to the first language of the sequence. we empirically show that lost performance can be recovered with as little as a single training epoch even if forgetting is high (like in multiconer). performance can even greatly improve and approach the i.i.d. multilingual topline after only one training epoch for multiatis++ and 5 epochs for multiconer. in light of the above, we believe that effective continual learning methods for this task would benefit from leveraging recovery capabilities (either for a single language or many languages jointly) to limit the effect of forgetting, while preserving or even boosting forward transfer. ## conclusion in this paper, we presented an analysis of crosslingual transfer in continual learning for the sequence labeling task using multilingual bert @xcite as well as the multiatis++ @xcite and multiconer @xcite corpora. our main finding suggests that although forgetting is present, cross-lingual transfer is retained in the form of forward transfer, which allows the model to have substantial recovery capabilities. moreover, we empirically show that: 1) high forward transfer is linked to a progressive shift of model parameters towards a better multilingual initialization, and 2) that most knowledge from past languages is stored in the word representation encoder (bert) and not in the task-specific classifier. finally, we also find that current continual learning metrics may need to be adapted if we want to better estimate the distribution of transfer across the adaptation axis. as future work, we would like to reduce training costs by leveraging fast recovery for continual learning across languages. another interesting research direction would be a study on the continual acquisition of languages not already present in multilingual bert.