Cross-lingual Continual Learning (2023)

Abstract: The longstanding goal of multi-lingual learning has been to develop a universal cross-lingual model that can withstand the changes in multi-lingual data distributions. There has been a large amount of work to adapt such multi-lingual models to unseen target languages. However, the majority of work in this direction focuses on the standard one-hop transfer learning pipeline from source to target languages, whereas in realistic scenarios, new languages can be incorporated at any time in a sequential manner. In this paper, we present a principled Cross-lingual Continual Learning (CCL) evaluation paradigm, where we analyze different categories of approaches used to continually adapt to emerging data from different languages. We provide insights into what makes multilingual sequential learning particularly challenging. To surmount such challenges, we benchmark a representative set of cross-lingual continual learning algorithms and analyze their knowledge preservation, accumulation, and generalization capabilities compared to baselines on carefully curated datastreams. The implications of this analysis include a recipe for how to measure and balance different cross-lingual continual learning desiderata, which go beyond conventional transfer learning.

## introduction with more than 7,000 languages spoken around the globe, downstream applications still lack proper linguistic resources across languages @xcite , necessitating the use of transfer learning techniques that take advantage of data that is mismatched to the application. in an effort to simplify architecture complexity and energy consumption, it is desirable to unify multi-lingual performance into a single, parameter-and memory-constrained model, and to allow this model to evolve, learning on multi-lingual training data as it becomes available without having to pre-train or fine-tune from scratch. such is the longstanding goal of language figure 1: an overview of ccl: we use an example of a non-stationary datastream moving from high to low resource languages. each bold and dashed box represents either a training or test data instance being fine-tuned or evaluated on, respectively. to support this problem setup, we evaluate the cross-lingual capabilities of continual approaches. those capabilities include knowledge preservation on old languages, accumulation to the current language, and generalization to unseen languages at each point of the training. in addition to that, we evaluate model utility at the end of continual learning. most previous work that attempts to deal with the challenge of transfer exploitation and forgetting mitigation focuses on the problem of sequentially learning over different nlp downstream tasks or domains @xcite @xcite , rather than on language shifts. indeed, the current literature for learning over sequences of languages is rather scarce, and is mostly reduced to cross-lingual transfer learning between a pair of languages @xcite @xcite @xcite . @xcite language model and then fine-tune it on a downstream task in one of several different (child) languages. this conflates task and language transfer, and confuses analysis -the interference between the pre-trained language model 'task' and the fine-tuned task along with the parent and child languages cannot be disentangled. @xcite an adaptation scheme to each new language pair independently while retaining the translation quality on the parent language pairs. similarly, @xcite and @xcite propose lexical and semantic level techniques to adapt to target languages. however, all these mentioned works still focus on the 'one-hop' case, consisting of two steps: (1) training on initial parent language(s) (pairs), then (2) adapting to new children language(s) (pairs); the effect of multiple shifts in the datastream is not trivially generalizable to more than one hop. @xcite propose an approach for language-specific modules based on adapters and evaluate that on sequential streams of languages. however, they only focus on adapters and two desiderata of continual learning: interference mitigation and transfer maximization. we need a more robust and comprehensive fine-grained evaluation that balances the dynamics between different cross-lingual continual learning desiderata. in this paper, we pave the way for a more comprehensive multi-hop continual learning evaluation that simulates the sequential learning of a single task over a stream of input from different languages. this evaluation paradigm requires experimentation over balanced streams of n data scenarios for @xmath0. unlike previous work, this paper concretely defines the following comprehensive goals along with their evaluation metrics as guidelines for analyzing the cross-lingual capabilities of multilingual sequential training: knowledge preservation, accumulation, generalization, and model utility as shown in figure 1 . we apply our test bed to a sixlanguage task-oriented dialogue benchmark and comprehensively analyze a wide variety of successful continual learning algorithms, from previous literature investigated in continual learning contexts different from the cross-lingual context, including (a) model-expansion @xcite , (b) regularization @xcite , (c) memory replay @xcite , and (d) distillation-based approaches @xcite . our findings confirm the need for a multi-hop analysis and the effectiveness of continual learning algorithms in enhancing knowledge preservation and accumulation of our multilingual language model. we additionally demonstrate the robustness of different continual learning approaches to variations in individual data setup choices that would be misleading if presented in a traditional manner. our main contributions are: (1) we are the first to explore and analyze cross-lingual continual finetuning foot_0 across multiple hops and show the importance of this multi-hop analysis in reaching clearer conclusions with greater confidence compared to conventional cross-lingual transfer learning ( §4.1). ## cross-lingual continual learning in this section, we formally define cross-lingual continual learning, describe its goals and challenges, and introduce the downstream tasks, datastreams, and evaluation protocols used. although we are not the first to define or investigate continual learning for languages, we are to the best of our knowledge the first to define and study crosslingual continual learning where continual learning is focused on languages only. thus, we formally define cross-lingual continual learning as learning over a set of languages seen sequentially in multiple hops which is truer to the term of cross-lingual and continual learning, respectively. we distinguish that from 'cross-lingual cross-task cross-stage continual learning' which continually learns over a set of pretraining and downstream tasks sampled from different languages @xcite and 'cross-lingual one-hop transfer learning' @xcite . ## methods for our base model, we use the same m-bertbased architecture as was used in @xcite to jointly learn the intent classification and slot filling subtasks of mtop. 4 on top of that, we define baselines, noncontinual learning reference models, and continual learning algorithms. ## results & analysis in this section, we provide an extensive analysis in the form of different ablation studies. we ask critical analysis questions that revolve around the continual learning goals described in §2.2. for §4.2, scores are reported using accuracy (acc) and f1score (f1) for intent classification and slot filling, respectively. for the remaining sections, all results are reported for intent classification only, slot filling results, for which the same trends are observed, can be found in appendix d. bootstrap sampling (over test data shuffling) is used to compute the average and 95% confidence intervals (averaged over all language permutations except for §4.4). more details can be found in appendix c.3. we also separately repeat key experiments over 3 different seeds and obtain similar findings which can be found in appendix e. we decide to report the results using bootstrap sampling since they have tighter confidence intervals. ## conclusion we formulate the cross-lingual continual learning problem setup. we show that naive sequential finetuning is prone to catastrophic forgetting and has poor accumulation capabilities sensitive to different language permutations. we provide the first benchmark to compare the effectiveness of different continual learning algorithms for the cross-lingual case. we show that continual learning models improve cross-lingual knowledge preservation, which also contributes to improving final model performance and to a lesser degree accumulation and generalization. we also discuss the challenges of sequentially training for certain language permutations. we hope that this study will encourage more analyses in the same spirit to cover more benchmarks and datastream setups to gain more insights that go beyond conventional cross-lingual transfer learning.