Some Tradeoffs in Continual Learning for Parliamentary Neural Machine Translation Systems (2024)

Abstract: In long-term translation projects, like Parliamentary text, there is a desire to build machine translation systems that can adapt to changes over time. We implement and examine a simple approach to continual learning for neural machine translation, exploring tradeoffs between consistency, the model’s ability to learn from incoming data, and the time a client would need to wait to obtain a newly trained translation system.

## introduction there are many translation use cases in which translation is ongoing, i.e., new translations are being produced by translators. we experiment with approaches to using the flow of new data produced by translators to continually update neural machine translation (nmt) systems indefinitely. the intuition behind this is a desire to keep the system(s) upto-date and optimal for the task of translation over time. this could include new topics that are being discussed, changes to terminology, use of spelling variants, or changes in translator/translation preferences; we specifically highlight terminology in this work due to the ease of measurement. our planned continual learning approach is to build a baseline-initial system (trained on all data up to a fixed point in time) and iteratively finetune (continue training) it on chunks of new data, which we refer to as stages. this simulates the reallife scenario in which new translations are produced and then the goal is to use them to improve the qual-ity of the existing initial translation system. we use the term "continual learning" in this work, though we note that a number of different terms have been used more or less interchangeably to refer to this concept in the machine translation (mt) and machine learning literature. 1 in the context of mt, they refer to the idea of using the flow of new data to continue to train an mt system indefinitely, producing systems that are always up-to-date: that learn new terms, phrases and formulations, new concepts, changing translations (of old terms-interestingly, this suggests that while most old knowledge should be retained, some of it should be forgotten/overwritten), etc., as they appear. within this work we will refer to these approaches as continual learning (hereafter cl). the research question we address is whether a simple approach of regularly finetuning a base model works successfully as a cl approach to nmt, especially in a parliamentary setting, where the domain may evolve over time, but is not expected to suddenly change completely. we describe proof-of-concept experiments, in an idealized setting, but with real data, where we compare with several baselines. while our work examines a realistic use case, it is limited by our focus on one language pair (english-french) and one domain (parliamentary text). we examine tradeoffs with respect to questions of various measures of model performance, and note areas where more study is needed to determine the usefulness of these approaches. we also provide a brief discussion of the technical infrastructure that would be required to implement these approaches in practice, with considerations around feasibility and potential costs and risks. ## related work we provide a brief overview of related cl literature from machine learning (more broadly) and machine translation (more specifically). ## data setup the general data setup for our experiments is designed to mimic in a controlled fashion a real-world scenario in which there exists a backlog of professional translations, and an incoming sequence of additional new translation pairs that are generated each day by translators. translations are collected regularly and used to incrementally update the mt model, which is then used to produce future translations. we call each such collection period a "stage". all experiments are based on english-french or french-english data from the proceedings of the debates (a.k.a. "hansard") in the canadian house of commons, one of the two chambers of the canadian parliament. this data ranges in time from 2006-04-03 to 2023-09-29, is segmented into sentences, and is timestamped so it can be ordered sequentially, with 5-minute precision. 2 we explain here how we use this data for both hyperparameter-tuning experiments and full data experiments where we train and test the continual learning approach and several baselines: (1) a baseline-initial system (trained in a standard, non-cl, manner on all data up to an initial stopping point in time); (2) a baseline-final system (trained in a standard, non-cl, manner on all data up to a final stopping point in time); ## performance evaluation we are interested in three main types of performance evaluation: plasticity (improvement on new data), stability (maintaining high performance on past data), and volatility (whether the translations change dramatically or incrementally between models). the goal for cl is high plasticity, high stability (i.e., no catastrophic forgetting), and low volatility. we study plasticity on two data sets: the epilogue-test test set (used only for testing and never in training or parameter setting) and the sequence of intermediate test set stages. on the epilogue-test, we measure translation quality (using automatic metrics) of the output produced by the baseline-initial system, each incremental cl system, and the baseline-final or baseline-recency system. we can compare these scores directly. we also consider a stage-wise evaluation representative of real-life applications. for this evaluation, a stage is initially used as a test set, and then the cl system trains on it, testing on the subsequent stage, until reaching the epilogue; a stage is never again used for testing after it has been trained on. while we can compare the cl system with each of the baselines on each stage, we cannot directly compare the scores of the stages to one another (they are different test sets and most automatic metrics are not directly comparable across test sets). instead, we compare the difference in metric score between the baseline-initial system and any systems of interest. we draw an idealized line between 0 at the start of cl to the difference between baseline-final and baseline-initial (measured on the epilogue-test) at the end of cl. to measure stability (and whether catastrophic forgetting occurs), we use the held-out baselineinitial-test data set. if catastrophic forgetting occurs, we might expect to see major decreases in quality on this test, whereas if the cl approach is successful, we might expect maintained quality (or small variation; there may be changes in terminology or other such domain evolution features). this measurement will also be done with automatic mt metrics. we examine volatility during our hp search experiments using the hp-epilogue-test set. a highly volatile system would see major changes from system to system; a less volatile system would likely see most translations remain similar to one another. ## hyperparameter tuning experiments even using the simple cl approach we have selected, we need to set some hyperparameters. we limit these to the learning rate (lr) and the number of epochs. in this section we describe the setup of our hyperparameter tuning experiments. we describe baseline models built specifically for hp tuning in section 5.1, the tuning procedure in section 5.2, performance over time in section 5.3, and volatility in section 5.4. we note that for our hp tuning data set, we are considering a smaller range in time than in our full data set. ## full data experiments using the full experimental data set allows us to examine cl performance over a longer timespan and to see whether it remains consistent, 6 and whether we need to build in additional safeguards, e.g. the ability to roll back to an earlier model if a particular update degrades quality. we also examine the various measures of plasticity, stability, and volatility over a larger timespan. we build baseline models using the same configurations as the hp-baselines, but using the larger data set. for the baseline-recency model, we finetuned the baseline-initial model on all but the last stage of the cl data, using a lower initial lr and using the last stage of cl as a validation set for early stopping rather than an oracle. we begin with the same preliminary experiments that we produced for hp selection, which we describe in section 6.1. we then examine questions of novel vocabulary in section 6.2. ## conclusion in conclusion, we have demonstrated in simulation that a simple approach to cl can be effectively applied to parliamentary machine translation systems. we find that choosing a good set of hyperparameters enables us to build models with high levels of plasticity and stability, with low levels of volatility. however, we also demonstrate tradeoffs: time, plasticity, and performance. our approach provides incremental improvements over time, but a client who is willing to wait longer for a better system may prefer to opt for finetuning on a larger amount of recent data rather than these incremental improvements. similarly, we see a tradeoff between rapidly learning new vocabulary and the overall performance. while we have measured the success of our models using automatic metrics, future work will also be needed to have human translators provide evaluation and feedback.