Findings of the First Shared Task on Lifelong Learning Machine Translation (2020)

Abstract: A lifelong learning system can adapt to new data without forgetting previously acquired knowledge. In this paper, we introduce the first benchmark for lifelong learning machine translation. For this purpose, we provide training, lifelong and test data sets for two language pairs: English-German and English-French. Additionally, we report the results of our baseline systems, which we make available to the public. The goal of this shared task is to encourage research on the emerging topic of lifelong learning machine translation.

## introduction lifelong learning can be defined as the ability to continually acquire new and retain previous knowledge. this ability characterizes humankind, but it is also reflected in several artificial intelligence systems @xcite . there are many challenges that have to be solved in order to achieve this goal of continual adaptation, among which catastrophic forgetting @xcite seems to be the most relevant. lifelong learning is very useful in the area of machine translation (mt), as it allows mt systems to adapt to new vocabularies and topics, and produce accurate translations across time. currently, there are no previous works that systematically try to solve the problem. this may be due to the lack of a benchmark to address the challenge @xcite . in this context, the main goal of the shared task on lifelong learning for mt is to develop systems that can self-adapt relying solely on domain expert data and are then freed from the necessity of machine learning expertise. what is more, this shared task also allows to investigate several mt research directions, such as: the continuous training/adaptation techniques; the preparation of additional pub-licly available corpora and evaluation sets; the active learning methods via a controlled simulated environment; the unsupervised adaptation of mt systems; the document-level approaches and the development and evaluation of mt systems across time. ## related work and tasks as mentioned in the introduction, there are not really any works in mt properly evaluating lifelong learning systems. however, there is a long history of studies in related tasks that are useful for addressing the lifelong learning objective. domain adaptation is based on the premise that the system can adapt to a target domain known in advance. this has been widely studied earlier for statistical mt e.g. @xcite and, more recently, for neural mt e.g. @xcite ). instance-based adaptation exploits similarity between training and inference instances @xcite , also in unsupervised scenarios @xcite . these studies have even led to the creation of adaptive mt commercial toolkits @xcite . importantly, in this task there is no target domain data available. unsupervised learning focuses on using monolingual corpora to train the translation system, without relying on any parallel corpora @xcite . active learning aims at selecting the most useful source sentences from a monolingual set and query their translation. this selection needs to minimize the post-edited cost and maximize the improvement of a finetuned model @xcite . interactive learning relies on a joint collaboration between a human and an mt system to obtain high-quality translations while reducing the human effort in the process @xcite . our lifelong learning setting differs from both domain and instance-based adaptation, as it depends on the target data (called the lifelong data). the lifelong data set, unlike the training data, is unsupervised. therefore, it is advisable to use techniques such as unsupervised learning, active learning, or interactive learning to approach the task. ## overview of the system and environment the toolchain developed to evaluate the autonomous systems is described in figure 1 . it is made of four parts: note that input datasets, user simulation and evaluation blocks are fixed and guarantee the reproducibility of the experiments. participants are free to edit the four blocks of the system in order to include their own code. once your code is included in this toolchain, the system will run automatically and the beat platform is responsible for managing the data exchanges between the different blocks of the architecture. thus, you don't need to take care about the communication between blocks, especially, the interaction between the system and the user simulation is automatic. ## baseline systems integrating an nmt system in the beat platform requires to rethink the code so that everything is done in memory. we chose to use the nmtpytorch toolkit to implement the baseline systems @xcite . our baseline systems consists of a 2-layer bidirectionnal gru @xcite encoder and a 2-layer conditional gru decoder @xcite equipped with an attention mechanism @xcite as implemented in nmtpytorch. given a source sequence of embeddings @xmath0, . . . , x s } and a target sequence of embeddings @xmath1, . . . , y t }, the bidirectional encoder first computes the sequence of annotations corresponding to the concatenation of the hidden states of the two gru @xmath2, . . . , a s }. at a given timestep t of decoding, the output layer estimates the probability of the next target word y t as follows: for a single training sample, we then maximise the joint likelihood of source and target sentences: ## adaptation techniques the first adaptation technique used is rather simple. it consists of selecting n sentences from training data that are the closest to the sentences in the document. the chosen similarity metric is the cosine between sentence embeddings obtained by a simple average of word embeddings, as described in @xcite . this data is then used to finetune the initial model for maximum 10 epochs with a learning rate of 0.00004, which is ten times smaller than during initial training of the model. furthermore, we employed an active learning strategy as an adaptation method. in principle, there are two steps involved. firstly, the model provides a translation for each document from the lifelong learning corpus. as the lifelong learning data are unsupervised; therefore, a quality estimation (qe) technique is used to evaluate the quality of the translations without any access to a reference translation. every document is evaluated using sentence-level hter scores @xcite . secondly, an openkiwi qe model @xcite is used to rank the sentences according to their quality, and those with the worst hter score are sent to the user simulation (active learning), which provides the correct translation of the selected sentences. this process implies the penalisation of the bleu score as explained in section 3. ## experimental setup the dimensions of embeddings and gru hidden states are set to 128 and 256, respectively. the embeddings are shared in the decoder (press and wolf, 2017). we use adam (kingma and ba, 2014) as the optimiser and set the learning rate and mini-batch size to 0.0004 and 64, respectively. regularisation is done by means of a weight decay of 1e-5 and the use of dropout on the embeddings, the source context and the output (set at 0.4) @xcite . we clip the gradients if the norm of the full parameter vector exceeds 1 @xcite . the data is processed by a joint bpe model with 30k merge operations @xcite . this leads to respectively 20.7k and 25.1k units for english and french and 17.2k and 26.5k units for english and german, respectively. we train each model for a maximum of 100 epochs and early stop the training if validation bleu @xcite does not improve for 10 epochs (figure 2 ). we also halve the learning rate if no improvement is obtained for three epochs. the number of learnable parameters is around 8.7m for en-fr and 8.5m for en-de. ## results the results of the baseline systems and the adapted ones can be found in results show that a simple data selection method along with finetuning can provide a small improvement of the system's performance for english to french. german is known to be a more complicated language, as demonstrated by the lower results and the inefficient effect of the adaptation method. ## discussion and next year evaluation we can see that the task, given the very constrained data is very hard. a simple comparison with the results of the systems that participated in the wmt14 news translation task shows more than 10 bleu points difference. we insist on the fact that the main goal of the challenge is to provide new methods to incrementally adapt the model to incoming documents. without loss of generality, it is very probable that even with a better baseline system (trained on more data), the adapted models would exhibit a similar improvement. many questions and challenges remain open as to how lifelong learning for mt should be implemented. next year, we ought to push further the evaluation by improving the qe model in order to better select the sentences to be sent to the user simulation (active learning module). hence, this will require to reconsider how the systems are evaluated. this year, we introduced a way of penalising the systems but without corresponding results. we hope to have more participants bringing new ideas either by using the current baseline models (and avoiding the integration burden) or by integrating their own systems into the platform.