Online Learning over Time in Adaptive Neural Machine Translation (2021)

Abstract: Adaptive Machine Translation purports to dynamically include user feedback to improve translation quality. In a post-editing scenario, user corrections of machine translation output are thus continuously incorporated into translation models, reducing or eliminating repetitive error editing and increasing the usefulness of automated translation. In neural machine translation, this goal may be achieved via online learning approaches, where network parameters are updated based on each new sample. This type of adaptation typically requires higher learning rates, which can affect the quality of the models over time. Alternatively, less aggressive online learning setups may preserve model stability, at the cost of reduced adaptation to user-generated corrections. In this work, we evaluate different online learning configurations over time, measuring their impact on user-generated samples, as well as separate in-domain and out-of-domain datasets. Results in two different domains indicate that mixed approaches combining online learning with periodic batch fine-tuning might be needed to balance the benefits of online learning with model stability.

## introduction machine translation (mt) quality has increased significantly in recent years, notably with the advent of modern neural machine translation (nmt) approaches @xcite . despite this progress, machine translated output requires post-editing in many cases, a process which is made more taxing when the same errors are repeated by mt systems segment after segment. to tackle this issue, adaptive approaches to machine translation aim to incorporate user feedback, oftentimes in post-editing scenarios @xcite ., * equal contribution. in this work, we study the evolution of online learning over time in a post-editing scenario, to determine optimal configurations in terms of both adaptation to continuous user input and model stability. for this purpose, we examine the behaviour of four different gradient-descent optimisers in two different domains, with varying learning rates, and evaluate their behaviour as the number of samples increases over time. to measure system responsiveness to user input over time, we evaluate online learning on dynamically increasing sets of samples formed by simulated user corrections. to determine model stability as online learning is performed, we also measure the quality of the mt models on static test sets pertaining to the domain at hand and on out-of-domain datasets, as additional measures of model evolution over time via online learning. additionally, we compare the best online learning variants to models trained via batch fine-tuning on accumulated user data. to our knowledge, this type of evaluation has not been previously explored and our results can support further work on online learning for nmt, as well as help practitioners in the field determine optimal configurations to design responsive and balanced adaptive mt systems. ## related work most studies of online learning for machine translation have taken place in the context of statistical machine translation (smt) @xcite . several methods have thus been proposed to adapt phrase tables and language models of smt models, in post-editing, interactive or streaming scenarios @xcite @xcite . evaluations of user productivity in adaptive smt have notably shown a significant overall reduction of effort in post-editing scenarios @xcite . in neural machine translation, comparatively fewer studies have been dedicated to online learning approaches. @xcite explore different strategies based on a posteriori integration of human post-edits, a priori adaptation by tuning to similar sentences in the training data, and a combination of both, showing substantial improvements over static models. @xcite compared smt and nmt models in interactive mt scenarios, demonstrating significant improvements in effort reduction with the latter over strong phrase-based systems. in @xcite , online training of nmt models is evaluated under both post-editing and interactive scenarios. similarly to the present work, they compared different optimisers with varying learning rates on datasets covering five different domains and different scenarios, assuming availability or lack of in-domain data prior to online learning. we complement their work in the present study by measuring the precise evolution of online learning over time and comparing it to batch fine-tuning at different time steps. 1 several user-centric studies have also demonstrated the usefulness of online learning for nmt, via analyses of user effort in static and adaptive environments @xcite @xcite . ## experimental setup in this section, we describe in turn the core components of our experiments, namely the selected corpora, the training modalities of the different types of nmt models, and the selected optimisers. 1 @xcite also include a scenario where online learning is applied over models first trained via batch fine-tuning on in-domain data, a setup which differs from our experiments. ## model stability over time as described in the previous sections, online learning can support post-editing by adapting to user corrections incrementally. this is obtained via relatively aggressive learning rates that enable updates to be significant on the basis of unique training figure 5 : results on all datasets after 100k samples with best@1 learning rates samples at each update point. this runs the risk of overfitting the model to the online training data, with incurred loss of quality on out-of-domain data, and crucially, on other portions of the in-domain data that did not undergo online training. to measure whether this is the case, we compared online learning with batch fine-tuning and the baselines, on all three types of datasets. for batch fine-tuning, the models were updated with batches that include all the online training pairs accumulated up to the current time step. figure 5 shows the results for ted and news-commentary at the 100k mark, when selecting the best learning rate for each optimiser based on the best bleu scores for the initial updates, which, as a reminder, were computed over random samples of 100 sentences. on ted, batch fine-tuning outperformed all variants of online learning on the indomain datasets, both static and dynamic, although only slightly over adagrad on the static id test set. on newscommentary, adagrad outperformed all variants on the dynamic id dataset, while also being only slightly under batch fine-tuning on the static id test set. however, as described in previous sections, this optimiser also performed significantly worse than sgd and rmsprop for initial updates, thus being less beneficial for initial stages of online learning. when compared to the most efficient optimisers for earlier online learning, namely sgd and rmsprop, batch fine-tuning would be the favoured option when reaching at least 100k data points. figure 6 presents the results after the final update stage when taking the best learning rates at the 100k mark for all optimisers. in this scenario, on the ted datasets all optimisers except adam feature results that are closer to those achieved via batch fine-tuning, although the latter obtained better results overall on both the dynamic and the static in-domain datasets. on newscommentary, sgd, rmsprop and adagrad significantly outperformed batch fine-tuning on the dynamic data, while the latter performed slightly better on the static id test set but with minor differences. among optimisers in online learning scenarios, sgd would be favoured when selecting more conservative learning rates, although both rmsprop and sgd would figure 6 : results on all datasets after 100k samples with best@100k learning rates also be favoured over batch fine-tuning in such a scenario for newscommentary. however, a more conservative learning rate also reduces its benefits at earlier stages, and the single case where online learning optimisers outperform batch fine-tuning at later stages might not be relevant in actual usage. in terms of out-of-domain data, batch fine-tuning performed better in all but one case, namely ted with the best learning rate for initial updates, where adagrad performed better. batch fine-tuning performed similarly to the baseline on newscommentary, which may be attributed to the relative proximity of newscommentary data to the generic training data, and conversely to the higher data difference between ted and the datasets that compose the generic training sets. ## conclusions in this paper, we explored the behaviour of online learning for neural machine translation over time, examining the results obtained with four different optimisers as the number of samples increases and evaluating translation model evolution after repeated network updates with different types of learning rates, from most aggressive to most conservative. we also compared online learning with batch fine-tuning on dynamic and static datasets, as well as out-of-domain test sets, to measure overall model stability. on the two domains we explored, based on ted and newscommentary data, there does not appear to be an optimal configuration, where online learning would be optimal in both the short and long term. sgd and rmsprop both feature a learning rate value which provides early benefits of online learning with relatively minor degradation over time, and might be viewed as the most balanced configuration in our experiments. however, at least in the domains we explored, batch fine-tuning was shown to be preferable at later stages in terms of model stability across dynamic, static and out-of-domain datasets. for practical adaptive nmt, it might thus be preferable to combine online learning, over limited time steps, with periodic batch fine-tuning over previous model checkpoints on the data accumulated over time.