O nline L earning Meets M achine T ranslation Evaluation: Finding the Best Systems with the Least Human Effort (2021)

Abstract: In Machine Translation, assessing the quality of a large amount of automatic translations can be challenging. Automatic metrics are not reliable when it comes to high performing systems. In addition, resorting to human evaluators can be expensive, especially when evaluating multiple systems. To overcome the latter challenge, we propose a novel application of online learning that, given an ensemble of Machine Translation systems, dynamically converges to the best systems, by taking advantage of the human feedback available. Our experiments on WMT’19 datasets show that our online approach quickly converges to the top-3 ranked systems for the language pairs considered, despite the lack of human feedback for many translations.

## introduction in machine translation (mt), measuring the quality of a large amount of automatic translations can be a challenge. automatic metrics like bleu @xcite remain popular due to their fast and free computations. yet, in the last few years we have seen that, as mt quality improves, automatic metrics become less reliable @xcite . for example, in the conference on machine translation (wmt)'19 news translation shared task, the winning system according to human annotators was not even in the top-5 according to bleu @xcite . on the other hand, using human assessments can be expensive, especially when evaluating multiple systems. in a real world scenario, given an arbitrary number of mt systems, one would need to evaluate them individually to find the best systems for a given language pair. however, that requires a considerable effort and there may not be enough human annotators to evaluate all the systems' translations. for instance, in the aforementioned wmt'19 shared task, many translations from the competing systems did not receive any human assessment. given an ensemble of competing, independent mt systems, how can we dynamically find the best ones for a given language pair, while making the most of existing human feedback? to address this question, we present a novel application of online learning to mt: each mt system in the ensemble is assigned to a weight, and the systems' weights are updated considering human feedback regarding the quality of their translations at each iteration. we use online learning algorithms with theoretical performance guarantees, under the frameworks of prediction with expert advice @xcite and multi-armed bandits @xcite . we contribute with an online mt ensemble that allows to reduce human effort by immediately incorporating human feedback in order to dynamically converge to the best systems foot_0 . our experiments on wmt'19 news translation test sets show that our online approaches indeed converge to the shared task's official top-3 systems (or to a subset of them) in just a few hundred iterations for all the language pairs experimented. moreover, it does so while coping with the aforementioned lack of human assessments for many translations, through the use of fallback metrics. ## online learning frameworks to provide some background on our proposal, we start by describing the online learning frameworks that we apply in this paper: prediction with expert advice and multi-armed bandits. a problem of prediction with expert advice can be described as an iterative game between a fore-caster and the environment, in which the forecaster seeks advice from different sources (experts) in order to provide the best forecast @xcite . at each iteration t, the forecaster consults the predictions pj,t , @xmath0. . . j, made by a set of j weighted experts, in the decision space d. considering these predictions, the forecaster makes its own prediction, pf,@xmath1. at the same time, the environment reveals an outcome y t in the decision space y (which may not necessarily be the same as d). a well-established algorithm to learn the experts' weights in this framework is exponentially weighted average forecaster (ewaf) @xcite . in ewaf, the prediction made by the forecaster is randomly selected following the probability distribution based on the experts' weights ω 1,t-1 . . . ω j,t-1 : pf,@xmath2,t-1 p j,t j @xmath3,t-1 . at the end of each iteration, the forecaster and each of the experts receive a non-negative loss based on the outcome y t revealed by the environment (ℓ f,t and ℓ j,t , respectively). the weight ω j,t of each expert @xmath4. . . j is then updated according to the loss received by each expert, as follows: if the parameter η is set to 8 log j t , it can be shown that the forecaster quickly converges to the performance of the best expert after t iterations @xcite . prediction with expert advice assumes that both the forecaster and all the experts receive a loss once the environment's outcome is revealed. however, this assumption may not always hold (i.e., there may not always be an environment's explicit feedback or a way to obtain the loss for all the experts). thus, we consider a related class of problems, multi-armed bandits, in which the environment's outcome is unknown @xcite . in this class of problems, one starts by attempting to estimate the means of the loss distributions for each expert (also known as arm) in the first iterations (the exploration phase), and when the forecaster has a high level of confidence in the estimated values, one may keep choosing the prediction with the smallest estimated loss (the exploitation phase). a popular online algorithm for adversarial multiarmed bandits is exponential-weighting for exploration and exploitation (exp3) @xcite . at each iteration t, the forecaster's action is randomly selected according to the probability distribution given by the weights of each arm j: in this framework, the forecaster is only able to measure the loss of the action it selects at each iteration, but it cannot measure the loss of other possible actions. thus, only the weight of the arm associated with this action is updated, as follows: both of these frameworks are relatively underexplored in nlp, despite their potential to converge to the best performing approach available in scenarios where feedback is naturally present. therefore, we propose to apply them in order to find the best mt models with little human feedback. ## machine translation with online learning in this work, we consider the following scenario as the starting point: there is an ensemble composed of an arbitrary number of mt systems; given a segment from a source language corpus, each system outputs a translation in the target language; then, the quality of the translations produced by each of the available systems is assessed by one or more human evaluators with a score reflecting their quality. we frame this scenario as an online learning problem under two different frameworks: (i) prediction with expert advice (using ewaf as the learning algorithm), and (ii) multi-armed bandits (using exp3 as the learning algorithm). the decision on whether to use one or another framework in an mt scenario depends on whether there is human feedback available for the translations outputted by all the available systems or only for the final choice of the ensemble of systems. mt 1 mt j mt 2 ω 1 ω 2 ω j ... ℓ j,t ℓ 2,t ℓ 1,t ... transl 1,t transl 2,t transl j,t ... human evaluators src t score 1,t score 2,t score j,t ... transl f,t score f,t ℓ f,t an overview of the online learning process is shown in fig. 1 , and can be summed up as follows. each mt system is an expert (or arm) @xmath5. . . j, associated with a weight ω j (all the systems start with same weights). at each iteration t, a segment src t is selected from the source language corpus and handed to all the mt systems. each system outputs a translation transl j,t in the target language, and one of these translations is selected as the forecaster's action according to the probability distribution given by the systems' weights (eq.1 for ewaf and eq. 3 for exp3). the chosen translation transl f,t (when using exp3) or the translations outputted by all the systems (when using ewaf) receive a human assessment score foot_1 score j,t , from which the loss ℓ j,t is derived for the respective mt system. finally, the weight of the chosen system or the weights of all the systems are updated as a function of the loss received, according to eq.4 (when using exp3) and eq.2 (when using ewaf), respectively (where ℓ j,@xmath6,t ). ## experimental setup to validate our proposal, we designed an experiment using data from an mt shared task. the main questions addressed by our experiment are: (i) whether an online learning approach can give a greater weight to the top performing systems for each language pair according to the shared task's official ranking, and (ii) if so, how quickly (i.e., how many translations need to be assessed by human evaluators in order to find the best system). below we detail the datasets used (section 4.1) and the feedback sources considered (section 4.2), as well as other experimental decisions (section 4.3). ## results and discussion in order to observe whether (and how soon) our online approach converges to the best systems, we report the overlap between the top @xmath7, 3 systems with greatest weights according to our approaches, ŝn , and the top @xmath8, 3 systems according to the shared task's official ranking, s * n , at specific iterations: we preferred this metric over a rank correlation metric, as we are focused on whether our online approach follows the performance of the best mt systems. in a realistic scenario (e.g., a web mt service), a user would most likely rely solely on the main translation returned, or would at most consider one or two alternative translations. moreover, due to the lack of a large enough coverage of human assessments, the scores obtained in the shared task are not reliable enough to discriminate between similarly performing systems. starting with en-de ( for fr-de ( ## conclusions and future work we proposed an online learning approach to address the issue of finding the best mt systems among an ensemble, while making the most of existing human feedback. in our experiments on wmt'19 news translation datasets, our approach converged to the top-3 systems (or a subset of them) according to the official shared task's ranking in just a few hundred iterations for all the language pairs considered (and just a few dozens in some cases), despite the lack of human assessments for many translations. this is a promising result, not only for the purpose of reducing the human evaluations required to find the best systems in a shared task, but also for any mt application that has access to an ensemble of multiple independent systems and to a source of feedback from which it can learn iteratively (e.g., web translation services).