Understanding the Properties of Minimum B ayes Risk Decoding in Neural Machine Translation (2021)

Abstract: Neural Machine Translation (NMT) currently exhibits biases such as producing translations that are too short and overgenerating frequent words, and shows poor robustness to copy noise in training data or domain shift. Recent work has tied these shortcomings to beam search – the de facto standard inference algorithm in NMT – and Eikema & Aziz (2020) propose to use Minimum Bayes Risk (MBR) decoding on unbiased samples instead. In this paper, we empirically investigate the properties of MBR decoding on a number of previously reported biases and failure cases of beam search. We find that MBR still exhibits a length and token frequency bias, owing to the MT metrics used as utility functions, but that MBR also increases robustness against copy noise in the training data and domain shift.

## introduction neural machine translation (nmt) currently suffers from a number of issues such as underestimating the true length of translations @xcite @xcite , underestimating the probability of rare words and over-generating very frequent words @xcite , or being susceptible to copy noise in the training data @xcite . in out-of-domain translation, hallucinations (translations that are fluent but unrelated to the source) are common @xcite @xcite . previous work has addressed these problems with decoding heuristics such as length normalization @xcite , data cleaning @xcite or model regularization @xcite @xcite @xcite . recently, eikema and aziz (2020) have highlighted the role of the decision rule, namely searching for the highest-scoring translation, and have argued that it is at least partially to blame for some of these biases and shortcomings. they found that sampling from an nmt model is faithful to the training data statistics, while beam search is not. they recommend the field look into alternative inference algorithms based on unbiased samples, such as minimum bayes risk (mbr) decoding. we believe mbr has potential to overcome several known biases of nmt. more precisely, if a bias can be understood as being caused by the modeseeking nature of beam search then we hypothesize that mbr could exhibit less bias. we view short translations, copies of the source text and hallucinations as hypotheses that are probable, but quite different to other probable hypotheses. if such pathological hypotheses are in a pool of samples, it is unlikely that mbr would select them as the final translation. while @xcite compare the statistical properties of samples and beam search outputs, and show that mbr can perform favourably compared to beam search according to automatic metrics, our paper aims to perform a targeted study of mbr and its properties, specifically its effects on the biases and shortcomings discussed previously. in our experiments we find that ## motivation for experiments we hypothesize that mbr decoding is useful for a certain class of failure cases encountered with beam search. namely, if an incorrect translation from beam search can be characterized as a hypothesis that is likely but fairly different from other hypotheses with similar probability, then mbr is expected to improve over beam search. several known deficiencies of nmt systems outlined in section 2.2 belong to this class of beam search failures. for instance, length bias occurs when a beam search translation is shorter than other hypotheses with comparable probability. likewise, translations that are copies of the input sentence or hallucinations (translations that are fluent, but unrelated to the input) can be avoided with mbr if they are not common in a pool of samples. finally, we study the skewedness of token frequencies in translations. eikema and aziz (2020) study lexical biases in nmt models, showing that model samples have higher agreement with the training distribution than map output. we investigate whether this is also true for mbr decoding, focusing on the well-known bias towards frequent tokens. ## length bias we evaluate mbr decoding with different utility functions. there is no single utility function which performs best on all evaluation metrics. instead, any of our evaluation metrics can be optimized by choosing a closely related utility function (see figure 2 and appendix d). for instance, chrf-2 as the utility function leads to the best chrf2 evaluation scores. ## token frequency bias beam search overgenerates tokens that are very common in the training data and undergenerates rare tokens (see section 2.2). sampling on the other hand assigns correct probabilities to common and rare tokens. given that mbr is based on samples, does it share this property with sampling? in figure 3 we show that this is not the case. although the skewedness of probabilities is less severe for mbr than for beam search, mbr still assigns too high a probability to frequent events. a reason for this is that our utility functions are based on surface similarity between samples, so rare tokens, which will be sampled rarely, will thus also have low utility. unfortunately, there is a trade-off between correct probability statistics for very common and very rare words and translation quality. the most faithful statistics can be obtained from sampling, but sampling leads to the worst overall translation quality. ## domain robustness in general, as the number of samples grows, mbr approaches but does not outperform beam search on our in-domain data (see figure 1 ). on our outof-domain data, the gap between mbr and beam search is smaller. we hypothesize that mbr may be useful for out-of-domain translation. we evaluate mbr on a domain robustness benchmark by @xcite . figure 4 shows that on this benchmark mbr outperforms beam search on 2 out of 4 unknown test domains. a possible reason why mbr is able to outperform beam search in unknown domains is that it reduces hallucinated translations. to test this hypothesis, we define a hallucination as a translation that has a chrf2 score of less than 0.01 when compared to the reference, inspired by @xcite . given this definition of hallucination, figure 5 shows that on average, mbr assigns a lower utility score to hypotheses that are hallucinations. similarly, mbr reduces the percentage of hallucinations found in the final translations, compared to beam search or sampling. to summarize, we find that mbr decoding has a higher domain robustness than beam search. ## impact of copy noise in the training data if copies of source sentences are present on the target side of training data, copies are overrepresented in beam search (section 2.2). here we test whether mbr suffers from this copy bias as well. we create several versions of our training sets where source copy noise is introduced with a proba2020 ). the medical test set is in-domain, the remaining sets are out-of-domain. bility between 0.1% and 50%. as shown in figure 6 , mbr and beam search are comparable if there are few copies in the training data. however, if between 5 and 25% of all training examples are copies, then mbr outperforms beam search by a large margin (> 10 bleu for arabic-german). as further evidence for the ability of mbr to tolerate copy noise we present an analysis of copies in figure 7 . we define a copy as a translation with a word overlap with the reference of more than 0.9. we show that mbr assigns a much lower utility to copy hypotheses than to all hypotheses taken together. in the final translations, mbr manages to reduce copies substantially. for instance, if around 10% of the training examples are copies, beam search produces around 50% copies, while mbr reduces this number to below 10%. we conclude from this experiment that mbr is more robust to copy noise in the training data. we acknowledge that this setting is artificial because copy noise can easily be removed from data sets. nonetheless, it is a striking example of a known shortcoming of nmt systems usually attributed to the model or training procedure, when in fact beam search is at least partially to blame. in our experiments, mbr did not generally outperform beam search according to automatic metrics, but we still deem it a promising alternative to map decoding due to its robustness. for future work, we are interested in exploring more sophisticated similarity metrics to be used as utility functions, including trainable metrics such as comet @xcite , and investigating how these utility functions affect the overall quality and biases of translations. ## note on reproducibility we will not only release the source code used to train our models (as is common in nlp papers at the moment), but a complete pipeline of code that can be run on any instance in a fully automated fashion. this will allow to reproduce our results, including the graphs and tables shown in this paper, in a consistent way with minimal changes. we encourage the community to attempt to reproduce our results and publish the results. a data set details iso3 abbreviation language pair size scripts dan-epo danish-esperanto 110k roman-roman aze-eng azerbaijani-english 680k roman -roman bel-rus belarusian-russian 70k cyrillic-cyrillic deu-fra german-french 47m roman-roman eng-mar english-marathi 370k roman-devanagari ara-deu arabic-german 12m arabic-roman deu-eng german-english 1m roman-roman