Learning to Evaluate Translation Beyond E nglish: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020)

Abstract: The quality of machine translation systems has dramatically improved over the last decade, and as a result, evaluation has become an increasingly challenging problem. This paper describes our contribution to the WMT 2020 Metrics Shared Task, the main benchmark for automatic evaluation of translation. We make several submissions based on BLEURT, a previously published which uses transfer learning. We extend the metric beyond English and evaluate it on 14 language pairs for which fine-tuning data is available, as well as 4 “zero-shot” language pairs, for which we have no labelled examples. Additionally, we focus on English to German and demonstrate how to combine BLEURT’s predictions with those of YiSi and use alternative reference translations to enhance the performance. Empirical results show that the models achieve competitive results on the WMT Metrics 2019 Shared Task, indicating their promise for the 2020 edition.

## introduction the recent progress in machine translation models has led researchers to question the use of ngram overlap metrics such as bleu, which focus solely on surface-level aspects of the generated text, and thus may correlate poorly with human evaluation @xcite @xcite @xcite . this has led to a surge of interest for more flexible metrics that use machine learning to capture semantic-level information @xcite . popular examples of such metrics include yisi-1 @xcite , esim @xcite , bertscore @xcite , the sentence mover's similarity @xcite , and bleurt @xcite . these metrics utilize contextual embeddings from large models such as bert @xcite which have been shown to capture linguistic information beyond surface-level aspects @xcite . the wmt metrics 2020 shared task is the reference benchmark for evaluating these metrics in the context of machine translation. it tests the evaluation of systems that are to-english (@xmath0, which requires a multilingual approach. an additional challenge for learned metrics is that human ratings are not available for all language pairs, and therefore, the models must use unlabeled data and perform zero-shot generalization. we describe several learned metrics based on bleurt @xcite , originally developed for english data. we first extend bleurt to the multilingual setup, and show that our approach achieves competitive results on the wmt metrics 2019 shared task. 1 we also present several simple bert-based baselines, which we submit for analysis. finally, we focus on english to german and enhance bleurt's performance by combining its predictions with those of yisi @xcite as well as by using alternative references. ## background and notations task reference-based nlg evaluation seeks to assign a score to a triplet of sentences (input, reference, candidate), where input is a sentence in the source language, reference is a reference translation kept secret at inference time, and candidate is a translation produced by an mt system. similar to bleu @xcite and the previous editions of the wmt metrics shared task, we omit the input and treat the task as a regression problem : we aim to learn a function f : (x, x) → y that predicts a quality score y for a candidate sentence @xmath1, .., xp ) given a reference sentence @xmath2, .., x q ). the function is supervised on a corpus of n human ratings {(x i , xi , y i )} n @xmath3. bleurt most experiments presented in this paper are based on bleurt, a metric that leverages transfer learning to achieve high accuracy and increase robustness @xcite . bleurt is a bert-based regression model @xcite . it embeds sentence pairs into a fixed-width vector v @xmath4, x) with a pre-trained transformer, and feeds this vector to a linear layer: in its original (english) version, bleurt is trained in three stages. (1) it is initialized from a publicly available bert checkpoint. (2) the model is then "warmed up" by exposing it to millions of sentence pairs (x, x), obtained by randomly perturbing sentences from wikipedia. during this phase, the model learns to predict a wide range of similarity scores that include existing metrics (bertscore, bleu, rouge), scores from an entailment model, and the likelihood that x was generated from x with a roundtrip translation by a given translation model. we denote this stage as mid-training. (3) in the final stage, the model is fine-tuned on human ratings from wmt metrics @xcite @xcite , using a regression loss ℓ @xmath5. we found that english bleurt achieved competitive performance on four academic datasets, webnlg @xcite , and the wmt metrics shared task years 2017 to 2019. we experiment with two pre-trained multilingual models: mbert and mbert-wmt, a custom multilingual variant of bert. the mbert-wmt model is larger that mbert (24 transformer layers instead of 12), and it was pre-trained on 19 languages of the wmt metrics shared task 2015 to 2020. ## results tables 1 and 2 show the results in the @xmath6, at the segment-and system-level respectively. in the majority of cases, one of the bleurt configurations yields the strongest results. the original bleurt metric seems to perform better at the segment-level. at the system-level it may be dominated by prism (3 out of 7 language pairs) or by one of the simpler bert-based models (4 out of 7 language pairs). tables 3 and 4 present the results for the other languages. mbert-wmt yields solid results at the segment-level (it achieves the highest correlations for 7 out of 11 language pairs), in particular for the "zero-shot" setups, @xmath7, @xmath8, and @xmath9. it outperforms mbert consistently, except for @xmath10. the results are consistent at the system-level. ## additional improvements on @xmath11 for @xmath12, the organizers of wmt20 provide three different reference translations: two standard references and one additional paraphrased reference. given this novel setup, we investigate how to combine our predictions. moreover, we use a similar framework to ensemble the predictions of different metrics. in particular, we average the predictions of bleurt, yisi-1 and yisi-2. all three metrics are different in their approaches. while bleurt and yisi-1 are reference-based metrics, yisi-2 is reference- ). in the remainder of this section, we report bleurt results using the mbert-wmt setup unless specified otherwise. 3 ## summary we submit the following systems to the wmt metrics shared task: