Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020)

Abstract: We frame the task of machine translation evaluation as one of scoring machine translation output with a sequence-to-sequence paraphraser, conditioned on a human reference. We propose training the paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot translation task (e.g., Czech to Czech). This results in the paraphraser’s output mode being centered around a copy of the input sequence, which represents the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and does not require human judgements for training. Our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT 2019 segment-level shared metrics task in all languages (excluding Gujarati where the model had no training data). We also explore using our model for the task of quality estimation as a metric—conditioning on the source instead of the reference—and find that it significantly outperforms every submission to the WMT 2019 shared task on quality estimation in every language pair.

## introduction machine translation (mt) systems have improved dramatically in the past several years. this is largely due to advances in neural mt (nmt) methods, but the pace of improvement would not have been possible without automatic mt metrics, which provide immediate feedback on mt quality without the time and expense associated with obtaining human judgments of mt output. however, the improvements that existing automatic metrics helped enable are now causing the correlation between human judgments and automatic metrics to break down @xcite especially for bleu @xcite , which has been the de facto standard for example, the mt system output "hi world" conditioned on the human reference "hello world" is found to have token probabilities [0.3, 0.6]. we train a single model in 39 languages and show that it: we release our model, metrics toolkit, and preprocessed training data. 2 2 related work mt metrics early mt metrics like bleu @xcite and @xcite use token-level n-gram overlap between the mt output and the human reference. overlap can also be measured at the character level @xcite or using edit distance @xcite . many metrics use word-and/or sentencelevel embeddings, including reval @xcite , ruse @xcite , @xcite , and esim @xcite . meant @xcite and meant 2.0 @xcite measure similarity between semantic frames and role fillers. state-of-the-art methods including yisi @xcite and bertscore @xcite rely on contextualized embeddings @xcite trained on large (non-parallel) corpora. bleurt @xcite applies fine tuning of bert, including training on prior human judgements. in contrast, our work exploits parallel bitext and doesn't require training on human judgements. ## method we propose using a paraphraser to force-decode and estimate probabilities of mt system outputs, conditioned on their corresponding human references. let p(y t |y @xmath0, x) be the probability our paraphraser assigns to the t th token in output sequence y, given the previous output tokens y @xmath1. let sys denote an mt system output, ref denote a human reference, and src denote the source. we expect scoring sys conditioned on ref to be most indicative of the quality of sys. however, we also explore scoring ref conditioned on sys as we find qualitatively that output sentences which drop some meaning conveyed by the input sentence are penalized less harshly by the model than output sentences which contain extra information not present in the input. scoring in both directions to penalize the presence of information in one sentence but not the other is similar, in spirit, to methods which use bi-directional textual entailment as an mt metric @xcite . 4 we postulate that the output sentence that best represents the meaning of an input sentence is, in fact, simply a copy of the input sentence, as precise word order and choice often convey subtle connotations. as such, we seek a model whose output distribution is centered around a copy of the input sentence, which we denote a "lexically/syntactically unbiased paraphraser." while a standard generative paraphraser is trained to retain semantic meaning, it does not meet our criteria because it is simultaneously trained to produce output which is lexically/syntactically different than its input, a key element in generative paraphrasing @xcite . we propose using a multilingual nmt system as a lexically/syntactically unbiased paraphraser. a multilingual nmt system consists of an encoder which maps a sentence in to an (ideally) languageagnostic semantic representation, and decoder to map that representation back to a sentence. the model has only seen bitext in training, but we propose to treat paraphrasing as a zero-shot "translation" (e.g., czech to czech). because our model is multilingual, we can also score mt system output conditioned on the source sentence instead of the human reference. this task is known as "quality estimation (qe) as a metric," and was part of the wmt19 qe shared task @xcite . we use "prism-ref" to denote our reference-based metric and "prism-src" to denote our system applied as a qe metric. our final metric and qe metric are defined based on results on our development set (see §5.2) as follows: to obtain system-level scores, we average segmentlevel scores over all segments in the test set. ## experiments we train a multilingual nmt model and explore the extent to which it functions as a lexically/syntactically unbiased paraphraser. we then conduct several preliminary experiments on the wmt18 mt metrics data @xcite to determine how to best utilize the token-level probabilities from the paraphraser, and report results on the wmt19 system-and segment-level metric tasks @xcite and qe as a metric task @xcite . ## analysis and discussion how helpful are human references? the fact that our model is multilingual allows us to explore the extent to which the human reference actually improves our model's ability to judge mt system output, compared to using the source instead. the underlying assumption with any mt metric is that the work done by the human translator makes it easier to automatically judge the quality of mt output. however, if our model or the mt systems being judged were strong enough, we would expect this assumption to break down. finally, with the exception of de-cs discussed above, we see statistically significant improvements for prism-ref over prism-src both into english (where human judgments were referencebased) and into non-english (where human judgments were source-based). this suggests that the high correlation of prism-ref with human judgements is not simply the result of reference bias @xcite . does paraphraser bias matter? our lexically/syntactically unbiased paraphraser tends to outperforms the generative english-only parabank 2 paraphraser, but usually not by a statistically significant margin. analysis indicate the lexi-cal/syntactic bias is only harmful in somewhat infrequent cases where mt systems match or nearly match the reference, suggesting it would be more detrimental with stronger systems or multiple references. our multilingual training method is much simpler than the alternative of creating synthetic paraphrases and training individual models in 39 languages, and our model may benefit from transfer learning to lower-resource languages. does fluency matter? despite nmt being very fluent, our results suggest that fluency is fairly discriminative, especially in non-english: lm scoring outperforms sentencebleu at segment-level correlation in 7/10 language pairs to non-english languages (excluding gujarati), for example. this is consistent with recent findings that lm scores can be used to augment bleu @xcite . lang bleu pair wmt19 best multilingual ∆ de-cs 20.1 † 21.8 ‡ +1.7 de-en 42.8 † 35.5 ‡ -7.3 de-fr 37.3 † 33.9 ‡ -3.4 en-cs 29.9 † 24.2 ‡ -5.7 en-de 44.9 † 38.1 ‡ -6.8 en-fi 27.4 † 21.9 ‡ -5.5 en-gu 28.2 † 0.0 ‡ -28.2 en-kk 11.1 † 8.6 ‡ -2.5 en-lt 20.1 † 15.0 ‡ -5.1 en-ru 36.3 † 28.1 ‡ -8.2 en-zh 44.6 † 30.1 ‡ -14.5 fi-en 33.0 † 26.2 ‡ -6.8 fr-de 35.0 † 26.4 ‡ -8.6 gu-en 24.9 † 0.4 ‡ -24.5 kk-en 30.5 † 27.7 ‡ -2.8 lt-en 36.3 † 28.5 ‡ -7.8 ru-en 40.1 † 36.1 ‡ -4.0 zh-en 39.9 † 20.6 ‡ -19.3 can we measure adequacy and fluency separately? the proposed method significantly outperforms the contrastive laser-based method in most language pairs, even when laser is augmented with a language model. this suggests that jointly optimizing a model for adequacy and fluency is better than optimizing them independently and combining after the fact-this is unsurprising given that neural mt has shown significant improvements over statistical mt, where a phrase table and language model were trained separately. can we train on monolingual data instead of bitext? the proposed method significantly outperforms scoring with the mbart auto-encoder, which is trained on large amounts of monolingual data, despite using substantially less compute power (1.3 weeks on 8 v100s for prism vs 2.5 weeks on 256 v100s for mbart). ## conclusion and future work we show that a multilingual nmt system can be used as a lexically/syntactically unbiased, multilingual paraphraser, and that the resulting paraphraser can be used as an mt metric and qe metric. our method achieves state-of-the-art performance on the most recent wmt shared metrics and qe tasks, without training on prior human judgements. we release a single model which supports 39 languages. to the best of our knowledge, we are the first to release a large multilingual nmt system, and we hope others follow suit. we are optimistic our method will improve further as stronger multilingual nmt models become publicly available. we compare our method to several contrastive methods and present analysis showing that we have not simply reduced the task of evaluation to that of building a state-of-the-art mt system; the work done by the human translator to create references helps the evaluation model to judge systems that are stronger (at translation) than it is. nothing in our method is specific to sentencelevel mt. in future work, we would like to extend prism to paragraph-or document-level evaluation by training a paragraph-or document-level multilingual nmt system, as there is growing evidence that mt evaluation would be better conducted at the document level, rather than the sentence level @xcite . mikel artetxe and holger schwenk. 2018. massively multilingual sentence embeddings for zeroshot cross-lingual transfer and beyond. arxiv preprint arxiv:1812.10464. satanjeev banerjee and alon lavie. 2005. meteor: an automatic metric for mt evaluation with improved correlation with human judgments. in proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, ann arbor, michigan. association for computational linguistics. colin bannard and chris callison-burch. 2005. paraphrasing with bilingual parallel corpora. in proceedings of the 43rd annual meeting of the association for computational linguistics (acl'05), pages 597-604, ann arbor, michigan. association for computational linguistics. rahul bhagat and eduard hovy. 2013. squibs: what is a paraphrase? computational linguistics, 39(3):463-472. ondřej bojar, rajen chatterjee, christian federmann, yvette graham, barry haddow, shujian huang, matthias huck, philipp koehn, qun liu, varvara logacheva, christof monz, matteo negri, matt post, raphael rubino, lucia specia, and marco turchi. 2017. findings of the 2017 conference on machine translation (wmt17). in proceedings of the second conference on machine translation, pages 169-214, copenhagen, denmark. association for computational linguistics. vishrav chaudhary, yuqing tang, francisco guzmán, holger schwenk, and philipp koehn. 2019. lowresource corpus filtering using multilingual sentence embeddings. in proceedings of the fourth conference on machine translation (volume 3: shared task papers, day 2), pages 261-266, florence, italy. association for computational linguistics. qian chen, xiaodan zhu, zhen-hua ling, si wei, hui jiang, and diana inkpen. 2017. enhanced lstm for natural language inference. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1657-1668, vancouver, canada. association for computational linguistics. julian chow, lucia specia, and pranava madhyastha. 2019. wmdo: fluency-based word mover's distance for machine translation evaluation. in proceedings of the fourth conference on machine translation (volume 2: shared task papers, day 1), pages 494-500, florence, italy. association for computational linguistics. william coster and david kauchak. 2011. simple english wikipedia: a new text simplification task. in proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 665-669, portland, ore-gon, usa. association for computational linguistics. michael denkowski and alon lavie. 2010. extending the meteor machine translation evaluation metric to the phrase level. in human language technologies: the 2010 annual conference of the north american chapter of the association for computational linguistics, pages 250-253, los angeles, california. association for computational linguistics. jacob devlin, ming-wei chang, kenton lee, and kristina toutanova. 2019. bert: pre-training of deep bidirectional transformers for language understanding. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171-4186, minneapolis, minnesota. association for computational linguistics. george doddington. 2002. automatic evaluation of machine translation quality using n-gram cooccurrence statistics. in proceedings of the second international conference on human language technology research, pages 138-145. daxiang dong, hua wu, wei he, dianhai yu, and haifeng wang. 2015. multi-task learning for multiple language translation. in proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pages 1723-1732, beijing, china. association for computational linguistics. markus dreyer and daniel marcu. 2012. hyter: meaning-equivalent semantics for translation evaluation. in proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 162-171, montréal, canada. association for computational linguistics. melania duma and wolfgang menzel. 2017. uhh submission to the wmt17 metrics shared task. in proceedings of the second conference on machine translation, pages 582-588, copenhagen, denmark. association for computational linguistics. sergey edunov, myle ott, marc'aurelio ranzato, and michael auli. 2020. on the evaluation of machine translation systems trained with back-translation. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 2836-2846, online. association for computational linguistics. andreas eisele and yu chen. 2010. multiun: a multilingual corpus from united nation documents. in lrec. anthony fader, luke zettlemoyer, and oren etzioni. 2013. paraphrase-driven learning for open question