Are we Estimating or Guesstimating Translation Quality? (2020)

Abstract: Recent advances in pre-trained multilingual language models lead to state-of-the-art results on the task of quality estimation (QE) for machine translation. A carefully engineered ensemble of such models won the QE shared task at WMT19. Our in-depth analysis, however, shows that the success of using pre-trained language models for QE is over-estimated due to three issues we observed in current QE datasets: (i) The distributions of quality scores are imbalanced and skewed towards good quality scores; (iii) QE models can perform well on these datasets while looking at only source or translated sentences; (iii) They contain statistical artifacts that correlate well with human-annotated QE labels. Our findings suggest that although QE models might capture fluency of translated sentences and complexity of source sentences, they cannot model adequacy of translations effectively.

## introduction quality estimation (qe) @xcite for machine translation is an important task that has been gaining interest over the years. formally, given a source sentence, s and a translated sentence, @xmath0, the goal of qe is to learn a function f such that f (s, t) returns a score that represents the quality of t, without the need to rely on reference translations. qe has many useful applications: qe systems trained to estimate human-mediated translation error rate (hter) @xcite can automatically identify and filter bad translations, thereby reducing costs and human post-editing efforts. industry players use qe systems to evaluate translation systems deployed in real-world applications. finally, qe can also be used as a feed-back mechanism for end-users who cannot read the source language. recently, language models pre-trained on large amounts of text documents lead to significant improvements on many natural language processing tasks. for instance, an ensemble of multilingual bert @xcite and xlm (conneau and lample, 2019) models @xcite won the qe shared task at the workshop on statistical machine translation (wmt19) @xcite , outperforming the baseline neural qe system @xcite ) by 42.9% and 127.7% on the english-german and english-russian sentence-level qe tasks respectively. while pre-trained language models contribute to tremendous improvements on publicly available benchmark datasets, such increases in performance beg the question: are we really learning to estimate translation quality? or are we just guessing the quality of the test sets? we performed a careful analysis which reveals that the latter is happening, given several issues with qe datasets which undermine the apparent success on this task: our results show that although qe systems trained on these datasets can capture fluency of the target sentences and complexity of the source sentences, they over-leverage lexical artifacts instead of modeling adequacy. from these findings, we conclude that qe models cannot generalize, and the successes in this task are over-estimated. ## methodology in this paper, we analyze three different instances of sample bias that are prevalent in qe datasets, which affect the generalization that models trained on them can achieve. ## discussion our results suggest that source sentences or translated sentences alone might already contain cues that correlate well with human-annotated scores in the qe datasets. given this, it seems highly unlikely that these qe models can capture interdependencies between source and translated sentences, which usually requires several levels of linguistic analysis. we hypothesize that qe models rely on either the complexity of source sentences or the fluency of translated sentences, but not on adequacy, to make their predictions. to test this, we create adversarial test sets across all language directions by randomly shuffling all source sentences and changing the hter scores to 1.0. a good model should be able to assign high hter scores to mismatched pairs. in ## conclusions and future work in this work, we presented our analysis of qe datasets used in recent evaluation campaigns. although recent advances in pre-trained multilingual language models significantly improve performances on these benchmark qe datasets, we highlight several instances of sampling bias embedded in the qe datasets which undermine the apparent successes of modern qe models. we identified (i) issues with the balance between highand low-quality instances (ii) issues with the lexical variety of the test sets and (iii) the lack of robustness to partial input. for each of these problems, we proposed recommendations. upon the submission of this paper, we implemented the proposed recommendations by creating a new dataset for quality estimation that addresses the limitations in current datasets. we collected data for six language pairs, namely two high-resource languages (english-german and english-chinese), two medium-resource languages (romanian-english and estonian-english), and two low-resource languages (sinhala-english and nepali-english). each language pair contains 10,000 sentences extracted from wikipedia and translated by stateof-the-art neural models, manually annotated for quality with direct assessment (0-100) by multiple annotators following industry standards for quality control. improving label diversity we selected language pairs with varying degrees of resource availability, which led to more diverse translation quality distributions (particularly for the mediumresource languages), mitigating the issue of imbalanced datasets, as shown in figure 2 .