Quality Estimation via Backtranslation at the WMT 2022 Quality Estimation Task (2022)

Abstract: This paper describes submission to the WMT 2022 Quality Estimation shared task (Task 1: sentence-level quality prediction). We follow a simple and intuitive approach, which consists of estimating MT quality by automatically back-translating hypotheses into the source language using a multilingual MT system. We then compare the resulting backtranslation with the original source using standard MT evaluation metrics. We find that even the best-performing backtranslation-based scores perform substantially worse than supervised QE systems, including the organizers’ baseline. However, combining backtranslation-based metrics with off-the-shelf QE scorers improves correlation with human judgments, suggesting that they can indeed complement a supervised QE system.

## introduction sophisticated approaches to mt quality estimation (qe) based on large pre-trained models and careful training regimen have enabled great progress in recent years. however, when using online mt systems, such qe technology is not yet available to users and backtranslation provides an appealingly simple strategy to estimate translation quality whether by humans or by automated sytems. lay users often rely on backtranslation to assess mt quality in languages that they do not understand @xcite . as a result, from a user experience standpoint, using backtranslation for qe is easy to explain. furthermore, with the increasing popularity of multilingual neural mt systems that can easily translate between multiple language pairs in any direction, backtranslations are very cheap to obtain, since they do not even require training an auxiliary mt system in the reverse translation direction. however, the effectiveness of backtranslation for estimating the quality of mt remains unclear. in early rule-based and statistical mt systems, @xcite shows that, when using automatic evaluation methods (e.g., bleu), backtranslation cannot discriminate good mt systems from bad ones, nor between texts that are easy or hard to translate. this led him to conclude that "round trip translation [is] good for nothing". recently, @xcite revisited the use of backtranslation for qe with neural systems for mt and with embedding-based similarity metrics to enable a more sophisticated comparison of the backtranslation with the source. they obtained strong results on the wmt 2019 qe task, outperforming the yisi-2 metric @xcite on system-level evaluations, but exhibited rather low correlations on the segment-level task which is more directly aligned with how humans use bt to gauge mt quality. the goal of our submission is to pitch a backtranslation-based qe score that can complement state-of-the-art quality estimation systems in the controlled settings of the wmt shared task @xcite and understand its reliability as a sentence-level quality estimation technique. ## approach following @xcite , given a source sentence x and a mt hypothesis, we translate y back into the source language using an off-the-shelf multilingual model m , yielding backtranslation x. we then compare x and x using standard machine translation evaluation metrics, and hypothesize that the distance between x and x, referred to as btscore(x, x), can be an indicative of the translation quality of y. however, mt systems are prone to making errors and are shown to hallucinate content. when the bt system makes an error, it can misguide the users in believing that the translation is a) erroneous when it is not and b) correct when the bt system magically recovers the source content. in order to improve the reliability of the bt-based qe ## official results using bt-based metrics we evaluate our approach on the english-czech sentence-level quality prediction subtask. as our approach is unsupervised, we do not use the training data provided by the organizers. we report results obtained on the development and test sets, using the pearson and spearman correlations with human judgments of quality. as can be seen in our results are complementary to @xcite in that they suggest that bt-based metrics might be better suited to ranking diverse outputs from systems of varying overall quality, than those from a single mt system, i.e. at predicting quality assessments at the segment level. while standalone evaluation using bt-based scoring significantly lags behind supervised sota qe baselines, we evaluate whether bt-based metrics can provide reliable complementary judgments to a supervised off-the-shelf qe scorer in in order to better understand the source of this improvement, we divide the en-cs development dataset into different buckets based on the direct assessment scores and report correlation on the resulsubsets in in ## conclusion we evaluated backtranslation-based unsupervised quality estimation systems on the sentence-level quality estimation task. our results show that backtranslation bases scorers fall substantially behind supervised models such as the organizers' baseline. however, they can complement off-the-shelf qe metrics in distinguishing bad translations. qualitative analysis on en-cs indicates that while backtranslation can be a poor indicator of translation quality, the automatic metrics derived using the source and the backtranslated source might also add to the unreliability of the scorer. this suggests that more investigation is needed to determine whether backtranslation can be used effectively for qe in practical systems, whether for automatic quality estimation or to provide quality feedback to human users.