Revisiting Round-trip Translation for Quality Estimation (2020)

Abstract: Quality estimation (QE), a task of evaluating the quality of translation automatically without human-translated reference, is one of the important challenges for machine translation (MT). As the QE methods, BLEU score for round-trip translation (RTT) had been considered. However, it was found to be a poor predictor of translation quality since BLEU was not an adequate metric to detect semantic similarity between input and RTT. Recently, the pre-trained language models have made breakthroughs in many NLP tasks by providing semantically meaningful word and sentence embeddings. In this paper, we employ the semantic embeddings to RTT-based QE metric. Our method achieves the highest correlations with human judgments compared to WMT 2019 quality estimation metric task submissions. Additionally, we observe that with semantic-level metrics, RTT-based QE is robust to the choice of a backward translation system and shows consistent performance on both SMT and NMT forward translation systems.

## introduction a good machine translation (mt) system converts one language to another while preserving the meaning of a sentence. given a pair of wellperforming translation systems between two languages, the meaning of a sentence should remain c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. ## rtt-based qe metrics given an input sentence @xmath0, x 2 , x 3 , ..., x n ) and a round-trip sentence @xmath1, x2 , x3 , ..., xm ), an rtt-based qe metric f is a scalar function computing the similarity of x and x. we consider the scalar output as a quality for the translation of x. the validity of f is assessed primarily by pearson correlation against the human judgments. previously, only surface-level similarity metrics were used for f . in this paper, we propose to use semantic-level metrics which can capture higherlevel concepts of the similarity. detailed implementations are described in the appendix a. ## experimental settings we compare rtt-based semantic-level qe metrics to lexical-level qe metrics in various conditions. initially, we prepare different bt systems to see the impact of the bt system to the performance change in rtt-based metrics. then, with a suitable bt system, we observe the proposed metrics on wmt 2019 metrics task evaluation dataset. we also examine whether our methods are biased to the certain type of ft system. furthermore, we investigate relations of the performance of rttbased qe metrics and their paraphrase detection ability. ## conclusions we have presented round-trip translation for translation quality estimation. it can be used for qe with suitable semantic-level similarity metrics like sbert @xcite and bertscore @xcite . rtt-sbert and rtt-bertscore are robust to the choice of a bt system, which alleviates the disadvantages of rtt being dependent on the bt system. moreover, both qe metrics significantly outperform the state-of-the-art qe metric, yisi-2. when the performance gap between the ft systems is large, rtt-sbert and rtt-bertscore provide comparable performance to bleu. they also perform well irrespective of the modeling architecture of ft systems. in future work, it would be interesting to investigate when rtt-based metrics become more reliable or unreliable. we find the high performance of rtt-sbert and rtt-bertscore is owing to sbert and bertscore's ability to detect paraphrased sentences. if better sentence similarity measurements appear, the performance of rtt-based metrics would increase as well. with the growing amount of the data and the advance of computing power, there certainly be a better measurement, thus rttbased qe metric is also promising.