Rethinking Round-Trip Translation for Machine Translation Evaluation (2023)

Abstract: Automatic evaluation methods for translation often require model training, and thus the availability of parallel corpora limits their applicability to low-resource settings. Round-trip translation is a potential workaround, which can reframe bilingual evaluation into a much simpler monolingual task. Early results from the era of statistical machine translation (SMT) raised fundamental concerns about the utility of this approach, based on poor correlation with human translation quality judgments. In this paper, we revisit this technique with modern neural translation (NMT) and show that round-trip translation does allow for accurate automatic evaluation without the need for reference translations. These opposite findings can be explained through the copy mechanism in SMT that is absent in NMT. We demonstrate that round-trip translation benefits multiple machine translation evaluation tasks: i) predicting forward translation scores; ii) improving the performance of a quality estimation model; and iii) identifying adversarial competitors in shared tasks via cross-system verification.

## introduction thanks to the recent progress of neural machine translation (nmt) and large-scale multilingual corpora, machine translation (mt) systems have achieved remarkable performances on high-to medium-resource languages @xcite @xcite . however, the development of mt technology on low-resource language pairs still suffers from insufficient data for training and evaluation @xcite . recent advances in multilingual pre-trained language model explore methods trained on monolingual data, using data augmentation and denoising auto-encoding @xcite figure 1 : given a corpus d a in language a, we are able to acquire the round-trip translation (rtt) results d ′ a and forward translation (ft) results d b via machine translation. one question was raised and discussed by machine translation community about two decades ago, "can rtt results be used to estimate ft performance?". while some early studies show the possibility @xcite , however some researchers tend to be against round-trip translation due to the poor correlations between ft and rtt scores. our work gives a clear and positive answer to the usefulness of rtt, based on extensive experiments and analysis. @xcite . however, high-quality parallel corpora are still required for evaluating translation quality. such requirement is especially resourceconsuming when working on i) hundreds of underrepresented low-resource languages @xcite @xcite and ii) translations for specific domains @xcite . standard mt evaluation requires parallel data which includes human translations as references, such that machine translations can be compared to the references with metrics such as bleu or chrf. in contrast, round-trip translation (rtt), as illustrated in figure 1 , instead uses a translation system to back-translate the machine translation into the source language, after which this roundtripped text can be compared to the original source (using standard reference-based metrics). this approach is compelling, in that it removes the requirement for parallel evaluation corpora, however influential work showed little correlation between evaluation scores measured using rtt versus standard reference-based evaluation @xcite @xcite , when applied to statistical machine translation (smt) and rule-based machine translation (rmt). consequently, the rtt method has seen little use, with a few recent notable exceptions in recent years, e.g., used to improve quality estimation methods @xcite @xcite . in this work, we revisit the dispute on the usefulness of rtt evaluation in the modern era of neural machine translation (nmt). we argue that the main reason for the negative findings was a consequence of historical systems using reversible rules in translation, notably copying, whereby systems copy unrecognized source tokens into target languages, which is often penalized in ft evaluation, but rewarded by rtt evaluation. we conduct extensive experiments to demonstrate the effect of the copying mechanism on smt. later, we illustrate strong correlations between ft-scores and rtt-scores on various mt systems, including nmt and smt without a copying mechanism. the finding sets the basis of using rtt-score for mt evaluation. three application scenarios in mt evaluation have been investigated to show the effectiveness of rtt-score. firstly, rtt-scores can be used to predict ft-scores by training a simple but effective linear regression model on several hundred language pairs. the prediction performance is robust in evaluating multiple mt systems in transferred domains and unseen language pairs including low-resource languages. then, rtt-scores are proved effective in improving the performance of a recently advanced quality estimation model which further supports the feasibility of rtt-score. finally, a cross-system check (x-check) mechanism is introduced to rtt evaluation for real-world mt shared tasks. by leveraging the estimation from multiple translation systems, x-check manages to identify those adversarial competitors, who know the mechanism of rtt evaluation and thus utilize the copying strategy as a shortcut to outperform other honest participants. ## related work reference-based machine translation evaluation metric. designing high-quality automatic evaluation metrics for evaluating translation quality is one of the fundamental challenges in mt research. most of the existing metrics largely rely on parallel corpora to provide aligned texts as references @xcite . the performance of the translation is estimated by comparing the system outputs against ground-truth references. a classic school of reference-based evaluation is based on string match methods, which calculate the matched ratio of word sequences as strings, such as bleu @xcite , chrf @xcite and ter @xcite . in addition, recent metrics utilize the semantic representations of texts to estimate their relevance, given pre-trained language models, such as bertscore @xcite and bleurt @xcite . these methods are demonstrated to be more correlated to human evaluation @xcite than string-based metrics. some other reference-based evaluation metrics require supervised training to work well @xcite on contextual word embeddings. while these automatic evaluation metrics are widely applied in mt evaluation, they are generally not applicable to low-resource language translation or new translation domains @xcite . our work demonstrates that referencefree mt metrics (rtt-score) could be used to estimate traditional reference-based metrics. reference-free quality estimation. in recent years, there has been a surge of interest in the task of directly predicting human judgment, namely quality estimation (qe), without access to parallel reference translations in the run-time @xcite @xcite . the recent focus on qe is mainly based on human evaluation approaches, direct assessment (da) and post-editing, where researchers intend to train models on data via human judgment features to estimate mt quality. among these recent qe metrics, learning-based models, yisi-2 @xcite , comet-qe-mqm @xcite , to name a few, demonstrate their effectiveness on wmt shared tasks. our work shows that rtt-score promotes a recently advanced qe model. the evaluation scores on round-trip translation (rtt-score) with regard to an automatic evalua- the main research question is whether ft-scores are correlated to therefore could be predicted by rtt-scores. ## predicting ft-score using rtt-score in this section, we validate whether ft-scores could be predicted by rtt-scores. then, we examine the robustness of the predictor on unseen language pairs and transferred mt models. ## rtt-scores for quality estimation in this section, we demonstrate that the features acquired by round-trip translation benefit quality estimation (qe) models. dataset. qe was first introduced in wmt11 @xcite , focusing on automatic methods for estimating the quality of neural machine translation output at run-time. the estimated quality should align with the human judgment on the word and sentence level, without accessing the reference in the target language. in this experiment, we perform sentence-level qe, which aims to predict human direction assessment (da) scores. we use da dataset collected from 2015 to 2021 by wmt news translation shared task coordinators. more details are provided in appendix d. ## conclusion this paper revisits the problem of estimating ft quality using rtt scores. the negative results from previous literature are essentially caused by the heavy reliance on the copying mechanism in # sys.