Assessing Reference-Free Peer Evaluation for Machine Translation (2021)

Abstract: Reference-free evaluation has the potential to make machine translation evaluation substantially more scalable, allowing us to pivot easily to new languages or domains. It has been recently shown that the probabilities given by a large, multilingual model can achieve state of the art results when used as a reference-free metric. We experiment with various modifications to this model, and demonstrate that by scaling it up we can match the performance of BLEU. We analyze various potential weaknesses of the approach, and find that it is surprisingly robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities.

## introduction traditional automatic metrics for machine translation (mt), such as bleu @xcite , score mt output by comparing it to one or more reference translations. this has several disadvantages. first, high-quality reference translations are expensive to create. this means that in practice, evaluation is usually carried out with relatively small, carefully curated test corpora. the need for careful preparation limits the number of domains for which an mt system can be conveniently assessed, and small test-set sizes can make it difficult to draw robust conclusions @xcite . second, enshrining ground truth in a small number of references (usually just one) is inherently problematic, since valid translations can vary along many dimensions; @xcite demonstrate that different (correct) references for the same test set can result in different system rankings according to the same reference-based metric. finally, scoring the similarity between an mt hypothesis and a reference translation involves recognizing the extent to which they are mutual paraphrases. when gross discrepancies exist, this is a relatively easy problem for which surface-level metrics can provide a reliable signal, but capturing the subtle errors typical of high-quality mt is more difficult, and it is not clear whether it is substantially easier than scoring the similarity between texts in different languages. these problems can be avoided by looking only at the source text when assessing mt output. there is evidence that this is the best practice for human evaluation @xcite . moreover, it has recently been investigated for automatic metrics as well @xcite @xcite . such reference-free metrics are flexible and scalable, but since they are essentially performing the same task as an mt model, they raise a circularity concern: if we can reliably score mt output, why wouldn't we use the scoring model to produce better output? one answer to this is practical: the scoring model might be too large to deploy, or it might not easily support efficient inference @xcite . a more interesting answer is that a scoring model could be set up to provide a signal that is complementary to the systems under evaluation. that is, it might be capable of correctly ranking competing mt hypotheses even when its own preferred hypothesis is worse on average than those of the systems it is evaluating. in our experiments we find that this can indeed be the case. in recent work, @xcite showed that a single multilingual mt model trained on 39 languages can achieve excellent paraphrase recognition when used in zero-shot mode to compare mt output with reference sentences in the same language. on the wmt 2019 metrics task, their method (prism) beat or tied all previous reference-based metrics on all languages. foot_0 although it was not the main focus of their work, prism achieved a new state-of-the-art as a referencefree metric, simply scoring target given source text using an mt model, in a post-competition comparison to the 2019 "quality estimation as a metric" shared task @xcite . our aim in this paper is to characterize the conditions under which the prism approach-using one mt system to perform peer evaluation on other systems-can be successful: what properties does the evaluating system need to have, how powerful should it be, and how close can it be to the systems under evaluation? we focus on system-level evaluation, which we believe is the most compelling use case for reference-free methods, targeting a broad characterization that complements the potentially more precise picture furnished by reference-based metrics for a specific test corpus. we first replicate the correlation with human judgment results from @xcite on wmt 2019, using the same corpora and architecture. next, we examine several alternative design decisions in an attempt to improve prism and further our understanding. these include the effects of varying training corpora (domain, number of languages, use of monolingual data); model capacity (scaling up and down from the original architecture); and different methods for regularizing token-level probabilities (monte-carlo dropout, subword sampling) and for combining them into system-level scores (summary statistics over tokens, confidence thresholds over sentences). finally, we analyze the results of our best model, measuring how its performance depends on various factors: language pair and human-judgment methodology, output quality, proximity to the systems under evaluation, and size of the test set. we demonstrate improvements over the original prism metric due to model capacity and different methods for combining probabilities; surprisingly, we find little gain from adjusting the domain or languages in the original multilingual corpus (although we show that a competition-grade english-german system outperforms the generic multilingual system). we find that the evaluating mt system's output quality is generally correlated with its performance as a metric, although we corroborate the surprising finding from @xcite that it is not necessary to be the best-our system is middle-of-the-road or worse according to bleu across most wmt 2019 languages. we measure the proximity between our system and the systems under evaluation and find no evidence that this is a source of bias. despite using no references, our model achieves approximate parity with bleu both in system-level correlation with human judgment, and when used for pairwise comparisons. ## related work reference-free evaluation is widely used for many nlp tasks such as grammatical error correction @xcite , dialog @xcite ) and text generation @xcite . there has been recent interest in reference-free evaluation for mt, which was a joint track between the wmt 2019 metrics task @xcite and quality estimation task @xcite . reference-free metrics competed head-to-head with standard metrics, and generally did worse. however, the results from the best reference-free systems, uni+ @xcite and yisi-2 @xcite were surprisingly close to the standard metric scores on the language pairs for which they were evaluated. uni+ computes word-level embeddings for source and mt output sentences using pre-trained multilingual bert and laser (artetxe and schwenk, 2019) models, then feeds averaged vectors to a neural classifier trained to predict human scores from previous mt metrics tasks. yisi-2 is similar, except that it works in an unsupervised fashion, computing similarities between mbert embeddings for aligned source and target words, and returning an f-measure statistic. in more recent work, @xcite adopt a similar approach based on mbert, aligning representations from multilingual embedding spaces before computing distances with moverscore @xcite , and adding a gpt-based target-side language model. the current state-of-the-art in reference-free evaluation for mt is represented by the prism approach (thompson and post, 2020) which we extend here. it is worth distinguishing reference-free evaluation from two related tasks that share formal similarities. the first is quality or confidence estimation @xcite @xcite , which aims to score the fitness of mt output for a downstream application. this is typically supervised, although a recent approach @xcite dispenses with the need to learn from human annotations, as do most of the approaches we study in this paper. quality estimation is most usefully applied at the sentence level, and it can make use of powerful "glass-box" features which capture the internals of an mt system. in contrast, reference-free evaluation is most naturally applied at the system (test-set) level, and ideally should make no assumptions about the sys-tems under evaluation. the second task is parallelcorpus mining @xcite , which aims to identify valid translations at various levels of granularity. its scoring aspect is similar to reference-free evaluation, but it is applied to a different input distribution, attempting to identify human-generated translation pairs rather than scoring mt outputs for a given human-generated source text. ## methods we aim to generate a quality score s(x, y ) = x,y s(x, y) for source and target texts x, y which consist of segment (nominally, sentence) pairs x, y. we assume no document or ordering information among segments, and do not directly evaluate scores for individual segment pairs. all methods we consider make use of token-level logprobabilities from a standard autoregressive neural mt system: log p(y t |@xmath0, x), where @xmath1. . . y t . we experimented with reverse probabilities p(x|y), but like @xcite found these gave no advantage, and do not include them in our reported results. the following sections describe our model architecture, scoring techniques, and evaluation methodology. ## results this section presents our main results. all correlations in the tables below are for system-level scores, after outlier systems have been discarded for each language pair. for brevity, we report average correlations, normalized and weighted as described in section 3.3; full results are provided in appendix b. unless otherwise stated, all methods score system outputs using average log probabilities normalized by segment length. ## analysis in this section we analyze various aspects of metric performance, confining our attention to the massive model with mean scoring for consistency. different languages have different relations to our model, to the systems participating in the wmt task, and to the human scoring procedure used in the wmt19 data. ## conclusion in this paper, we have shed some light on the remarkable finding by @xcite that a multilingual model trained on a large (but not enormous) general-domain corpus can be highly effective as an mt metric when used to score the outputs of other mt systems in the absence of reference translations. by scaling up the model and making small adjustments to tagging and scoring, we improve over the original results and achieve approximate parity with bleu in correlation with human judgment on wmt19 data. we argue that this metric is a useful complement to reference-based metrics-including ones that are significantly more powerful than bleu-due to its flexibility; and we provide evidence that scoring reliability can be further improved by using larger source-side-only test sets. we find that the major determinant of success in peer evaluation is the quality of the evaluating model. however, there is no hard requirement that it be better than the models under evaluation: surprisingly, it can correctly rank models that outperform it on average. if we abstract away from quality, performance does not appear to be highly sensitive to the domain or the multilingual versus bilingual nature of the training corpus. taken together, these results have the important practical implication that a single multilingual system such as ours could be broadly applicable for evaluating systems in a large number of language pairs (706 in our case), at different quality levels, and across a wide range of domains. in future work, we look forward to probing these results further, and determining whether alternative architectures or loss functions might be valuable in specializing an mt model for evaluating its peers.