On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation (2020)

Abstract: Evaluation of cross-lingual encoders is usually performed either via zero-shot cross-lingual transfer in supervised downstream tasks or via unsupervised cross-lingual textual similarity. In this paper, we concern ourselves with reference-free machine translation (MT) evaluation where we directly compare source texts to (sometimes low-quality) system translations, which represents a natural adversarial setup for multilingual encoders. Reference-free evaluation holds the promise of web-scale comparison of MT systems. We systematically investigate a range of metrics based on state-of-the-art cross-lingual semantic representations obtained with pretrained M-BERT and LASER. We find that they perform poorly as semantic encoders for reference-free MT evaluation and identify their two key limitations, namely, (a) a semantic mismatch between representations of mutual translations and, more prominently, (b) the inability to punish “translationese”, i.e., low-quality literal translations. We propose two partial remedies: (1) post-hoc re-alignment of the vector spaces and (2) coupling of semantic-similarity based metrics with target-side language modeling. In segment-level MT evaluation, our best metric surpasses reference-based BLEU by 5.7 correlation points.

## introduction a standard evaluation setup for supervised machine learning (ml) tasks assumes an evaluation metric which compares a gold label to a classifier prediction. this setup assumes that the task has clearly defined and unambiguous labels and, in most cases, that an instance can be assigned few labels. these assumptions, however, do not hold for natural language generation (nlg) tasks like machine trans-lation (mt) @xcite and text summarization @xcite , where we do not predict a single discrete label but generate natural language text. thus, the set of labels for nlg is neither clearly defined nor finite. yet, the standard evaluation protocols for nlg still predominantly follow the described default paradigm: (1) evaluation datasets come with human-created reference texts and (2) evaluation metrics, e.g., bleu @xcite or meteor @xcite for mt and rouge @xcite for summarization, count the exact "label" (i.e., n-gram) matches between reference and system-generated text. in other words, established nlg evaluation compares semantically ambiguous labels from an unbounded set (i.e., natural language texts) via hard symbolic matching (i.e., string overlap). the first remedy is to replace the hard symbolic comparison of natural language "labels" with a soft comparison of texts' meaning, using semantic vector space representations. recently, a number of mt evaluation methods appeared focusing on semantic comparison of reference and system translations @xcite @xcite . while these correlate better than n-gram overlap metrics with human assessments, they do not address inherent limitations stemming from the need for reference translations, namely: (1) references are expensive to obtain; (2) they assume a single correct solution and bias the evaluation, both automatic and human @xcite , and (3) limitation of mt evaluation to language pairs with available parallel data. reliable reference-free evaluation metrics, directly measuring the (semantic) correspondence between the source language text and system translation, would remove the need for human references and allow for unlimited mt evaluations: any monolingual corpus could be used for evaluating mt systems. however, the proposals of referencefree mt evaluation metrics have been few and far apart and have required either non-negligible supervision (i.e., human translation quality labels) @xcite or language-specific preprocessing like semantic parsing @xcite , both hindering the wide applicability of the proposed metrics. moreover, they have also typically exhibited performance levels well below those of standard reference-based metrics @xcite . in this work, we comparatively evaluate a number of reference-free mt evaluation metrics that build on the most recent developments in multilingual representation learning, namely cross-lingual contextualized embeddings @xcite and cross-lingual sentence encoders @xcite . we investigate two types of crosslingual reference-free metrics: (1) soft token-level alignment metrics find the optimal soft alignment between source sentence and system translation using word mover's distance (wmd) @xcite . @xcite recently demonstrated that wmd operating on bert representations @xcite substantially outperforms baseline mt evaluation metrics in the reference-based setting. in this work, we investigate whether wmd can yield comparable success in the reference-free (i.e., cross-lingual) setup; (2) sentence-level similarity metrics measure the similarity between sentence representations of the source sentence and system translation using cosine similarity. our analysis yields several interesting findings. (i) we show that, unlike in the monolingual reference-based setup, metrics that operate on contextualized representations generally do not outperform symbolic matching metrics like bleu, which operate in the reference-based environment. (ii) we identify two reasons for this failure: (a) firstly, cross-lingual semantic mismatch, especially for multi-lingual bert (m-bert), which construes a shared multilingual space in an unsupervised fashion, without any direct bilingual signal; (b) secondly, the inability of the state-of-the-art crosslingual metrics based on multilingual encoders to adequately capture and punish "translationese", i.e., literal word-by-word translations of the source sentence-as translationese is an especially persistent property of mt systems, this problem is particularly troubling in our context of referencefree mt evaluation. (iii) we show that by executing an additional weakly-supervised cross-lingual re-mapping step, we can to some extent alleviate both previous issues. (iv) finally, we show that the combination of cross-lingual reference-free metrics and language modeling on the target side (which is able to detect "translationese"), surpasses the performance of reference-based baselines. beyond designating a viable prospect of webscale domain-agnostic mt evaluation, our findings indicate that the challenging task of reference-free mt evaluation is able to expose an important limitation of current state-of-the-art multilingual encoders, i.e., the failure to properly represent corrupt input, that may go unnoticed in simpler evaluation setups such as zero-shot cross-lingual text classification or measuring cross-lingual text similarity not involving "adversarial" conditions. we believe this is a promising direction for nuanced, fine-grained evaluation of cross-lingual representations, extending the recent benchmarks which focus on zeroshot transfer scenarios @xcite . ## related work manual human evaluations of mt systems undoubtedly yield the most reliable results, but are expensive, tedious, and generally do not scale to a multitude of domains. a significant body of research is thus dedicated to the study of automatic evaluation metrics for machine translation. here, we provide an overview of both reference-based mt evaluation metrics and recent research efforts towards reference-free mt evaluation, which leverage cross-lingual semantic representations and unsupervised mt techniques. ## reference-free mt evaluation metrics in the following, we use x to denote a source sentence (i.e., a sequence of tokens in the source language), y to denote a system translation of x in the target language, and y to denote the human reference translation for x. ## analysis we first analyze preferences of our metrics based on m-bert and laser ( §5.1) and then examine how much parallel data we need for re-mapping our vector spaces ( §5.2). finally, we discuss whether it is legitimate to correlate our metric scores, which evaluate the similarity of system predictions and source texts, to human judgments based on system predictions and references ( §5.3). ## conclusion existing semantically-motivated metrics for reference-free evaluation of mt systems have so far displayed rather poor correlation with human estimates of translation quality. in this work, we investigate a range of reference-free metrics based on cutting-edge models for inducing cross-lingual semantic representations: cross-lingual (contextualized) word embeddings and cross-lingual sentence embeddings. we have identified some scenarios in which these metrics fail, prominently their inability to punish literal word-by-word translations (the so-called "translationese"). we have investigated two different mechanisms for mitigating this undesired phenomenon: (1) an additional (weakly-supervised) cross-lingual alignment step, reducing the mismatch between representations of mutual translations, and (2) language modeling (lm) on the target side, which is inherently equipped to punish "unnatural" sentences in the target language. we show that the reference-free coupling of cross-lingual similarity scores with the target-side language model surpasses the reference-based bleu in segment-level mt evaluation. we believe our results have two relevant implications. first, they portray the viability of referencefree mt evaluation and warrant wider research efforts in this direction. second, they indicate that reference-free mt evaluation may be the most challenging ("adversarial") evaluation task for multilingual text encoders as it uncovers some of their shortcomings-prominently, the inability to capture semantically non-sensical word-by-word translations or paraphrases-which remain hidden in their common evaluation scenarios. we release our metrics under the name xmover-score publicly: https://github.com/aiphes/