ACES : Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics (2022)

Abstract: As machine translation (MT) metrics improve their correlation with human judgement every year, it is crucial to understand the limitations of these metrics at the segment level. Specifically, it is important to investigate metric behaviour when facing accuracy errors in MT because these can have dangerous consequences in certain contexts (e.g., legal, medical). We curate ACES, a translation accuracy challenge set, consisting of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. We use ACES to evaluate a wide range of MT metrics including the submissions to the WMT 2022 metrics shared task and perform several analyses leading to general recommendations for metric developers. We recommend: a) combining metrics with different strengths, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference and c) explicitly modelling additional language-specific information beyond what is available via multilingual embeddings.

## introduction challenge sets have already been created for measuring the success of systems or metrics on a particular phenomenon of interest for a range of nlp tasks, including but not limited to: sentiment analysis foot_1 @xcite @xcite , natural language inference @xcite , question answering @xcite , machine reading comprehension @xcite , machine translation (mt) @xcite , and the more specific task of pronoun translation in mt @xcite . they are useful to compare the performance of different systems, or to identify performance improvement/degradation between a modified system and a previous iteration. in this work, we describe the university of zurich -university of edinburgh submission to the challenge sets subtask of the conference on machine translation (wmt) 2022 metrics shared task. our translation accuracy challenge set (aces) consists of 36,476 examples covering 146 language pairs and representing challenges from 68 phenomena (see @xcite for the distribution of examples across language pairs and appendix a.5 for the distribution of language pairs across phenomena). we focus on translation accuracy errors and base the phenomena covered in our challenge set on the multidimensional quality metrics (mqm) ontology @xcite . we include phenomena ranging from simple perturbations involving the omission/addition of characters or tokens, to more complex examples involving mistranslation e.g. ambiguity and hallucinations in translation, untranslated elements of a sentence, discourse-level phenomena, and real-world knowledge. we evaluate the metrics submitted to the wmt 2022 metrics shared task and a range of baseline metrics on aces. additionally, we perform an extensive analysis, which aims to reveal: ## motivation with the advent of neural networks and especially transformer-based architectures @xcite , machine translation outputs have become more and more fluent @xcite @xcite . fluency errors are also judged less severely than accuracy errors by human evaluators @xcite which reflects the fact that accuracy errors can have dangerous consequences in certain contexts, for example in the medical and legal domains @xcite . for these reasons, we decided to build a challenge set focused on accuracy errors. specifically, we use the hierarchy of errors under the class accuracy from the mqm ontology to design these challenge sets. we extend this ontology by two er-ror classes (translations defying real-world knowledge and translations in the wrong language) and specify several more specific subclasses such as discourse-level errors or ordering mismatches. a full overview of all error classes can be seen in figure 1. our challenge set consists of synthetically generated adversarial examples, examples from repurposed contrastive mt test sets (both marked in red), and manually annotated examples (marked in blue). to create the challenge sets, we use test sets from tasks such as adversarial paraphrase detection, natural language inference, and contrastive mt test sets created independently of the wmt shared tasks to avoid overlap with the data that is used to train neural evaluation metrics. another aspect we focus on is including a broad range of language pairs in aces. whenever possible we create examples for all language pairs covered in a source dataset when we use automatic approaches. for phenomena where we create examples manually, we also aim to cover at least two language pairs per phenomenon, but are of course limited to the languages spoken by the authors. finally, we aim to offer a collection of challenge sets covering both easy and hard phenomena. while it may be of interest to the community to continuously test on harder examples to check where machine translation evaluation metrics still break, we believe that easy challenge sets are just as important to ensure that metrics do not suddenly become worse at identifying error types that were previously considered "solved". therefore, we take a holistic view when creating aces and do not filter out individual examples or exclude challenge sets based on baseline metric performance or other factors. we first discuss previous efforts to create challenge sets (section 3), before giving a broad overview of the datasets used to construct aces (section 4) and discussing the individual challenge sets in more detail (section 5). we then introduce the metrics that participated in the shared task (section 6), present an overview of their performance on aces (section 7) and detailed analyses (section 8) that lead to a set of recommendations for future metric development (section 9). ## related work challenge sets are used to study a particular phenomenon of interest rather than the general distribution of phenomena in standard test sets @xcite . the earliest introduction of challenge sets was by @xcite who probed acceptability of machine translations for different domains. challenge sets have been prevalent in different fields within nlp such as parsing @xcite , @xcite , question answering @xcite , reading comprehension @xcite and sentiment analysis @xcite @xcite , to name a few. these challenge sets provide insights on whether state-of-the-art models are robust to domain shifts, and whether they have some understanding of linguistic phenomena like negation/commonsense or they simply rely on shallow heuristics. another line of work under "adversarial datasets" also focuses on creating examples by perturbing the standard test set to fool the model @xcite ; @xcite , inter-alia). challenge sets for evaluating mt systems have focused on the translation models' ability to generate the correct translation given a phenomenon of interest. these include word sense ambiguity @xcite , gender bias @xcite @xcite , structural divergence @xcite and discourse level phenomena @xcite . while such challenge sets focus on evaluating specific machine translation models, it is necessary to identify whether the existing machine translation evaluation metrics also perform well under these and related phenomena. developing challenge sets for machine translation metric evaluation has gained considerable interest because recently, neural mt evaluation metrics have shown improved correlation with human judgements @xcite . however, their weaknesses remain relatively unknown and only a small number of works (e.g. @xcite and @xcite ) have proposed systematic analyses to uncover them. previous challenge sets for metric evaluation focused on negation and sentiment polarity @xcite and synthetic perturbations such as antonym replacement, word omission, number swapping, punctuation removal, etc. @xcite . @xcite developed a manually constructed test suite of linguistically motivated perturbations for identifying weaknesses in reference-free evaluation. however, these challenge sets for metrics are only focused on highresource language pairs such as english↔german and @xmath0. in this work, we repurpose existing machine translation challenge sets to evaluate machine translation evaluation metrics. we introduce several synthetically generated and manually created challenge sets that broadly focus on translation accuracy errors for 146 language pairs. ## datasets the majority of the examples in our challenge set were based on data extracted from three main datasets: flores-101, paws-x, and xnli (with additional translations from xtreme). the flores-101 evaluation benchmark @xcite consists of 3,001 sentences extracted from english wikipedia and translated into 101 languages by professional translators. @xcite expands the set of languages in flores-101. originally intended for multilingual and low-resource mt evaluation, these datasets have a particular focus on low-resource languages. paws-x @xcite , a cross-lingual dataset for paraphrase identification, consists of pairs of sentences that are labelled as true or adversarial paraphrases. it comprises the wikipedia portion of the paws corpus @xcite translated from english into six languages: french, spanish, german, chinese, japanese, and korean. the development and test sets (23,659 sentences total) were manually translated by professional translators, and the training set was translated using nmt systems via google cloud translation 3 . xnli @xcite is a multilingual natural language inference (nli) dataset consisting of 7,500 premise-hypothesis pairs with their corresponding inference label. the english examples were generated by crowd source workers before being manually translated into 14 languages: french, spanish, german, greek, bulgarian, russian, turkish, arabic, vietnamese, thai, chinese, hindi, swahili and urdu. in addition, we use the automatic translations from xtreme @xcite of the xnli test set examples from these 14 languages into english. for the mistranslation phenomena gender in occupation names and word sense disambiguation, we leveraged the winomt and mucow datasets. winomt @xcite , a challenge set developed for analysing gender bias in mt, contains 3,888 english examples extracted from the winogender @xcite and winobias @xcite coreference test sets. winomt sentences cast participants into non-stereotypical gender roles and the dataset has an equal balance of male and female genders, and of stereotypical and non-stereotypical gender-role assignments (e.g., a female nurse vs. a female doctor). mu-cow @xcite is a multilingual contrastive, word sense disambiguation test suite for machine translation. the dataset covers 16 language pairs with more than 200,000 contrastive sentence pairs. it was automatically constructed from word-aligned parallel corpora and babelnet's @xcite wide-coverage multilingual sense inventory. for the discourse-level phenomena, we relied on annotated resources developed specifically to support work on those phenomena in an mt setting. the wmt 2018 english-german pronoun translation evaluation test suite @xcite contains 200 examples of the ambiguous english pronouns it and they extracted from the ted talks portion of parcorfull @xcite . the example sentences were translated into german by the 16 english-german systems submitted to wmt 2018, and the (german) pronoun translations were manually judged by human annotators as "good/bad". wino-x (emelin 3 https://cloud.google.com/translate and sennrich, 2021) is a parallel dataset of german, french, and russian winograd schemas, aligned with their english counterparts. it was developed for commonsense reasoning and coreference resolution and used for this purpose to generate examples for commonsense co-reference disambiguation. the europarl concodisco corpus @xcite comprises the english-french parallel texts from europarl @xcite over which automatic methods were used to perform pdtb-style discourse connective annotation. discourse connectives are labelled with their sense type and are aligned between the two languages. ## challenge sets creating a contrastive challenge set for evaluating a machine translation evaluation metric requires a source sentence, a reference translation, and two translation hypotheses: one which contains an error or phenomenon of interest (the "incorrect" translation) and one which is a correct translation in that respect (the "good" translation). one possible way to create such challenge sets is to start with two alternative references (or two identical copies of the same reference) and insert errors into one of them to form an incorrect translation while the uncorrupted version can be used as the good translation. this limits the full evaluation scope to translation hypotheses that only contain a single error. to create a more realistic setup, we also create many challenge sets where the good translation is not free of errors, but it is a better translation than the incorrect translation. for automatically created challenge sets, we put measures in place to ensure that the incorrect translation is indeed a worse translation than the good translation. ## evaluation methodology we shall now briefly describe the metrics that participated in the challenge set shared task. the organisers of the shared task also provided scores by a number of baseline metrics, as described below. bertscore @xcite uses contextual embeddings from pre-trained language models to compute the similarity between the tokens in the reference and the generated translation using cosine similarity. the similarity matrix is used to compute precision, recall, and f1-scores. bleurt20 @xcite ) is a bert-based @xcite regression model, which is first trained on scores of automatic metrics/similarity of pairs of reference sentences and their corrupted counterparts. it is then fine-tuned on the wmt human evaluation data to produce a score for a hypothesis given a reference translation. comet-20 @xcite uses a cross-lingual encoder (xlm-r @xcite ) and pooling operations to obtain sentence-level representations of the source, hypothesis, and reference. these sentence embeddings are combined and then passed through a feedforward network to produce a score. comet is trained on human evaluation scores of machine translation systems submitted to wmt until 2020. comet-qe was trained similarly to comet-20 but as this is a reference-free metric, only the source and the hypothesis are combined to produce a final score. yisi-1 @xcite measures the semantic similarity between the hypothesis and the reference by using cosine similarity scores of multilingual representations at the lexical level. it optionally uses a semantic role labeller to obtain structural similarity. finally, a weighted f-score based on structural and lexical similarity is used for scoring the hypothesis against the reference. ## analysis aside from high-level evaluations of which metrics perform best, we are mostly interested in metricspanning weaknesses that we can identify using aces. this section shows an analysis of three general questions that we aim to answer using aces. ## recommendations based on the metrics results on aces and our analysis, we derived the following list of recommendations for future mt evaluation metric development: no metric to rule them all: both the evalua-tion on phenomena and on language pair categories in section 7 showed that there is no single bestperforming metric. this divergence is likely to become even larger if we evaluate metrics on different domains. for future work on mt evaluation, it may be worthwhile thinking about how different metrics can be combined to make robust decisions as to which is the best translation. this year's submissions to the metrics shared task already suggest that work in that direction is ongoing as some groups submitted metrics that combined ensembles of models or multiple components (comet-22, comet-kiwi, kg-bertscore, mee*, reuse). the source matters: our analysis in section 8.1 highlighted that many reference-based metrics that take the source as input do not consider it enough. cases where the correct translation can only be identified through the source are currently better handled by reference-free metrics. this is a serious shortcoming of reference-based metrics and should be addressed in future research, also considering that many reference-based metrics do not even take the source as input. surface-overlap still prevails: in section 8.2, we showed that despite moving beyond only surface-level comparison to the reference, most reference-based metric scores are still considerably influenced by surface-level overlap. we expect future metrics to use more lexically diverse references in their training regime to mitigate this issue. multilingual embeddings are not perfect: some properties of multilingual representations, especially, being language-agnostic, can result in undesirable effects on mt evaluation (section 8.3). it could be helpful for future metrics to incorporate strategies to explicitly model additional languagespecific information. ## conclusion we presented aces, a translation accuracy challenge set based on the mqm ontology. aces consists of 36,476 examples covering 146 language pairs and representing challenges from 68 phenomena. we used aces to evaluate the baseline and submitted metrics from the wmt 2022 metrics shared task. our overview of metric performance at the phenomena and language levels in section 7 reveals that there is no single best-performing metric. the more fine-grained analyses in section 8 highlight that 1) many reference-based metrics that take the source as input do not consider it enough, 2) most reference-based metric scores are still considerably influenced by surface overlap with the reference, and 3) the use of multilingual embeddings can have undesirable effects on mt evaluation. we recommend that these shortcomings of existing metrics be addressed in future research, and that metric developers should consider a) combining metrics with different strengths, e.g. in the form of ensemble models, b) developing metrics that give more weight to the source and less to surface-level overlap with the reference, and c) incorporating strategies to explicitly model additional language-specific information (rather than simply relying on multilingual embeddings). we have made aces publicly available and hope that it will provide a useful benchmark for mt evaluation metric developers in the future.