ACES : Translation Accuracy Challenge Sets at WMT 2023 (2023)

Abstract: We benchmark the performance of segment-level metrics submitted to WMT 2023 using the ACES Challenge Set (Amrhein et al., 2022). The challenge set consists of 36K examples representing challenges from 68 phenomena and covering 146 language pairs. The phenomena range from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. For each metric, we provide a detailed profile of performance over a range of error categories as well as an overall ACES-Score for quick comparison. We also measure the incremental performance of the metrics submitted to both WMT 2023 and 2022. We find that 1) there is no clear winner among the metrics submitted to WMT 2023, and 2) performance change between the 2023 and 2022 versions of the metrics is highly variable. Our recommendations are similar to those from WMT 2022. Metric developers should focus on: building ensembles of metrics from different design families, developing metrics that pay more attention to the source and rely less on surface-level overlap, and carefully determining the influence of multilingual embeddings on MT evaluation.

## introduction challenge sets are a useful tool in measuring the performance of systems or metrics on one or more specific phenomena of interest. they may be used to compare the performance of a range of different systems or to identify performance improvement/degradation between successive iterations of the same system. challenge sets exist for a range of natural language processing (nlp) tasks including sentiment analysis @xcite @xcite , natural language inference @xcite , question answering @xcite , machine reading comprehension @xcite , machine translation (mt) @xcite , and the more specific task of pronoun translation in mt @xcite . the wmt 2021 metrics shared task @xcite introduced the task of constructing contrastive challenge sets for the evaluation of mt metrics. contrastive challenge sets aim to assess how well a given metric is able to discriminate between a good and incorrect translation of the source text. the provision of a reference translation allows for flexibility: it may be included for the assessment of reference-based (i.e. mt) metrics, or excluded for the assessment of reference-free (i.e. quality estimation (qe)) metrics. we re-submitted aces foot_0 @xcite , originally developed for the wmt 2022 challenge sets shared task @xcite , to the corresponding shared task at wmt 2023. aces largely focuses on translation accuracy errors and consists of 68 phenomena ranging from simple perturbations at the word/character level to more complex errors based on discourse and real-world knowledge. we report on both the performance of metrics submitted to wmt 2023, and on the incremental performance of those metrics that were submitted to both wmt 2022 and wmt 2023. we also repeat the analyses in @xcite for the wmt 2023 metrics to confirm whether the findings from wmt 2022 still hold. overall, we find similar trends to those observed last year. again, we do not find one clear winner and whilst neural metrics tend to perform better than their non-neural counterparts, different categories of metrics exhibit different strengths and weaknesses. the major challenges identified in @xcite still hold: (i) reference-based metrics are still overly reliant on the reference and do not pay enough attention to the source, (ii) reference-based metrics still rely on surface-level overlap, and (iii) the over-reliance on multilingual embeddings still persists -evidence from our analyses suggests that language agnostic representations present in the multilingual space may harm performance. accordingly, our recommendations are also similar to those of last year. metric developers should focus on: constructing ensembles of metrics with different design principles, developing metrics that also focus on information in the source, reducing dependence on surface-level overlap with the reference, and reassessing the impact of multilingual embeddings on mt evaluation. with respect to incremental performance changes between metrics submitted to both 2022 and 2023, we observe mixed results. whilst improvements are observed for some metrics, there is a degradation in performance for other metrics. however, even for those metrics for which an overall improvement was observed, this improvement was inconsistent across the top-level categories in aces. further, the performance even degraded for some categories. ## aces overview the translation accuracy challenge set (aces) consists of 36,476 examples covering 146 language pairs and representing challenges from 68 linguistic phenomena. these phenomena are grouped into ten top-level categories: addition, omission, mistranslation, untranslated, do not translate errors, overtranslation, undertranslation, real-world knowledge, wrong language, and punctuation foot_1 . the mistranslation and real-world knowledge categories are further sub-categorised to include additional fine-grained categories. we illustrate the broad accuracy error categories in fig 1 and give examples for each of the top-level categories in appendix a. the focus of aces is on translation accuracy errors, reflecting the need to evaluate contemporary mt systems that are capable of producing fluent but potentially error-prone output. the selection of the top-level categories in the aces error hierarchy is based on the accuracy class in the multidimensional quality metrics (mqm) ontology @xcite , and extended to include translations defying real-world knowledge and translations in the wrong language. aces includes a wide range of phenomena ranging from simple perturbations that involve the omission/addition of characters or tokens, to more complex scenarios involving mistranslations e.g. ambiguity or hallucinations in translation, untranslated elements of a sentence, discourse-level phenomena, and realworld knowledge. each aces example consists of a source sentence, a reference translation, a phenomenon label indicating the error type, and two translation mistranslation -overly literal (idioms) src (de): er hat versucht, mir die spielregeln zu erklären, aber ich verstand nur bahnhof. ref (en): he tried to explain the rules of the game to me, but i did not understand them. real-world knowledge -commonsense src (en): die luft im haus war kühler als in der wohnung. ref (de): the air in the house was cooler than in the apartment. ## related work challenge sets have been used for several tasks @xcite ; @xcite ; @xcite , inter alia) to investigate the behaviour of these tasks under a specific phenomenon rather than the standard test distribution @xcite . lately, with the success of neural metrics, the development of challenge sets for mt evaluation has promoted great interest in studying the strengths and weaknesses of these metrics. we summarise here recent work on challenge sets for mt metric evaluation. demetr @xcite , which comprises 31k english examples translated from ten languages, was developed for evaluating mt met-ric sensitivity to a range of 35 different types of linguistic perturbations, belonging to semantic, syntactic, and morphological error categories. these were divided into minor, major, and critical errors according to the type of perturbation, similar to the grading of error categories to compute the weighted aces-score. as in aces, example generation was carefully designed to form minimal pairs such that the perturbed translation only differs from the actual translation in one aspect. the application of demetr in evaluating a suite of baseline metrics revealed a similar pattern to the analyses in amrhein et al. (2022) -that metric performance varies considerably across the different error categories, often with no clear winner. it is worth noting that demetr and aces each have their respective advantages: all examples in demetr have been verified by human annotators; aces provides broader coverage in terms of both languages and linguistic phenomena. in addition to aces, three other datasets were submitted to the wmt 2022 challenge sets shared task @xcite : smaug (alves et al., 2022), the hwtsc challenge set @xcite , and the dfki challenge set @xcite . these datasets differ from aces in terms of their size, and the languages and phenomena/categories they cover (see both smaug and hwtsc contained five different phenomena each pertaining to a single category of critical error for meaning change. in comparison, the dfki challenge set has over 100 linguistically motivated phenomena, organised into 14 categories. whereas the aim of aces was to provide a broad coverage of language pairs, the other datasets provide an in-depth focus on specific lan- ## metrics we list below the metrics that participated in the 2023 challenge set shared task and the baselines provided by the organisers. ## conclusion we re-submitted the aces challenge set to wmt2023 to identify the strengths and weaknesses of the metrics submitted to this year's shared task. overall, we find similar trends to that of last year. while neural metrics tend to be better, different categories of metrics have different strengths, and we do not find one clear winner. with respect to the metrics that were resubmitted with some design changes, we find that these design changes have variable outcomes with a performance drop in some cases. the major challenges of (i) metrics not paying enough attention to the source, (ii) referencebased metrics still relying on surface-level overlap, and (iii) over-reliance on multilingual embeddings still persist. hence, our recommendations are also similar to that of last year: build ensembles of different design families, encourage development that better utilises information in the source, include diverse training examples to reduce the influence of surface-level overlap, and carefully determine the influence of multilingual embeddings/llms on mt evaluation.