Challenging the State-of-the-art Machine Translation Metrics from a Linguistic Perspective (2023)

Abstract: We employ a linguistically motivated challenge set in order to evaluate the state-of-the-art machine translation metrics submitted to the Metrics Shared Task of the 8th Conference for Machine Translation. The challenge set includes about 21,000 items extracted from 155 machine translation systems for three language directions, covering more than 100 linguistically-motivated phenomena organized in 14 categories. The metrics that have the best performance with regard to our linguistically motivated analysis are the Cometoid22-wmt23 (a trained metric based on distillation) for German-English and MetricX-23-c (based on a fine-tuned mT5 encoder-decoder language model) for English-German and English-Russian. Some of the most difficult phenomena are passive voice for German-English, named entities, terminology and measurement units for English-German, and focus particles, adverbial clause and stripping for English-Russian.

## introduction most nlp evaluation has relied for years on testing the system performance on randomly picked test sets and producing a single generic score. yet, machine learned systems learn to make abstractions and due to these, phenomena who are on the long tail of the training and test data may be overlooked hidden behind a very high generic score. additionally, generic scores are often helpful to show relative improvement and reflect overall quality, but cannot explain the performance in a comprehensive way. for example, old-style machine translation (mt) metrics measuring lexical overlap would equally penalize the omission of an article and the omission of the particle forming the negation in a sentence, although negation is more crucial for its meaning. while the evaluation of so obvious errors has been addressed by the trained mt metrics, their evaluation relies on correlations with human judgments on randomly picked test-sets. in this case, a single correlation score may not be able to explain the strengths and weaknesses of the metrics with regard to the functioning of language. motivated by these considerations, we employ a multifold test set with linguistically-motivated challenges that will allow us to understand the metric performance from a linguistic perspective. these challenges are organized in smaller sets, one set per phenomenon, whereas the phenomena are organized in broader categories. by measuring the ability of the metrics to detect the errors in these challenge sets, we can get scores that indicate different aspects of linguistic performance. this paper describes the application of such a challenge set on the evaluation of the mt metrics submitted at the relevant shared task of the 8th conference of machine translation @xcite . the rest of the paper is structured as following: section 2 describes related work, and section 3 describes the way the challenges were selected. in section 4 the results are presented and described, first from the perspective of metric comparison and then focusing on the performance for particular linguistically-motivated categories and phenomena per language direction. some conclusions are given in section 5. ## related work there has been a growing interest for more finegrained evaluation of natural language processing (nlp) tools, as shown by the increasing number of publications many of whom have received distinctions @xcite @xcite . concerning machine translation (mt), initial efforts were made in the 1990s with the introduction of test suites @xcite , and these efforts have been revitalized in light of recent advancements in the field @xcite . to the best of our knowledge, the first endeavours related to the use of challenge sets in a meta-level in order to evaluate mt metrics were applied to quality estimation metrics @xcite , based on the first version of our linguistically-motivated test suite @xcite . the analysis was broadened to cover a broader range of mt metrics, including reference-based ones, as appeared in the findings paper of the metrics shared task of the 6th conference on machine translation @xcite , which was based on a later version of our test suite on german-english @xcite @xcite , a resource also employed in this paper. with the occasion of the first challenge set subtask for the metrics shared task of the 7th conference on machine translation @xcite , a few more challenge sets emerged. aces @xcite for example, focuses on 68 accuracy errors. similarly, @xcite evaluate the robustness of mt metrics by generating translations with critical errors. in a more linguistic direction, @xcite examine the capability of the metrics to correlate synonyms in different areas and to discern catastrophic errors at both wordand sentence-levels. our submission at that sub-task (avramidis and macketanz, 2022) augmented the preliminary analysis appearing at @xcite by adding the language direction of english-german and presenting a more fine-grained analysis, not only in the category level but also on the phenomenon level. this year's submission, explained on our paper, includes that same challenge set as last year, whereas english-russian has been added as an additional language direction. ## results the results are displayed in detail in tables 1 , 2 and 3 for the category level and in tables 4, 5 and 6 for the phenomenon level, for the three language directions respectively. ## conclusion in this paper we analysed the performance of several state-of-the-art metrics with regard to particular linguistically-motivated phenomena for three language pairs, german-english, english-german and for the first time, english russian. the analysis gave a multitude of observations, regarding both the performance of the metrics and the corresponding linguistic observations. the metrics demonstrating the best performance in average are cometoid22-wmt23 for the german-english language pair, and metricx-23-c for both the english-german and english-russian language pairs. quality estimation methods have impressively good performance in several phenomena. some metrics that report usage of llms (embed_llama) have not scored very high in overall, indicating that more work is required in this direction. among the various linguistic phenomena, we could identify some of the particularly challenging ones. in german-english, metrics have difficulties scoring the passive voice properly. in english-german named entities and terminology as well as specific measurement units pose the most difficulties. in english-russian translation, translations with focus particles, adverbial clause, and stripping phenomena emerge as particularly complex challenges. chantal amrhein, nikita moghe, and liane guillou.