Extrinsic Evaluation of Machine Translation Metrics (2023)

Abstract: Automatic machine translation (MT) metrics are widely used to distinguish the quality of machine translation systems across relatively large test sets (system-level evaluation). However, it is unclear if automatic metrics are reliable at distinguishing good translations from bad translations at the sentence level (segment-level evaluation). In this paper, we investigate how useful MT metrics are at detecting the segment-level quality by correlating metrics with how useful the translations are for downstream task. We evaluate the segment-level performance of the most widely used MT metrics (chrF, COMET, BERTScore, etc.) on three downstream cross-lingual tasks (dialogue state tracking, question answering, and semantic parsing). For each task, we only have access to a monolingual task-specific model and a translation model. We calculate the correlation between the metric’s ability to predict a good/bad translation with the success/failure on the final task for the machine translated test sentences. Our experiments demonstrate that all metrics exhibit negligible correlation with the extrinsic evaluation of the downstream outcomes. We also find that the scores provided by neural metrics are not interpretable, in large part due to having undefined ranges. We synthesise our analysis into recommendations for future MT metrics to produce labels rather than scores for more informative interaction between machine translation and multilingual language understanding.

## introduction although machine translation (mt) is typically seen as a standalone application, in recent years mt models have been more frequently deployed as a component of a complex nlp platform delivering multilingual capabilities such as cross-lingual information retrieval @xcite or automated multilingual customer support @xcite . when an erroneous translation is generated by the mt systems, it may add new errors in the task pipeline leading to task failure and poor user ex-perience. for example, consider the user's request in chinese 剑桥有牙买加菜吗？ ("is there any good jamaican food in cambridge?") machinetranslated into english as "does cambridge have a good meal in jamaica?". the model will erroneously consider "jamaica" as a location, instead of cuisine, and prompt the search engine to look up restaurants in jamaica foot_0 . to avoid this breakdown, it is crucial to detect an incorrect translation before it causes further errors in the task pipeline. one way to approach this breakdown detection is using segment-level scores provided by mt metrics. recent mt metrics have demonstrated high correlation with human judgements at the system level for some language pairs @xcite . these metrics are potentially capable of identifying subtle differences between mt systems that emerge over a relatively large test corpus. these metrics are also evaluated on respective correlation with human judgements at the segment level, however, there is a considerable performance penalty @xcite . segment-level evaluation of mt is indeed more difficult and even humans have low inter-annotator agreement on this task @xcite . despite mt systems being a crucial intermediate step in several applications, characterising the behaviour of these metrics under task-oriented evaluation has not been explored. in this work, we provide a complementary evaluation of mt metrics. we focus on the segmentlevel performance of metrics, and we evaluate their performance extrinsically, by correlating each with the outcome of downstream tasks with respective, reliable accuracy metrics. we assume access to a parallel task-oriented dataset, a task-specific monolingual model, and a translation model that can translate from the target language into the language of the monolingual model. we consider the translate-test setting -where at test time, the examples from the test language are translated to the task language for evaluation. we use the outcomes of this extrinsic task to construct a breakdown detection benchmark for the metrics. we use dialogue state tracking, semantic parsing, and extractive question answering as our extrinsic tasks. we evaluate nine metrics consisting of string overlap metrics, embedding-based metrics, and metrics trained using scores from human evaluation of mt. surprisingly, we find our setup challenging for all existing metrics; demonstrating poor capability in discerning good and bad translations across tasks. we present a comprehensive analysis of the failure of the metrics through quantitative and qualitative evaluation. our contributions are summarised as follows: 1) we derive a new breakdown detection task, for evaluating mt metrics, measuring how indicative segment-level scores are for downstream performance of an extrinsic cross-lingual task (section 3). we evaluate nine metrics on three extrinsic tasks covering 39 unique language pairs. the task outputs, the breakdown detection labels, and metric outputs are publicly available. ## related work evaluation of machine translation has been of great research interest across different communities @xcite . notably, the conference on machine translation (wmt) has been organising annual shared tasks on automatic mt evaluation since 2006 @xcite ) that invites metric developers to evaluate their methods on outputs of several mt systems. metric evaluation typically includes a correlation of the scores with human judgements collected for the respective translation outputs. but, designing such guidelines is challenging @xcite , leading to the development of several different methodologies and analyses over the years. the human evaluation protocols include general guidelines for fluency, adequacy and/or comprehensibility @xcite on continuous scales @xcite (direct assessments) or fine-grained annotations of mt errors @xcite based on error ontology like multidimensional quality metrics (mqm) @xcite or rank outputs from different mt systems for the same input @xcite . furthermore, the best way to compare mt scores with their corresponding judgements is also an open question @xcite @xcite . the new metrics claim their effectiveness by comparing their performance with competitive metrics on the latest benchmark. the progress and criticism of mt evaluation are generally documented in a metrics shared task overview @xcite . for example, @xcite highlighted the effectiveness of neural embedding-based metrics; @xcite show that metrics struggle on segmentlevel performance despite achieving impressive system-level correlation; @xcite investigate how different metrics behave under different domains. in addition to these overviews, @xcite show that meta-evaluation regimes were sensitive to outliers and minimal changes in evaluation metrics are insufficient to claim metric efficacy. @xcite conducted a comprehensive evaluation effort to identify which metric is best suited for pairwise ranking of mt systems. @xcite look at a specific phenomenon of whether metrics are capable of evaluating translations involving pronominal anaphora. recent works have also criticised individual metrics such as comet @xcite and bertscore @xcite . these works draw their conclusions based on some comparison with human judgement or on specific pitfalls of individual metrics. our work focuses on the usability of the metrics as solely judged on their ability to predict downstream tasks where mt is an intermediate step (with a primary emphasis on segment-level performance). taskbased evaluation has been well studied @xcite ; @xcite ; zhang et al. ## methodology our aim is to determine how reliable mt metrics are for predicting success on downstream tasks. our setup uses a monolingual model (e.g., a dialogue state tracker) trained on a task language and parallel test data from multiple languages. we use mt to translate a test sentence (from a test language to the task language) and then infer a label for this example using the monolingual model. if the model predicts a correct label for the parallel task language input but an incorrect label for the translated test language input, then we have observed a breakdown due to a material error in the translation pipeline. we then study if the metric could predict if the translation is suitable for the end task. we refer to figure 1 for an illustration. we frequently use the terms test language and task language to avoid confusion with the usage of source language and target language in the traditional machine translation setup. in figure 1 , the task language is english and the test language is chinese. we now describe our evaluation setup and the metrics under investigation. ## results we report the aggregated results for semantic parsing, question answering, and dialogue state tracking in ## recommendations our experiments suggest that evaluating mt metrics on the segment level for extrinsic tasks has considerable room for improvement. we propose recommendations based on our observations: prefer mqm for human evaluation of mt outputs: we reinforce the proposal of using the mqm scoring scheme with expert annotators for evaluating mt outputs in line with @xcite . as seen in section 4.2, different tasks have varying tolerance to different mt errors. with explicit errors marked per mt output, future classifiers can be trained on a subset of human evaluation data containing errors most relevant to the downstream application. mt metrics could produce labels over scores: the observations from section 4.2 and section 4.3 suggest that interpreting the quality of the produced mt translation based on a number is unreliable and difficult. we recommend exploring whether segment-level mt evaluation can be approached as an error classification task instead of regression. specifically, whether the words in the source/hypothesis can be tagged with explicit error labels. resorting to mqm-like human evaluation will result in a rich repository of human evaluation based on an ontology of errors and erroneous spans marked across the source and hypothesis @xcite . similarly, the post-editing datasets @xcite , inter alia) also provide a starting point. recent exploration in this direction are the works by @xcite ; @xcite that treat mt evaluation as a sequence-tagging problem by labelling the errors in an example. such metrics can also be used for intrinsic evaluation by assigning weights to the labels to produce a weighted score. add diverse references during training: from section 4.2, we find that both the neural metric and the task-specific model are not robust to paraphrases. we also recommend the inclusion of diverse references through automatic paraphrasing @xcite or data augmentation during the training of neural metrics. ## conclusion we propose a method for evaluating mt metrics which is reliable at the segment-level and does not depend on human judgements by using correlation mt metrics with the success of extrinsic downstream tasks. we evaluated nine different metrics on the ability to detect errors in generated translations when machine translation is used as an intermediate step for three extrinsic tasks: semantic parsing, question answering, and dialogue state tracking. we find that segment-level scores provided by all the metrics show negligible correlation with the success/failure outcomes of the end task across different language pairs. we attribute this result to segment scores produced by these metrics being uninformative and that different extrinsic tasks demonstrate different levels of sensitivity to different mt errors. we propose recommendations to predict error types instead of error scores to facilitate the use of mt metrics in downstream tasks.