M a TES e: Machine Translation Evaluation as a Sequence Tagging Problem (2022)

Abstract: Starting from last year, WMT human evaluation has been performed within the Multidimensional Quality Metrics (MQM) framework, where human annotators are asked to identify error spans in translations, alongside an error category and a severity. In this paper, we describe our submission to the WMT 2022 Metrics Shared Task, where we propose using the same paradigm for automatic evaluation: we present the MaTESe metrics, which reframe machine translation evaluation as a sequence tagging problem. Our submission also includes a reference-free metric, denominated MaTESe-QE. Despite the paucity of the openly available MQM data, our metrics obtain promising results, showing high levels of correlation with human judgements, while also enabling an evaluation that is interpretable. Moreover, MaTESe-QE can also be employed in settings where it is infeasible to curate reference translations manually.

## introduction and related work for many years, machine translation (mt) has mainly been evaluated using untrained evaluation techniques, such as bleu @xcite , meteor @xcite and chrf @xcite , which rely heavily on lexical-level matching of either token, or character, n-grams. unfortunately, these metrics present two major drawbacks: i) it is not possible to carry out the evaluation without manually-curated references and, most importantly, ii) the evaluation is too dependent on the surface form of the translation, and its reference. more recently, attempts have been made to address these problems using machine-learned metrics, which have shown better correlations with human judgements @xcite . more specifically, last year's wmt metrics shared task saw c-spec pn @xcite , bleurt-20 foot_0 and comet-mqm_2021 @xcite emerge as distinctly better than the other participants @xcite . these metrics consist of regression models trained to mimic human annotators by directly assigning quality scalar scores to candidate translations. in detail, comet-mqm_2021 is based on the estimator architecture introduced by @xcite , where features extracted from the embeddings of the source sentence, candidate translation, and reference translation are passed to a feed-forward regressor; c-spec first concatenates the embeddings derived from paired inputs of candidate-source and candidate-reference, and then passes the resulting vector to a multi-layer perceptron; bleurt, instead, feeds the candidate translation and its reference to rebalanced mbert @xcite , and regresses on the representation provided by the [cls] token. moreover, bleurt and c-spec add automaticallygenerated negative pairs to the standard training data: bleurt applies random token perturbations, while c-spec uses word attribute transfer to replace words in the translations. although undoubtedly effective, regression metrics have the major drawback of not being interpretable, meaning that users are not able to gauge the quality of assessments that are returned, which is of paramount importance for an evaluation metric. recently, @xcite have proposed a shift in the standard practices for human machine translation evaluation, employing the multidimensional quality metrics framework @xcite , and moving away from direct assessments @xcite , which were computed via requiring (even non-expert) annotators to assign a scalar value to a candidate translation, given a reference. furthermore, freitag et al. (2021a) pointed out the limitations of non-professional direct assessments, also show-ing their unreliability compared to mqm. indeed, differently from direct assessments, annotators who follow the mqm guidelines look at the source sentence rather than the reference, and are expected to tag the spans of the candidate translations that contain errors, foot_2 together with their error category (e.g., fluency/grammar, fluency/punctuation or style/awkward) and severity (e.g., major or minor), which, combined, determine the score associated with the error span. finally, a scalar quality score for the entire sentence is derived from the various annotated spans. in this work, we introduce the matese and matese-qe metrics, reframing the evaluation of machine-translated text as a sequence tagging problem based on the mqm framework, in an attempt to develop metrics that are interpretable, while also displaying high levels of correlation with human judgements. ## matese metrics inspired by the novel mqm evaluation framework, our work aims at employing a similar paradigm for automatic evaluation. we propose the matese metrics which, given a candidate translation and its reference (or source, for matese-qe), assign a label to each token of the candidate. these labels identify error spans, together with their severity, chosen among major and minor. finally, in order to associate a score with the entire tagged sentence, we follow a weighting scheme similar to the one presented by @xcite for mqm-based human evaluation: we assign a score to an entire error span based on its severity, i.e., -5 and -1 for major and minor, respectively. the score assigned to a translation is the sum of the scores assigned to its error spans, with a minimum total score of -25. following freitag et al. (2021a), we compute a corpus-level score by averaging the scores of the sentences in the corpus. although human mqm annotators are asked to report a maximum of 5 errors per translation, foot_3 we decided to let our metrics detect as many errors as they can find; nevertheless, in order to keep our scores in the same range as those computed on gold mqm annotations, we set a minimum score of -25, which is equal to the ## experimental setup in this section, we describe the different architectures we experiment with, the data for training and evaluation, and the metrics we use to measure performances. ## results in this section, we show the results obtained by our metrics. unless explicitly specified, all experiments have been performed using reference-based systems. ## conclusions in this paper, we described our submission to the wmt 2022 metrics shared task: we presented the matese metrics, a new way of automatically assessing the quality of translations, putting forward evaluation techniques that are interpretable, while at the same time displaying high levels of correlation with human judgements. scores are in the same ballpark of the best performing metrics proposed in the wmt 2021 metrics shared task. furthermore, the matese metrics can also be used in the absence of humanly-curated references, with matese-qe being slightly less accurate than its reference-based counterpart, but still presenting encouraging levels of correlation with human judgements. in future work, we plan to improve the matese metrics to also detect the type of errors, and not only their severity, in order to approximate even better the mqm annotation process.