MTEQA at WMT 21 Metrics Shared Task (2021)

Abstract: In this paper, we describe our submission to the WMT 2021 Metrics Shared Task. We use the automatically-generated questions and answers to evaluate the quality of Machine Translation (MT) systems. Our submission builds upon the recently proposed MTEQA framework. Experiments on WMT20 evaluation datasets show that at the system-level the MTEQA metric achieves performance comparable with other state-of-the-art solutions, while considering only a certain amount of information from the whole translation.

## introduction the goal of automatic machine translation (mt) evaluation is to automatically evaluate the output quality produced by mt systems. metrics used for this task assign a score by comparing the mt output to either a reference translation or to the source sentence. the main indicator that is used to assess the performance of a specific metric is the correlation with human judgement computed for outputs from several systems. it was recently shown that metrics based on contextualized embeddings, such as yisi @xcite or esim @xcite , are able to achieve better performance than the most common bleu @xcite . in this paper, we describe application of the recently proposed metric -mteqa @xcite for the task of evaluating the quality of mt outputs in the context of the wmt21 metric task. the mteqa 1 framework is inspired by previous works on evaluating abstractive summaries. it builds upon the fact that state-of-the-art (neural) mt systems tend to produce a fluent output but sometimes fail in adequacy of the translation. it leverages the recent progress in question generation (qg) and question answering (qa) to formulate and answer questions based on the mt output. the recently proposed chrf @xcite considers the character-level n-grams, making it possible to reward partially matched tokens. recently, various works (e.g., @xcite @xcite explored the usage of contextualized word-or sentence-level embeddings to compare the numerical representations of reference and hypothesis. such metrics enable explicit regression towards the desired human-produced labels. ## experiments our implementation of the mteqa metric is based on the state-of-the-art system capable of solving the initial three tasks of the procedure: answer extraction, question generation, question answering. it is the t5 model @xcite fine-tuned on the squadv1 dataset @xcite by @xcite and available from github 2 . the limitation of the t5 model is that it was trained on english data and most importantly tuned on the squadv1 2 https://github.com/patil-suraj/ question_generation dataset which is in english. thus, this model only allows evaluation of mt systems translating from any language to english. to overcome that, we used the multilingual mt5 model @xcite and fine-tuned it on machine translation of squadv1 dataset. we exploited the existing translations into german @xcite and into czech @xcite which allows score translations into german (xx-de) and into czech (xx-cs) directions. due to time constraints we were not able to train qa and qg systems in other languages. figure 1 presents examples of extracted answers and generated questions. ## mqm scores recently, @xcite demonstrated that the wmt da method traditionally used for human evaluations has actually lower correlation with expert-based labels than the multidimensional quality metrics (mqm) scoring method developed in the eu qtlaunchpad and qt21 projects. following their findings, the wmt21 metric task will report the correlation with mqm labels in the official results. to provide a more complete picture of the performance of the mteqa metric, we also report correlation with the mqm assessments. the results are surprising and to a large extent unintuitive. metrics performing well in comparison with mqm are often bad in comparison with da. ## conclusions in this paper we described our submission to the wmt21 metrics shared task. we showed that the degree to which the mt output can be used to answer questions about the reference can be used as a proxy to evaluate the translation quality. we showed a gradual improvement of our submission. we examined a linguistically motivated way of extracting keyphrases from the sentence, and showed that it boosts both the segment-and system-level correlation with da human judgments. we were able to further boost the final performance by using the chrf metric to compare the reference and test answers. finally, we examined the performance against the mqm labels and compared the performance against the da labels.