Just Ask! Evaluating Machine Translation by Asking and Answering Questions (2021)

Abstract: In this paper, we show that automatically-generated questions and answers can be used to evaluate the quality of Machine Translation (MT) systems. Building on recent work on the evaluation of abstractive text summarization, we propose a new metric for system-level MT evaluation, compare it with other state-of-the-art solutions, and show its robustness by conducting experiments for various MT directions.

## introduction the goal of automatic machine translation (mt) evaluation is to automatically evaluate the output quality produced by mt systems. metrics used for this task assign a score by comparing the mt output to either a reference translation or to the source sentence (the latter is called quality estimation). the main indicator that is used to assess the performance of a specific metric is the correlation with human judgement computed for outputs from several systems. it was recently shown that metrics based on contextualized embeddings, such as yisi @xcite or esim @xcite , are able to achieve better performance than the most widely used bleu @xcite . in this paper, we propose a new method for automatic evaluation of mt systems -mteqa foot_0 (machine translation evaluation with question answering), building on previous works on evaluating abstractive summaries. we build upon the fact that state-of-the-art (neural) mt systems tend to produce a fluent output but sometimes fail in adequacy of the translation. we leverage the recent progress in question generation (qg) and question answering (qa) to formulate and answer human readable questions about the mt system output. our experiments show that the effectiveness of the proposed metric is comparable to performance of other automatic metrics, while considering only a certain amount of information from the whole translation. we also examine the robustness of the metric by considering several translation directions and target languages. the remainder of this paper is structured as follows. in section 2, we introduce relevant research on question-based evaluation. in section 3, we describe our metric in detail. in section 4, we present and discuss the results of our experiments including s the influence of different human scoring methods. section 5 presents conclusions. ## related work metrics that are most widely used for automatic evaluation of mt outputs produce a score by comparing surface-level forms of hypothesis and reference translation. the most dominant one, bleu @xcite , is a version of n-gram precision calculated by averaging over different values of n with penalization for overly short translations (brevity penalty). another one, chrf @xcite , considers the character-level n-grams, making it possible to reward partial token matches. the standardised implementation provided in the sacrebleu foot_1 package takes care of pre-processing and enables direct comparison between mt outputs. recently, various works (e.g., @xcite @xcite ) explored the usage of contextualized word-level or sentence-level embeddings to compare the numerical representations of reference and hypothesis. such metrics enable explicit regression towards the desired humanproduced labels. ## mteqa our idea of evaluating mt quality by asking and answering questions is based on the assumption that a good translation should preserve all of the key information that one can extract from the reference. we propose to use a question answering framework as the proxy to measure this. to check whether a piece of information is preserved, we automatically generate pairs of a question and its (gold-standard) answer from the reference translation and employ a question answering system to provide a new (test) answer given the question and the mt output (translation) used as the context. the generated (test) answer is then compared to the gold-standard answer. we assume that if it was possible to answer a question looking only at the reference, it should also be possible to answer this question looking only at the mt output and that the two answers should be identical or very similar. in principle, the proposed mteqa metric requires solving the following tasks: 1) answer extraction identifies the key information in a sentence (keyphrases) which should be also present in the mt output. this extraction can be treated in a hierarchical/nested manner. for instance, given the sentence "today for dinner i had an organic pasta with garlic.", the question "what did you have for dinner today?" can be correctly answered by all the following phrases pasta, organic pasta and organic pasta with garlic. thus, answer extraction is performed first and the questions are generated afterwards for each of the answers independently. the same question can be paired with multiple (nested) answers which allows capturing a partial correspondence. ## experiments we evaluate the proposed mteqa metric using the submissions to the wmt20 news translation task @xcite and their (direct) human assessments (da). for each of the mt systems participating in the task, we compute a single score as the average of segment-level scores and report the system-level pearson correlation with the human assessment. we report individual results for selected translation directions into english plus aggregated results (averages) for all to-english directions which were part of the wtm20 metric task @xcite ) evaluation campaign foot_3 . ## conclusions in this paper we introduced a new metric for automatic evaluation of machine translation systems. we showed that the degree to which the mt output can be used to answer questions about the reference can be used as a proxy to evaluate the translation quality. we proved that our metric is robust by conducting experiments over multiple translation directions. we examined a linguistically motivated way of extracting key phrases from the sentence and showed that it boosts the final performance. we checked the influence of various word-level comparison metrics used to compare the test and goldstandard answers, and reported how it affects the correlation with human scores. in our work, we focused on translation directions into english. the only limiting factor in applying our metric to other translation directions is the availability of question generation and question answering systems in a given language. however, automatic translation of squad can be an effective way to obtain data for training such systems. finally, we examined the performance against the mqm labels and compared the performance against the da labels. while for the da labels our metric performs close to state-of-the-art solutions, for the mqm labels there is a noticeable drop in performance. in the future, we plan to examine the crosslingual approach -instead of generating questions and answers from the reference, one may instead use the source directly.