Estimation vs Metrics: is QE Useful for MT Model Selection? (2020)

Abstract: This paper presents a case study of applying machine translation quality estimation (QE) for the purpose of machine translation (MT) engine selection. The goal is to understand how well the QE predictions correlate with several MT evaluation metrics (automatic and human). Our findings show that our industry-level QE system is not reliable enough for MT selection when the MT systems have similar performance. We suggest that QE can be used with more success for other tasks relevant for translation industry such as risk prevention.

## introduction machine translation quality estimation (qe) is a technique for predicting machine translation (mt) quality @xcite . as mt becomes the dominant tool in the translation industry, accurate estimation of the quality of mt output would be of great benefit to many business concerns such as budget allocation for human post-editing, estimating the usefulness of the mt output for gisting purposes, and selecting the best mt system out of a selection of systems. in addition to that, a reliable qe model would also help linguists make more efficient use of their time. as opposed to mt evaluation, where mt output is compared to one or several human reference translations, qe attempts to perform the much more challenging task of predicting mt quality in the absence of a reference translation. qe can be performed on a word, sentence or document level, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. and the output of a qe system is typically a score that is intended to correlate with a certain automatic or human mt evaluation metric. in this paper we present a case study where qe models were applied for the purpose of ranking different mt engines for a given document or text corpus. our approach is based on obtaining segment-level qe scores for all segments in the document/corpus and using the average to select the best mt system. we use two qe systems to score the output of different mt engines and compare the results of the qe model with several automatic mt evaluation metrics, which include the post-editing distance (ped), hter, bleu score @xcite , and weighted and unweighted sentence embedding similarity, as well as with human mt evaluation scores. we conclude the paper by reflecting on the usefulness of qe for mt engine selection, possible improvements to qe, and the limitations of our model and method. ## background and motivation the main motivation of exploring the qe method for mt model selection is the fact that we are often faced with a scenario where we need to choose the best mt system without a reference translation. the only existing method for this is involving human evaluators, which is, however, quite costly and requires more time. we are looking for a more cost-efficient and fast (almost immediate) way of deciding which mt system to use. qe is a very well explored topic in natural language processing given that predicting mt quality has clear practical benefits @xcite . multiple qe frameworks have been developed, some of which are open-source: quest++ @xcite , postech @xcite , qebrain @xcite , openkiwi @xcite , yisi @xcite and others. details about state-of-the-art qe tools are presented in detail in the corresponding wmt2019 shared task @xcite . despite the extensive research interest in qe, there is less information on how useful it actually is in specific commercial workflows. for example, @xcite compare the performance of various qe models using specific user business metrics, as well as implementation and computation cost. they demonstrate that the system with the highest performance can be also the most computationally expensive and simpler, faster systems can provide satisfactory results. more aspects of applying qe in commercial settings are discussed in @xcite and @xcite . we believe that evaluation of the performance of any nlp system must firstly take into account the end use of the system. in our case, the goal is to be able to automatically select the best mt engine out of two or more engines on a segment level in scenarios where a reference translation is not available. it is relevant to mention that our goal is not to fully replace automatic mt evaluation metrics; the findings from task 3 of the shared task on qe @xcite confirm that this is still a challenge. rather, the objective of this study is to investigate whether our qe systems are reliable enough to be used to select the best among multiple mt engines. segment-level qe is typically evaluated by calculating the correlation of the qe predictions with human judgement or one or several mt evaluation metrics, most commonly with hter @xcite , using the root mean square error (rmse) and/or mean absolute error (mae) @xcite . apart from that, @xcite describe a more fine-grained, linguisticallyinformed evaluation method which enables greater understanding of the behaviour of the qe system. for this study, in addition to the standard metrics, we utilize the metrics that correspond to our business goals. one of the most important metrics for us is the post-editing distance (ped), a standard mt quality metric used at the company and is the current industry standard. similarly to hter, ped represents post-editing effort in terms of the number of editing operations, but it is characterbased (while hter is word-based) and therefore more accurately reflects the effort expended in editing. another important metric for us is the bleu score @xcite , which is used mostly for mt development in order to measure improvement over a baseline. there are many valid criticisms of automatic mt evaluation metrics such as bleu @xcite , one of the most salient of which is the fact that they require one or more reference translations against which mt output is compared. however, a given sentence can have multiple correct translations depending on a certain context and end use, and for this reason reference-based metrics cannot always cover the entire space of valid translations for the sentence. for this reason, we also include human mt evaluation -direct assessment of the mt output by linguists, which is not reference-dependant. in addition, we experiment with text similarity metrics @xcite . in particular, we use word embedding similarity in order to reflect how semantically close the mt translation is to the reference translation. we produced sentence embeddings for the mt output and the reference translation, and calculated the cosine distances between these embeddings. cosine distances between sentence embeddings capture how closely the meanings of two sentences correspond in highdimensional vector space, and as such are less sensitive to the substitution of similar words in alternative translations. while this latter metric is insufficient on its own as a measure of translation quality (due to its insensitivity to word-order, among other reasons), we hypothesize that it may be a useful auxiliary metric. ## methodology our primary research goal was to investigate how well the scores from qe systems correlate with commonly-used metrics for evaluating translation quality, and based on these results understand how useful qe is for mt model selection in cases when a reference translation is not available. as a secondary goal, we also studied the impact of content domain on these correlations. domain is known to be a highly significant factor in the performance of mt engines, and we hypothesized that this would also be true of qe systems. therefore, for this study we used two qe models: one in a general domain (qe-gen) and the other in the domain of life sciences (qe-domain). below we present im-plementation details about these models. ## results we evaluated the two qe models on the corresponding data sets in terms of the model performance.