Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors (2021)

Abstract: This paper discusses a classification-based approach to machine translation evaluation, as opposed to a common regression-based approach in the WMT Metrics task. Recent machine translation usually works well but sometimes makes critical errors due to just a few wrong word choices. Our classification-based approach focuses on such errors using several error type labels, for practical machine translation evaluation in an age of neural machine translation. We made additional annotations on the WMT 2015-2017 Metrics datasets with fluency and adequacy labels to distinguish different types of translation errors from syntactic and semantic viewpoints. We present our human evaluation criteria for the corpus development and automatic evaluation experiments using the corpus. The human evaluation corpus will be publicly available upon publication.

## introduction most machine translation (mt) studies still evaluate their results using bleu @xcite because of its simple, language-agnostic, and model-free methodology. recent remarkable advances in neural mt (nmt) have cast an important challenge in its evaluation; nmt usually generates a fluent translation that cannot always be evaluated precisely by simple surface-based evaluation metrics like bleu. a recent trend in the mt evaluation is to use a large-scale pre-trained model like bert @xcite . @xcite proposed bert regressor based on sentence-level regression using a fine-tuned bert model, as an extension of their prior study using sentence embeddings @xcite . @xcite proposed bertscore based on hard token-level alignment using cosine similarity of contextualized token embeddings. @xcite proposed moverscore based on soft token-level alignment using word mover's distance @xcite . @xcite proposed bleurt that incorporates auxiliary task signals into the pre-training of a bert-based sentence-level regression model. these methods aim to evaluate a translation hypothesis using the corresponding reference with a high correlation to human judgment. the evaluation of this kind of mt evaluation, often called meta-evaluation, is usually based on some benchmarks. the meta-evaluation in the recent studies uses the wmt metrics task dataset consisting of human judgment on mt results. the human judgment is given in the form of human direct assessment (da) @xcite , a 100-point rating scale. the human da results are standardized into z-scores (human da scores, hereinafter) and used as the evaluation and optimization objective of regression-based mt evaluation methods. recent mt evaluation methods achieved more than 0.8 in pearson correlation on wmt 2017 test set foot_0 . however, @xcite reported a weaker correlation in low human da score ranges. such a finding suggests the difficulty of the mt evaluation on low-quality results. in this work, we focus on the problem in the evaluation of low-quality translations that cause serious misunderstanding. judging erroneous translations in the 100-point rating scale would be very difficult and unstable, because the extent of errors cannot be mapped easily into a one-dimensional space. suppose we are evaluating a translation hypothesis, (1) it is our duty to remain at his sides with its reference, it is not our duty to remain at his sides. 2 the difference in this example is just in one missing word not in the hypothesis, but it may cause a serious misunderstanding. such translation errors are considered as critical ones by professional translators. there are several metrics for translation quality assessment (qa) proposed in the translators' community, such as lisa qa metric 3 and multidimentional quality metrics (mqm) 4 . these metrics use a couple of error seriousness categories (minor, major, critical) in several viewpoints, such as mistranslation, accuracy, and terminology. the missing negation is a kind of critical error. nevertheless, most existing automatic mt evaluation metrics fail to penalize such errors. human da is also difficult from this viewpoint. suppose we have other translation hypotheses, (2) he bought some bags at a duty-free store. and (3) not is to duty remain it sides his at. for the same reference. we can easily identify these hypotheses are wrong. however, evaluating them together with (1) in the same 100-point rating scale by mapping these differences into one dimension is not trivial. this work pursues a classification-based human and automatic mt evaluation based on the multidimensional evaluation. current nmt technologies would still be far from the level of professional human translators but are also utilized in various applications. mt in practical applications should be evaluated as same as human translations by practical metrics, not just by incremental and engineering-oriented metrics like bleu. we propose a classification-based mt evaluation framework motivated by the discussion about critical errors. in human evaluation, we use conventional evaluation dimensions of fluency and adequacy @xcite and define several categories different from a conventional 1-5 likert scale. we developed a corpus with such additional annotations on wmt metrics dataset and found that human da scores penalize incomprehensible and unrelated mt hypotheses more than those with other critical errors that cause serious misunderstanding and contradiction. we then implemented a classification-based automatic mt evaluation using the corpus and conducted experiments on the ## related work mt evaluation has evolved along with the advance of mt technologies. @xcite reviewed some attempts of human evaluation and presented adequacy, fluency, and comprehension results in the early 1990s. the quality panel approach presented in their paper was motivated by the evaluation of human translations, but it was finally abandoned due to human evaluation difficulties. @xcite presented meta-evaluation of the mt evaluation in wmt shared tasks. according to the findings there, the wmt shared tasks had employed ranking-based human evaluation for a while. @xcite defined human-targeted translation edit rate (hter) that measures the translation quality by the required number of postedits on a translation hypothesis. @xcite and @xcite discussed the differences among those human evaluation approaches. @xcite proposed human da for the mt evaluation, and da has been used as standard human evaluation in recent wmt metrics tasks. there is another line of human mt evaluation studies focusing on semantics. @xcite proposed meant and its human evaluation variant hmeant based on semantic frames. @xcite proposed hume based on a semantic representation called ucca. this kind of fine-grained semantic evaluation requires some linguistic knowledge for annotators but enables explainable evaluation instead. however, the meaning of the sentence can be changed by small changes, as discussed later in section 3. looking at sub-structures and using their coverage in the mt evaluation may suffer from this problem. one recent approach has been proposed by popovic @xcite . her work analyzed the differences between comprehensibility and adequacy in machine translation outputs. the human annotations in her work are major and minor errors in comprehensibility and adequacy on words and phrases. these fine-grained annotations are helpful for detailed translation error detection. the focus of our work is different; we are going to develop sentence-level mt evaluation through simpler human and automatic evaluation schemes. in this work, we suggest revisiting the classification-based evaluation with fluency and adequacy, for absolute human and automatic evaluation. da-based human evaluation is beneficial in demonstrating the correlation with automatic evaluation metrics. however, it is not very intuitive in the evaluation of different kinds of translation errors. our work is also related to some studies using semantic equivalence and contradiction. bleurt @xcite employed nli in its pretraining phase. nli includes contradiction identification, which should also contribute to the mt evaluation. bleurt has revealed its advantage in the example shown in ## critical translation errors the main focus of this work is to penalize critical errors in translation hypotheses that cause serious misunderstanding. this kind of translation errors must be avoided, as well as possible. suppose we have some translation hypotheses with their reference, the pleiades cluster is situated 445 light-years from earth 5 . the translation hypotheses are artificial ones with some adversarial edits over the reference, as shown in the second column of we put automatic evaluation scores in the table using bleu-4 6 , chrf 7 , bertscore 8 , and bleurt 9 . hyp9 is correctly penalized by all the 5 this example is taken from the metrics dataset of wmt 2017. on the other hand, mt sometimes generates incomprehensible sentences with various kind of errors, even though nmt works much better than conventional statistical mt, especially in fluency. such incomprehensible translations are also very problematic as well as content errors in easy-tounderstand and fluent translations. however, it is not easy to penalize both of them in a single evaluation criterion. existing automatic evaluation methods often fail to penalize content errors, although they work well for incomprehensible and unrelated sentences, as revealed by the adversarial examples in ## human evaluation corpus we have developed a new human evaluation corpus from the viewpoints of fluency and adequacy. the evaluation corpus is available at github foot_2 under cc by-nc-sa 4.0 foot_3 . in this section, we present the details of the corpus. here, the human evaluation is designed in monolingual way; an mt hypothesis is evaluated against only its reference, supposing the reference is semantically equivalent to the source language input. we made a contract with a linguistic data development company to conduct the human evaluation foot_4 with three annotators who are native speaker ## experiments we conducted experiments using the evaluation corpus, to investigate the performance of automatic classification-based mt evaluation. ## conclusions in this paper, we present our approach to classification-based human and automatic mt evaluation, focusing on critical translation errors in mt outputs. we revisited the use of fluency and adequacy metrics with some modifications on evaluation criteria, motivated by our thoughts on the critical content errors. we developed a human evaluation corpus based on the criteria using the wmt metrics dataset, which will be publicly available upon publication. our corpus analyses revealed the human da penalizes unrelated and incomprehensible hypotheses much more than contradiction and other critical errors in the content. we also conducted automatic mt evaluation experiments using the human evaluation corpus and achieved around 60% classification accuracy both in fluency and adequacy. our future work includes further development of human evaluation corpora that are not limited to wmt metrics data, and data augmentation methods to tackle the label imbalance problem. it is also promising to apply the classification-based automatic mt evaluation to the neural mt training.