Informative Manual Evaluation of Machine Translation Output (2020)

Abstract: This work proposes a new method for manual evaluation of Machine Translation (MT) output based on marking actual issues in the translated text. The novelty is that the evaluators are not assigning any scores, nor classifying errors, but marking all problematic parts (words, phrases, sentences) of the translation. The main advantage of this method is that the resulting annotations do not only provide overall scores by counting words with assigned tags, but can be further used for analysis of errors and challenging linguistic phenomena, as well as inter-annotator disagreements. Detailed analysis and understanding of actual problems are not enabled by typical manual evaluations where the annotators are asked to assign overall scores or to rank two or more translations. The proposed method is very general: it can be applied on any genre/domain and language pair, and it can be guided by various types of quality criteria. Also, it is not restricted to MT output, but can be used for other types of generated text.

## introduction while automatic evaluation metrics are very important and invaluable tools for rapid development of machine translation (mt) systems, they are only a substitution for human assessment of translation quality. various methods have been proposed and used for the human evaluation of mt output, such as @xcite @xcite @xcite @xcite , and all of them are essentially based on some of the following three quality criteria: adequacy (accuracy, fidelity), comprehensibility (intelligibility) and fluency (grammaticality). adequacy measures how well the meaning of the original text is conveyed to the translated text. comprehensibility reflects how well a reader is able to understand the translated text without access to the original text. fluency describes the grammar of the target language in the translated text. the choice of criteria often depends on the task and on the purpose of the translation: for example providing translations to users requires high comprehensibility and adequacy, comparing different mt systems is based mainly on adequacy, etc. regardless of the quality criterion, the annotators are usually asked to assign an overall quality score for the given mt output, or to rank two or more competing outputs from best to worst. one advantage of direct scoring over ranking is that the resulting annotations include not only the information that one mt output is better than another, but also the degree to which that output was better than the other. also, it gives an estimation about the absolute quality of each mt output. another advantage is that direct estimation of translation quality extends the usefulness of the annotated data to other tasks such as quality estimation. still, neither of these two annotation methods provides any details about actual errors and problems. the usual way to overcome this drawback is to perform error classification, where the evaluators are asked to mark each translation error and assign an error tag from a set of predefined categories. however, this approach requires much more time and effort, both from annotators as well as from organisers (to define an appropriate error taxonomy which is not a trivial task, to prepare clear guidelines for each error class, and to train the annotators). in the proposed approach, the evaluators are asked to mark problematic parts in translation without assigning any scores or error labels. the advantage of such approach is two-fold. first, it is more informative than assigning overall scores because the actual problematic words/phrases/sentences are marked and they can be further used for more detailed analysis. the results of one possible analysis, namely comparison of comprehensibility and adequacy issues, are presented in @xcite . second, the annotation process does not require any additional effort in comparison to assigning scores or ranking, which is much less effort than for error classification. also, overall scores (in the form of percentage of problematic words) can be automatically extracted from annotated text. although the described experiment was carried out on user reviews (a case of "mid-way" genre between formal and informal written language) translated into croatian and serbian (a case of mid-size lessresourced morphologically rich european languages), the method can be applied on any genre/domain and language pair. two quality criteria were used in this work, comprehensibility (monolingual) and adequacy (bilingual), but the method can be guided by any other criterion (such as fluency). in addition, the method is not necessarily restricted to evaluation of mt output, it can be applied on any type of generated text. ## related work the first report about manual evaluation of machine translation @xcite defines "intelligibility" (comprehensibility) and "fidelity" (adequacy) as the two major characteristic of a translation. for each of those two criteria, the evaluators assigned overall quality scores on 9 point scales. the arpa mt initiative @xcite defines "adequacy", "fluency" and "comprehension" as a standard set of concepts for mt evaluation, and the evaluators are instructed to assign overall scores based on those three criteria. years later, the first wmt (workshop/conference on machine translation) translation shared task in 2006 foot_1 as well as the subsequent task in 2007 adopted adequacy and fluency as official metrics @xcite . the participants in the shared task are asked to assign adequacy and fluency scores on 5 point scales to each submitted mt output. later on, a binary ranking of two mt outputs was proposed @xcite , and in a slightly changed form (comparing up to five outputs instead of two) became the official metric at @xcite . it was reported that it required less effort and showed better inter-annotator agreement than adequacy and fluency scores, and it remained the official wmt metric until 2016. the quality critera for ranking were never explicitly defined, but it was implicitly based on a combination of adequacy and fluency. another method for assigning quality scores, continuous direct assessment @xcite , does not use discrete scales but assigns a continuous score between 0 and 100. both adequacy and fluency were investigated as the guiding criterion, and it was concluded that the best option was to be guided by adequacy and to use fluency only as auxiliary criterion for adequacy ties @xcite . continuous direct assessment eventually replaced ranking at the wmt shared tasks in 2017 @xcite and is still used as the official wmt metric @xcite . some publications dealt with evaluating user reviews: the annotators were asked to assign scores on 5 point scale for comprehensibility and scores on 2 point scale ("true" or "false") for fidelity @xcite . those annotations were later used for quality estimation of these two evaluation criteria @xcite . other publications propose alternative methods for assessing comprehensibility, namely question answering @xcite as well as filling gaps as its cheaper version @xcite . nevetheless, none of these methods provide information about actual parts of the translation which are problematic in terms of the given quality criterion. our method overcomes this drawback by marking actual issues. furthermore, our method is much less demanding than error classification, where the evaluators are asked to assign error classes according to a predefined typology, for example @xcite , the mqm scheme foot_2 @xcite , etc. although our method does not involve categorisation of marked issues, the annotated texts can be further used for many tasks with different error typologies (such as classifying translation errors, identifying linguistic phenomena causing the errors, concentrating on particular type(s) of erros and/or of phenomena, etc.) furthermore, categorisation is then performed on already marked errors which reduces effort in comparison to finding and classifying errors from scratch. ## evaluation method the novelty of the proposed method is that the evaluators are not assigning any scores, nor classifying errors, but marking all problematic parts of the text (words, phrases, sentences). the method can be seen as a "mid-way" between overall assessment and error classification. the advantage over overall assessment is that our annotation method provides much more information without requiring additional effort: assigning overall scores is guided by issues/errors in the translation, and our method requires direct marking of these issues instead of thinking about an overall score based on them. our method also provides overall scores, by counting all marked words. the advantage over error classification is that it is much less demanding while still informative -the annotations can be further used for different kinds of analyses (with reduced effort in comparison to analysing from scratch), such as error classification, identification of problematic (linguistic) phenomena, etc. the proposed evaluation process makes use of two well-known quality criteria, namely comprehensibility and adequacy which were already used for evaluating user reviews but in the form of assigning scores @xcite . comprehensibility reflects the degree to which a translated review can be understood, and adequacy reflects the degree to which the translation conveys the meaning of the original text in the source language. the annotators were asked to distinguish two levels of issues for each criterion: major issues (e.g. incomprehensible/not conveying the meaning of the source) and minor issues (e.g. grammar or stylistic errors/not an optimal translation choice for the source). it should be stressed that comprehensibility is not fluency -a fluent text can be incomprehensible (for example "colorless green ideas sleep furiously."), and vice versa (for example "all these experiment was carry out this year."). the evaluation procedure consisted of two independent subsequent tasks: the evaluation was performed by computational linguistics students and researchers, fluent in the source language and native speakers of the target language. in total, thirteen evaluators participated in the study, of which three have had experience with mt. the translation outputs were given to the evaluators in the form of google doc, and they were asked to mark major issues with red colour and minor issues with blue colour. in addition to general definitions of comprehensibility and adequacy, the following detailed guidelines were given to the evaluators: ne shvaćajte ih ako udarite u tešku torbu. don't understand them if you hit in heavy suitcase. oni jednostavno neće zaštititi / podržavati vaše zglobove ili ručne zglobove. they simply won't protect / support your joints or hand joints. (b) adequacy source if you wrap your wrists properly, you'll see these are both too narrow and too short, way too short. do not get these if you are hitting the heavy bag. they just won't protect/support your wrists or knuckles. mt output ako pravilno zavijate zglobove, vidjet ćete da su i preuska i prekratka, prekratka. + gloss if properly you are wrapping wrists, see you will that they are both too narrow and too short, too short. ne shvaćajte ih ako udarite u tešku torbu. don't understand them if you hit in heavy suitcase. oni jednostavno neće zaštititi / podržavati vaše zglobove ili ručne zglobove. they simply won't protect / support your joints or hand joints. an example of a short review annotated for comprehensibility (above) and adequacy (below) can be seen in each annotator performed first the monolingual comprehensibility task and afterwards the bilingual adequacy task. in total, each annotator evaluated about 5000 (non-tokenised) words for each of the two quality criteria. all annotators managed to complete both tasks in 3 to 7 (non-consecutive) hours. each mt output was annotated by two annotators in order to obtain more reliable annotations and estimate inter-annotator agreement. it should be noted that in the view of recent findings and recommendations regarding manual mt evaluation @xcite , the evaluation is carried out on the review ("document") level, and not on the sentence level. in this way, it was ensured that the annotators were able to spot context-dependent issues. ## data sets we have been working with two types of publicly available user reviews: in total, 28 imdb and 122 amazon reviews (16807 untokenised english source words) are covered in this evaluation. average length of the selected imdb reviews was 12.4 sentences and 214.3 words, whereas for amazon reviews was 6.5 sentences and 87.7 words. the selected english user reviews were then translated into croatian and into serbian. the goal of the experiment is not to evaluate or compare particular mt systems, but to explore a new evaluation strategy. for this purpose, mt outputs 6 of three on-line systems were used: google translate 7 , bing 8 and amazon translate 9 . from the selected english reviews, 900 mt outputs were generated (150 reviews were translated by three mt systems into two target languages, thus six translations for each of the 150 @xmath0, and 270 of them were included in the manual evaluation. more details about the data can be seen in ## conclusions this work presents a novel method for manual evaluation of mt output where the evaluators are not assigning any scores, nor classifying errors, but are marking actual problematic parts of the text (words, phrases, sentences) in the translation. the proposed method does not provide only overall scores by counting the assigned tags, but also enables further detailed analysis of the annotated texts. the annotators were asked to distinguish only two levels of issues, major and minor. since it does not include fine-grained error classification, inter-annotator agreement is high -annotators agree on about 70-80% of issues on the word level. many disagreements are due to assigning different spanssome annotators preferred to mark whole phrases/sentences whereas others preferred to chose one or two words. a detailed analysis of disagreements which could represent an interesting direction towards better understanding the evaluation process is also possible on texts annotated in the described way. the evaluation was carried out on english user reviews translated into croatian and serbian, and was based on two quality criteria: comprehensibility and adequacy. hovever, the method can be applied on any genre/domain and language pair, and can be guided by any other quality criterion (such as fluency). it could also be used for comparing ("ranking") two foot_4 mt outputs by marking differences between them as "better" or "worse". generally, the method can be applied to any type of generated text. future evaluations should include more domains and language pairs, and take into account the most important lessons learnt from the annotators' feedback: show the original-translation pairs on the sentence level (instead of showing whole translated review ("document") below the whole original review) in order to reduce cognitive load, and clearly explain in the guidelines that comprehensibility does not include figuring out the "true" meaning of the original text. apart of that, it might be worth to develop a more sofisticated annotation tool, or to extend an existing tool. furthermore, it would be good to directly compare the proposed method with conventional types of manual evaluation, namely ranking, assigning overall scores, or even post-editing. also, it might be worth to explore how to end up with a final quality score, because simple average overall error count might not be the optimal solution.