Relations between comprehensibility and adequacy errors in machine translation output (2020)

Abstract: This work presents a detailed analysis of translation errors perceived by readers as comprehensibility and/or adequacy issues. The main finding is that good comprehensibility, similarly to good fluency, can mask a number of adequacy errors. Of all major adequacy errors, 30% were fully comprehensible, thus fully misleading the reader to accept the incorrect information. Another 25% of major adequacy errors were perceived as almost comprehensible, thus being potentially misleading. Also, a vast majority of omissions (about 70%) is hidden by comprehensibility. Further analysis of misleading translations revealed that the most frequent error types are ambiguity, mistranslation, noun phrase error, word-by-word translation, untranslated word, subject-verb agreement, and spelling error in the source text. However, none of these error types appears exclusively in misleading translations, but are also frequent in fully incorrect (incomprehensible inadequate) and discarded correct (incomprehensible adequate) translations. Deeper analysis is needed to potentially detect underlying phenomena specifically related to misleading translations.

## introduction while automatic evaluation metrics are very important and invaluable tools for rapid development of machine translation (mt) systems, they are only a substitution for human assessment of translation quality. various methods have been proposed and used for the human evaluation of mt quality by assigning overall scores to @xcite @xcite @xcite @xcite , and all of them rely on at least one of the three translation quality criteria: comprehensibility (comprehension, intelligibility), adequacy (fidelity, semantic accuracy), and fluency (grammaticality). comprehensibility reflects the degree to which a translated text can be understood, adequacy reflects the degree to which the translation conveys the meaning of the original text in the source language, and fluency reflect the grammar of the translated text. the raters are usually asked to assign an overall score for the given translation criterion. in order to get more details about translation performance, error classification and analysis emerged in the field of mt @xcite @xcite . however, there is less work dealing with human perception of mt quality and errors. for statistical phrase-based mt systems (smt), @xcite and @xcite were identifying error types which are mostly disliked by readers. in the last five years, systems based on artificial neural networks (nmt) have become the new state of the art. several evaluation studies, such as @xcite @xcite reported that these systems are able to produce more fluent and readable translations, but that they are still sufferring from adequacy issues. in addition, many participants mentioned that good fluency of nmt outputs makes it more difficult to spot adequacy errors such as omissions or mistranslations. such "fluently inadequate" errors may mislead readers into trusting the content based on fluency alone, especially when surrounded by fluent and adequate parts of a text @xcite . automatic identification of such errors for both smt and nmt systems has been investigated in @xcite and it is confirmed that these errors appear much more often in nmt system. to the best of our knowlegde, comprehensibility, while being a very important translation quality factor, has not been investigated in depth yet. it should be stressed that comprehensibility is very different from fluency -a fluent text can be incomprehensible (for example "colorless green ideas sleep furiously."), and vice versa (for example "all these experiment was carry out this year."). our main research questions are: rq1 are there "comprehensible inadequate" translations which are misleading human readers so that they fully trust the mt output despide adequacy errors? in other words: how many adequacy errots are perceived as comprehensible? rq2 if the answer to the rq1 is "yes", which types of translation errors are mainly related to these translations? as a first step, a group of evaluators annotated problematic parts of the given machine translated text. they were not asked to assign any error labels, only to mark the parts of the text which they perceived as problematic for the given translation criterion. they first annotated all comprehensibility issues, and after about two weeks, all adequacy issues. for each criterion, they were asked to distinguish major and minor issues. we then analysed all major issues in order to examine relations between comprehensibility and adequacy and identify error types. the analysis was carried out on english user reviews (as a case of "mid-way" genre between formal and informal written language) translated into croatian and serbian (as a case of mid-size less-resourced morphologically rich european languages). it is worth noting that the aim of this work is not to compare mt systems, nor to estimate their overall performance for the given language pairs and domain in order to potentially improve them. the aim of this work is to explore relations between two aspects of human perception of translation quality. ## related work lot of research on mt evaluation deals with classification and analysis of mt errors, for example @xcite @xcite @xcite . few papers deal with human perception of these errors, but neither of them defines precisely which criterion is the translation quality based on. @xcite uses conjoint analysis to investigate user preferences for error types of smt systems. first, the errors in mt outputs were annotated, and then mt outputs with different error types were given to the crowd evaluators. they were asked to choose the mt output which they like best and to give the reason for their preference. one of the findings is that the frequencies of error types are not related to the user preferences. the most dispreferred error type was word order error, although it was the least frequent one. it was followed by word sense errors (ambiguity), then morphological errors (most frequent ones), whereas errors in function words were the most tolerable. a similar study on smt outputs based on linear mixed-effects models is described in @xcite , aiming to estimate the impact of different translation errors to the overall translation quality. for each mt output, experts were asked to assign a score on a 5-point scale while other experts annotated the errors. the results confirmed that the frequency of errors of a given type does not correlate with human preferences. another finding is that omissions and mistranslations have the highest impact on the overall translation quality. in addition, it is observed that certain combinations of errors have less impact than each of those error types ocurring in isolation. in the last few years, with the emergence of nmt systems which generate much more fluent and readable outputs but still are prone to adequacy errors, some studies have concentrated on investigating adequacy and fluency errors. @xcite carried out a survey to determine how users respond to good translations compared to translations that are either adequate but not fluent, or fluent but not adequate. this study showed that users strongly disliked disfluent translations, but were much less bothered with adequacy errors. therefore, it was concluded that fluent translations with adequacy errors can mislead the reader to trust an incorrect meaning. automatic identification of these misleading "fluently inadequate" translations using source text, reference human translation and mt output was proposed in @xcite , and the main finding was that nmt systems generate more misleading translations than smt systems. however, the question about how many adequacy errors are actually hidden by fluency remained open. to the best of our knowledge, the relation be-tween adequacy and comprehensibility has not been investigated yet. comprehensibility, similarly to fluency, has an immediate effect on the reader, while adequacy problems can be perceived only if the reader has access to the source text or to a correct translation to find out that the meaning is wrong. this means that comprehensibility may have the same misleading effect making the reader accept an incorrect information. on the other hand, because comprehensibility is different than fluency (fluent sentences can be incomprehensible and vice versa), the effects might be different. ## data set our analysis has been carried out on written usergenerated content, namely user reviews. two types of publicly available user reviews written in english have been analysed: imdb movie reviews foot_0 @xcite and amazon product reviews foot_1 @xcite . a set of those user reviews was translated into croatian and serbian, two closely related mid-size less-resourced morphologically rich european languages. the reviews were translated foot_2 by three on-line systems: google translate foot_3 , bing foot_4 and amazon translate foot_5 . the analysed text consists of a mixture of mt outputs from the three systems including 222 translated reviews consisting of about 1500 sentences (segments) and 19837 untokenised words in total. this text was then given to the annotators to mark comprehensibility and adequacy issues, and the process is described in details in the next section. the annotated text is publicly available under the creative commons cc-by licence. foot_6 ## analysis of comprehensibility and adequacy issues it can be noted that the overall amounts of comprehensibility and adequacy issues are similar. however, it does not necessarily mean that the majority of words is perceived both as incomprehensible and inadequate. therefore, we examined major comprehensibility and adequacy issues in depth. 9 raw counts divided by the total number of words in the text including those without issues and the omission marks "xxx" quality aspect grade raw count %words comprehension major 1887 9.5 minor 2673 13.5 adequacy major 1963 9.9 minor 2539 12.8 ## conclusions this work presents the results of a detailed analysis of translation errors perceived by readers as major comprehensibility and/or major adequacy issues. the main finding is that good comprehensibility, similarly to good fluency, can mask a number of adequacy errors. of all major adequacy errors, 30% were fully comprehensible, thus fully misleading the reader to accept the incorrect information. an-incorrect information is: discarded information is: accepted potentially accepted discarded almost correct correct only amaj amaj+cmin amaj+cmaj cmaj+amin only cmaj ambiguity 24.0 ambiguity 24.8 ambiguity 26.7 ambiguity 16.4 propagation 33.3 mistranslation 6.0 mistranslation 8.9 mistranslation 8.2 noun phrase 8.4 ambiguity 14.4 word-by-word 5.0 noun phrase 8.4 noun phrase 6.9 mistranslation 5.8 noun phrase 7.0 noun phrase 4.4 untranslated 8.2 untranslated 6.7 {noun case 5.8} word-by-word 4.1 source spelling 3.8 word-by-word 5.4 word-by-word 5.5 word-by-word 5.8 mistranslation 3.7 subject-verb 3.8 (subject-verb 3.2) (subject-verb 4.5) subject-verb 4.9 (subject-verb 2.0) untranslated 3.8 (source spelling 2.4) (source spelling 3.5) {pos ambiguity 3.5} (source spelling 1.2) further analysis of those misleading translations was carried out in order to find out which types of translation errors are perceived in this way. ambiguous words, mistranslations, noun phrases, untranslated words, word-by-word translations, subject-verb agreement and spelling errors in the original text were identified as the most frequent error types in misleading translations. although noun phrase problems are typical for slavic languages and errors in the source text are typical for user generated content, the rest of the error types is rather general. however, none of these error types is exclusively related to misleading translations, but are also frequent in fully incorrect (incomprehensible inadequate) and discarded correct (incomprehensible adequate) translations. deeper analysis is needed to potentially detect underlying phenomena specifically related to misleading translations. apart from the obvious directions for future work such as analysing more texts and including more language pairs and domains, the presented analysis can be expanded in the following directions: including fluency in the analysis, including all minor issues in the analysis, further analysis of omissions, and investigating co-ocurrences of different error types. another experiment could include monolingual annotators for comprehensibility in order to completely eliminate potential influence of knowlegde of the source language.