On nature and causes of observed MT errors (2021)

Abstract: This work describes analysis of nature and causes of MT errors observed by different evaluators under guidance of different quality criteria: adequacy and comprehension and and a not specified generic mixture of adequacy and fluency. We report results for three language pairs and two domains and eleven MT systems. Our findings indicate that and despite the fact that some of the identified phenomena depend on domain and/or language and the following set of phenomena can be considered as generally challenging for modern MT systems: rephrasing groups of words and translation of ambiguous source words and translating noun phrases and and mistranslations. Furthermore and we show that the quality criterion also has impact on error perception. Our findings indicate that comprehension and adequacy can be assessed simultaneously by different evaluators and so that comprehension and as an important quality criterion and can be included more often in human evaluations.

## introduction and related work machine translation (mt), like many other natural language generation tasks, is difficult to evaluate because there is no single correct output for a given input: for each source text, there is a large set of possible correct translations. therefore, while costly both in time and resources, human evaluation is required to provide a reliable feedback for measuring mt quality and progress, as well as to serve as a gold standard for development of automatic evaluation metrics. while better and better automatic metrics are constantly emerging @xcite , many of them being based on semantic word representations (embeddings), all of them represent only an approximate substitution for human assessment of translation quality. various methods have been proposed and used for the human evaluation of mt output from the beginning of mt until now @xcite @xcite @xcite @xcite @xcite , and all of them are essentially based on some of the following three quality criteria: adequacy (how much meaning is preserved), comprehensibility (how comprehensible/readable the translation is) and fluency (grammar of the target language). the evaluators are usually asked to assign an overall quality score for the given mt output @xcite @xcite @xcite or to rank two or more competing outputs from best to worst @xcite @xcite . for assessing comprehension, question answering @xcite and filling gaps @xcite were explored, too. recently, evaluators have been asked to highlight the observed translation errors @xcite . in order to get more details about the actual errors, error classification according to a predefined error scheme is often performed. the mostly applied schemes have been the one proposed by @xcite , and the mqm scheme 1 @xcite in recent years @xcite . another method to better understand particular strengths and weaknesses of mt systems is to identify nature and causes of the errors in form of linguistically motivated phenomena which, although related, often go beyond the usual error types. this type of analysis is being increasingly employed in the last years in order to better understand the ocurring errors @xcite and also to create specialised test sets ("challenge test sets" or "test suites") in order to perform more focussed evaluation procedures on identified phenomena @xcite @xcite . this work goes in this direction, but in a slightly different way: we do not try to identify the phenomena from scratch, but from translation errors already observed and highlighted by several evaluators @xcite . the error marking was not guided by any pre-defined error scheme, so that the evaluators had more freedom in annotating errors than in typical error classification tasks such as mqm. we analysed the nature of these errors by tagging them with possible causes and/or plausible explanations of their origin (referred to as "phenomena"). the definition of these phenomena is based both on general linguistic knowledge as well as on phenomena related to the (machine) translation process. we did not have any pre-defined scheme for the phenomena, but we started by looking into errors and identifying the phenomena on the fly. it is worth noting that we did not create any test suite -we do not know how many instances of each of the identified phenomena exists in the data in total, nor how many of them are correctly translated. we only analyse the observed translation errors. nevertheless, our findings can be inspiring and useful for future work on creation of test suites. the main goal of this work is to identify nature and causes of translation errors perceived by a set of evaluators and to get a better insight about the underlying phenomena and their impact on translation quality. in addition, we investigate the perception of major and minor errors, and also explore perception of errors for two different quality criteria: adequacy and comprehension. we used two publicly available data sets containing @xmath0, @xmath1. we first identified a set of 26 underlying phenomena around these errors and then analysed them. ## data sets we worked on two publicly available data sets with highlighted mt errors: one provided by dublin city university (dcu) 2 and one provided by heidelberg university (hu). 3 while both data sets contain mt outputs with highlighted translation errors, there are several important differences between them. dcu data set this data set was created for purposes of mt evaluation @xcite . the set consists of english user reviews translated into croatian and serbian. for each of the target languages, five different mt systems were used: three online systems (amazon, bing and google) and two in-house systems based on the sockeye 4 @xcite implementation. in total, the data set contains outputs of ten different mt systems. two quality criteria were used for highlighting errors: adequacy and comprehension. an important difference between the two (apart from the definition) which can lead to differences 1 http://www.qt21.eu/mqm-definition/definition-2015-12-30.html 2 https://github.com/m-popovic/qrev-annotations 3 https://www.cl.uni-heidelberg.de/statnlpgroup/humanmt/ 4 https://github.com/awslabs/sockeye in perception of errors is that seeing the source text was required for adequacy while seeing the source text was forbidden for comprehension. for both quality aspects, the evaluators were asked to concentrate on problematic parts of the text and to highlight them. they were also asked to distinguish between major and minor errors. all translations were evaluated in context -the evaluators were seeing entire reviews. in total, 15 evaluators participated in the annotation. the largest part of the text is annotated by two evaluators, while a small part of the text (about 40 sentences) is annotated by three or four evaluators. nothing is annotated by a single evaluator. inter-annotator agreement in terms of krippendorf's α is 0.61 for adequacy errors and 0.51 for comprehension errors. hu data set this data set was not created for purposes of mt evaluation, but for improving an nmt system by giving it feedback about errors @xcite . the set consists of english ted talks translated into german by one mt system, an in-house system based on the joey nmt foot_0 @xcite implementation. a very important difference in comparison to the dcu data set is that no specific quality criterion was used: the evaluators were only asked to "highlight the errors". usually, such "generic" criterion represents a mixture of adequacy and fluency. also, they were not asked to distinguish between major and minor errors. another very important fact is, since the data set is created in order to improve a system, and the used loss function did not support omissions and reordering errros, the evaluators are specifically asked not to highlight these two types of errors. as for context, translated sentences were judged in isolation, however in consecutive order as they appeared in the original documents so that a reasonable amount of context was provided. ten evaluators participated in this annotation, although the largest part of the text is annotated by a single evaluator. eleven sentences are, however, annotated by all ten evaluators and the reported krippendorf's α is 0.201. an overview of the two data sets together with the overal percentage of highlighted words is presented in ## identified phenomena the errors in the described data sets were analysed in the following way: they were tagged as a particular phenomenon if 1) they were marked by at least one evaluator 2) it was possible to define a plausible cause and/or explanation for their origin. in order to motivate and facilitate future work of creating test suites and getting ideas for potential improvements of mt systems, we also tagged all corresponding english words. the analysed data sets with phenomena tags are available together with the original dcu foot_1 data set. the identified phenomena are different by their nature: some of them are equivalent to the typical error classes (such as "mistranslation", "tense/aspect/mood") while some are going far beyond that, often bringing on several different interwining types of errors. some of them involve single words, while others might involve a large group of words, even entire sentences. for the phenomena with larger spans, we tagged all consecutive words although not necessarily all those words are marked as errors. a typical example is negation where all words within the negation span were considered as "negation" although the evaluators might perceive only some of the words as problematic. in total, we identified 26 phenomena which will now be described and explained in alphabetical order. its translation can result in different types of often interwined errors (lexical, morphological, omissions, order) because formation rules for serbian and croatian are rather different than for english and there is often no unique solution. and even though formation rules in german are similar to the english ones, translation errors are still occurring. the examples in pos ambiguity a source word which can be interpreted as different pos tags. for example, the english word "works" can be plural of the noun "work" or third person singular of the verb "to work". rephrasing refers to a sequence of source words which is not translated properly for some of the following reasons or their combination: 1) the choice of each target language word looks random, both lexically and morphologically, without taking any context into account 2) rephrasing is needed in the target language but the translation follows the structure of the source language 3) rephrasing is not needed in the target language but is applied 4) rephrasing is needed in the target language but it is incorrectly applied. the phenomenon also comprises incorrect translation of multi-word expressions and collocations. it is usually manifested by several consecutive different but interwined types of errors, such as morphological (case, gender, person/tense/mood/aspect, etc.), lexical (ambiguity, mistranslation, multi-word expression), word order, etc. in all examples except the fourth one, the translation output is rather literal, namely the system failed to apply rephrasing and the output follows the structure of the source text. in the fourth example, however, the system rephrased the source text, but the applied rephrasing was incorrect and changed the meaning. ## distribution of the observed errors over the identified phenomena once the phenomena were identified and tagged, for each of them the contribution was calculated as percentage of observed errors related to it. due to the differences between the two data sets described in section 2 as well as the two different quality criteria in the dcu data set, the results in the numbers should be interpreted as follows: the first number in the first column means that from all highlighted adequacy errors in the dcu set, 17.6% are related to rephrasing, 11.2% are related to an ambiguous source word, 7.67% are related to a noun phrase, etc. the other columns are to be interpreted in the same way (second column: "from all highlighted comprehension errors in the dcu set", third column: "from all highlighted errors in the hu set"). phenomena contributing with at least 2% of highlighted words are shown in bold. to errors which could not be interpreted by any particular phenomenon, a tag "none" was assigned. a number of these errors is related to individual preferences of different annotators, and therefore is less frequent in the hu corpus which was mainly annotated by a single evaluator. some of these words are marked due to "error propagation", when several consecutive words are marked although only one of them is actually an errors. this effect is much stronger for comprehension, because adequacy is guided by the source text. rephrasing errors seem to be partly dependent on mt system: some systems tend to stay close to the source text (generating overly literal translations) while others tend to diverge from the source (generating incorrect rephrasings). these effects should be investigated further in more details, also by creating appropriate test suites. as for ambiguous source words, our analysis confirmed that they represent a challenge for modern nmt systems. several test suites have already been developed @xcite @xcite , but creating more test suites covering different types of ambiguous words and various language pairs would be certainly beneficial. it should be noted that, while translation of ambiguous words can be improved by context-aware ("document-level") nmt systems, incorporating external context often could be more helpful than extending context to more sentences. for example, if a source text is a product review, it can indicate that "i will get this part" most probably means "i will buy this part of some object", while for a movie or book review "i don't get this part" probably means "i don't understand this part of a movie/book". mistranslations mostly consist of simply incorrect lexical choices, however a number of them looks as "false friends". sub-word units are the most probably reason for this type of errors, but it should be investigated further in more details. untranslated words contribute to errors, too, although to lesser extent. the same can be observed for omissions, however it has to be noted that the contribution of omissions is underrated in both analysed data sets; they are not at all marked in the hu corpus, and even though they are marked in the dcu corpus by omission mark, the evaluators mostly added one single omission mark for missing phrases. furthermore, the nature of omissions should be investigated more, for example how many of them are related to the source text and how many to the target text. another difference between the two data sets can be seen for named entites: they seem to be rather problematic only in the dcu corpus. therefore, errors related to named entities are probably domain and/or language dependent. the largest difference between the two corpora can be observed for prepositions and extra words, which resulted in much more errors in the hu corpus. this indicates possible dependance on domain and language, but also on mt system (since only one mt system was annotated in this corpus) and on quality criterion (because it was not specified for this corpus). also, contribution of gender and especially case is larger in morphologically rich(er) slavic languages than in german. it should be noted that these two phenomena include only single-word errors exclusively related to gender and/or case: there are more gender and case errors, but within other phenomena with larger spans: rephrasing, noun phrase, conjunction. ## summary and outlook we have carried out an extensive analysis of mt errors observed and highlighted by different evaluators according to different quality criteria. the analysis includes three language pairs, two domains and eleven nmt systems. our main findings show that the majority of perceived errors are caused by rephrasing, ambiguous words, noun phrases and mistranslations, followed by untranslated words and omissions. furthermore, it is shown that perception of errors is dependent on the pre-defined quality criterion. for example, non-existing and untranslated words, as well as errors in the source text are perceived similarly for different quality aspects, but there is a large discrepance between adequacy and comprehension errors caused by negation, hallucinations and missing english conjunctions. therefore, the ideal evaluation would include both quality criteria. however, comprehension cannot be properly assessed if the source text is seen, so that it cannot be evaluated together with adequacy, but has to be performed beforehand as a separated task. this is time and resource-consuming, so that usually a (often unspecified) combination of adequacy and fluency is used, while comprehension, although more important than fluency, is rarely included. our findings indicate that evaluating both adequacy and comprehension can be facilitated, because it is not necessary that the same evaluators work on both quality criteria. the findings also open several directions for future work. for some phenomena, further analysis is recommended, for example the type of rephrasing (literal translation or not), more details about the negation (span, type of negation marker(s), etc.), source vs target omissions, etc. test suites should also be created for some of the phenomena, in order to provide more information about errors and give ideas for potential improvements of the current systems.