A Taxonomy and Study of Critical Errors in Machine Translation (2022)

Abstract: Not all machine mistranslations are equal. For example, mistranslating a date or time in an appointment, mistranslating the number or currency in a contract, or hallucinating profanity may lead to consequences for the users even when MT is just used for gisting. The severity of the errors is important, but overlooked, aspect of MT quality evaluation. In this paper, we present the result of our effort to bring awareness to the problem of critical translation errors. We study, validate and improve an initial taxonomy of critical errors with the view of providing guidance for critical error analysis, annotation and mitigation. We test the taxonomy for three different languages to examine to what extent it generalises across languages. We provide an account of factors that affect annotation tasks along with recommendations on how to improve the practice in future work. We also study the impact of the source text on generating critical errors in the translation and, based on this, propose a set of recommendations on aspects of the MT that need further scrutiny, especially for user-generated content, to avoid generating such errors, and hence improve online communication.

## introduction machine translation (mt) has now become ubiquitous in many online platforms (e.g. social networks) and generally used without any human post-editing due to cost, timeliness, and accessibility. the rapid development and adoption of mt has advanced efforts to improve and standardise mt evaluation, and increased discussion on how we should evaluate mt @xcite @xcite . this need escalated with the use of mt to translate user-generated content (ugc), e.g. in social media platforms. unlike formal text, ugc often has colloquial language, including profanities, spelling errors, emojis, hashtags and abbreviations, and is grammatically ill-formed, which makes it hard for mt, often resulting in incorrect translations @xcite . some of these incorrect translations can contain critical errors. in this work, we refer to critical errors as instances of translations where the meaning in the target text deviates drastically from the source text where such translations can be misleading and may carry health, safety, legal, reputation, religious or financial implications. the volume of content shared by users means that the mt-translated content cannot be manually post-edited. therefore, users have to rely on mt as is and usually do not have the linguistic skills to identify the errors. as a consequence, users may be negatively affected if they misunderstand the intention or sentiment of the source text or could take inappropriate action if they act on critically corrupted translations. there are many instances where innocuous statements on social media have been translated by the machine to say something quite different, the opposite, or even turn a simple greeting into hate speech -translating 'good morning' in arabic into 'attack them' in hebrew by the machine, leading to the arrest of a palestinian worker who posted it on his social media profile by israeli police, as reported by the guardian @xcite . therefore, it is important that the issue of critical error is directly addressed. to mitigate such a problem, recent research has looked into automatic methods to detect critical errors in machine translation, with a view to inform users of such errors. this was framed as a track in the wmt 2021 shared task on quality estimation @xcite . a taxonomy was proposed to annotate training and evaluation data for this task. the annotation effort focused on critical errors only, i.e. other errors were disregarded. this differs from previous work, where critical errorsif evaluated -are seen as an extra level of annotation on general errors, i.e. as a severity judgement on errors @xcite . from a practical perspective, we believe this focused annotation is a good strategy as it saves annotation effort and allows gisting-oriented quality prediction models, under the assumption that mt is still usable even though it may contain minor (non-critical) errors. according to @xcite , however, the annotation of critical errors proved very challenging, with low agreement amongst annotators. a taxonomy is an important step as it establishes which types of errors should be considered critical. we revisit and extend the taxonomy proposed in @xcite in order to (a) perform a more focused, smaller-scale study with well-trained annotators to understand the general challenges in annotating critical errors, and (b) validate the extended taxonomy on different languages. for that, we commission the manual annotation of such errors and conduct an in-depth analysis of their impact on the translations. we reflect on the annotation process as an essential part of any evaluation task that aims to examine the performance and usability of mt systems for better evaluation and annotation practices. we also show how the source text can affect the quality of mt translations when it comes to the presence of critical errors. we start by presenting an overview of popular quality evaluation taxonomies (section 2) to then introduce the taxonomy we study, developed in @xcite , with two additional categories we propose to add to the taxonomy (section 3). we then explain our approach and criteria to validating the extended taxonomy and follow that with a data analysis through which we show how the taxonomy is validated (section 4). we also reflect on the annotation process for different languages (section 5). finally, we explore how the quality or lack of quality of the source text could contribute to the generation of critical errors (section 6). ## related work with the rapid development and increasing adoption of machine translation systems, evaluating the quality has become a common practice. this has led to advances in the area of translation quality assessment (tqa) and inspired initiatives that aimed to standardise this practice. foot_0 tqa is used to assess the performance of a system, and whether its output fits to be used either as is or as a first draft that requires some post-editing @xcite . tqa can also be utilised to enhance the performance of systems, as a point of comparison between various systems, or to estimate the effort required to post-edit machine-translated content @xcite . examining the quality of the mt output has been conducted through either the identification of errors, the overall assessment of mt quality or both. various classifications of errors have been developed, against which mt system outputs are assessed @xcite @xcite . the two most comprehensive frameworks, which have been widely adopted in industry, academia and by end-users, are (i) multidimensional quality metrics (mqm), proposed under the eu-funded qtlaunchpad project @xcite , and (ii) dynamic quality framework (dqf) by the translation automation user society (taus) @xcite . these initiatives offering general taxonomies are based on, and inspired by, earlier error-specific models including lisa qa model, developed in the 1990s by the localisation industry standards, and the sae j2450 metric, among others @xcite . another group of individual error classifications includes language-related and linguisticallymotivated taxonomies that aim to evaluate the quality of mt output according to specific linguistic phenomena that occur in the translation and are associated with certain languages. for example, @xcite 's study classifies translation errors from english into european portuguese. their work extends previous taxonomies to study errors associated with morphologically rich languages. some other studies focus specifically on the impact of certain features of the text on the output. for example, abu-ayyash (2017) explores errors and non-errors for the english-arabic pair in mt-translated gender-bound constructs in tech-nical texts, and @xcite proposes a categorisation of error types generated by mt systems when translating multiword expressions. in addition to classifying types of errors, other aspects of quality evaluation are considered, i.e. the importance and severity of the errors. still, these are optional criteria and considered depending on the task and the purpose of the translation. in the mqm framework, importance is assigned to categories of errors. for example, if one category is considered as a priority for a given task, it is deemed as important for that specific task. severity, however, is applicable to individual errors, and is related to their nature and their impact on the usability of the translation. 'the more severe an error is, the more likely it is to negatively affect the user in some fashion' @xcite . mqm identifies four levels of severity: critical, major, minor, and null that align to some extent with those adopted in the dqf framework @xcite . more recent work has focused on classifying only the most severe errors (referred to as critical errors). for example, the wmt 2021 shared task on quality estimation @xcite organised a track on predicting the presence of critical errors in sentence translations. as part of this track, a taxonomy of critical errors was proposed and a large amount of data was annotated for such errors: 10k translations from english into four languages (chinese, japanese, czech and german). each translation was annotated by three professional translators. however, the authors observed that the annotation was problematic, with overall low annotator agreement. it was not clear from the effort whether this was because of the general lack of understanding of the task by the annotators, the complexity of the task or because of other factors. one interesting outcome of the report in specia et al. (2021) was the high proportion of critical errors in ugc. it is clear that error-free mt is still unattainable and that critical errors are not rare. therefore, further research towards understanding, formalising, and annotating such errors is much needed before prediction and mitigation strategies can be put in place. we, therefore, devote this work to bring attention to this issue. we study critical errors that have the same level of severity (highest), and treat them as critical errors because of their potential negative impact on those who use the translations as they are. the assump-tion, which we test in this paper, is that the types of critical errors should be applicable to any language pair. as far as we know, this is the first work which focuses on studying critical errors in ugc. ## a taxonomy of critical errors in what follows, we present @xcite 's taxonomy of critical errors a) to serve as the base for a new extended taxonomy developed in this work and b) to be tested and analysed in detail. it recognises three ways in which meaning deviations from the source sentence can happen: ## validating the taxonomy in this section, we report on a study we performed on this extended taxonomy by means of an annotation exercise with additional languages, followed by an in-depth analysis. ## evaluation of the annotation task: challenges and recommendations data was annotated for the three selected languages by professional translators. we provided them with guidelines based on the extended taxonomy with clear instructions that they must only annotate critical errors with catastrophic impact on the translation. however, we have found that: some annotators felt unsure about why such translations with critical errors should be accepted and the purpose of carrying out the annotation task. these annotators tended to consider errors as critical when they did not follow the general rules of a language (grammatical or stylistic rules), overlooking what the guidelines stated, ending up annotating both minor and critical errors. it is, therefore, vital to not only provide clear instructions on how to carry out the annotation task, but to also highlight that they need to treat it as a serious task similar to translating official documents and that they should always follow the guidelines (i.e. annotation brief). ## source-text impact on the mt output this section presents an analysis of the source text to show whether there is a correlation between the quality of the source text and the generation of critical errors. for this purpose, we analyse translations produced by the three online mt systems (google translate, bing and systran) for one language combination, i.e. english-arabic, using the same dataset (100 sentences). our focus on arabic was driven by the availability of language expertise (i.e. one of the authors is a native speaker of arabic). the assumption is that if the different systems struggle with the same source sentences, producing critical errors, it would give indications about the potential output the machine could produce when handling such texts. our aim is to detect patterns in source sentences that can cause critical errors to be considered when developing mt systems to improve the performance of such systems, especially for ugc. we use, as a point of reference, al @xcite 's taxonomy of aspects of non-standard text that could affect the quality of the translation. for readability, back translations of the errors are provided in english. offensive language the importance of looking at this aspect of the data comes from its extensive existence in ugc and its sever impact on the output. our analysis shows that most translations that have critical errors are those of sentences which contain offensive language. when the sentence has a large number of swearing/offensive words and idiomatic phrases, the machine tends to produce wrong translations that are unreadable or completely different from the source. when it comes to translating offensive language, we recognise the use of different 'strategies' including literal translation, transliteration, omission, random translation (hallucination) or substitution of one strong word with another milder word and vice versa. sometimes, the machine uses a mix of these strategies when translating the same sentence, failing to convey correct translations as a result. for example, the three systems failed to provide correct translations of the offensive language in this sentence 'piss off homo, no one wants to hear from you, also hahahahaha you can't get married #asshole', leading to major errors which have affected the original meaning. these systems vary in how they handled this type of language. gt translated 'piss off' as 'rape punctuation marks misusing punctuation marks (e.g. deletion, addition, or use of wrong punctuation marks), especially commas and full stops, could lead to a mix up of the different parts of the sentence or different sentences, generating critical errors. for example, the translation of 'i give up thanks for ruining the lion king pages' shows the impact a missing punctuation mark has on the translation. the three systems translated the first part as 'i gave up thanking'. they, therefore, do not deliver the original meaning where the writer intended to say he/she is giving up trying to keep the pages, and that the word 'thanks' is used in a sarcastic way to express his/her frustration. negation negation can lead to critical errors when reversed from negative to positive or vice versa; through e.g. dropping or reversing negative words (e.g. not, never, nobody); or reversing the meaning of some words (for instance, the three systems translated the verb 'reverting' in 'why keep reverting my edits?' as 'bringing back'. named entities named entities can be confusing to the machine especially when the name has different meanings and the mt system fails to treat it as a proper name, or when the names are unknown to the machine. names are either mistranslated, left untranslated or deleted completely. for instance, bing translated the proper name 'rocky' in the sentence 'your fucking ass doesn't know shit about it at all.rocky' as a noun rather than transliterating it, resulting in a wrong translation, while gt and systran dropped it completely. ## conclusion this work validated an extended taxonomy of critical errors developed to serve as a stand-alone taxonomy that can be used to evaluate or detect critical errors in machine-translated content. findings emphasise the need to address critical errors with catastrophic impact on the output and for further attention to be paid not only to developing guidelines on critical errors, but to also training annotators on how to spot and assess them. it has proved that critical errors are not rare, and they are not specific to certain languages. it also underlines the need to improve current mt systems to specifically deal with user-generated content, considering aspects of the text that could lead to critical errors to improve online communication and enhance mt's role in enabling, rather than hindering, communication among speakers of different languages.