Assessing the Comprehensibility of Automatic Translations ( A ris T o CAT ) (2020)

Abstract: The ArisToCAT project aims to assess the comprehensibility of ‘raw’ (unedited) MT output for readers who can only rely on the MT output. In this project description, we summarize the main results of the project and present future work.

## introduction machine translation (mt) systems cannot guarantee that the text they produce will be fluent and coherent in both syntax and semantics. errors occur frequently in machine-translated text, leaving the reader to guess parts of the intended message. with the arrival of neural machine translation (nmt), however, the quality of machine translation has increased significantly. as such, machine translation is becoming an attractive solution to deal with the increased need for translated content. this could mean that, in the near future, readers will be more often confronted with 'raw' (unedited) mt output. ## quality of mt output to assess the quality improvements in mt, we compared the quality of three different mt systems for english-dutch: a commercial neural system, a phrase-based system and a predominantly rule-based system. we used web-anno 1 as annotation tool and adopted a two-step approach to annotate all errors in the mt output. in a first step, only the target text was visible and we marked all fluency errors; in a second step all accuracy errors c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 https://webanno.github.io/webanno/ were labelled in both source and target text and were linked. @xcite found that the neural system, in general, outperformed the phrase-based and rule-based systems when considering fluency. the output of the neural system contained fewer grammatical errors and hardly any spelling mistakes. for accuracy, the improvements of nmt are less apparent. the target sentence does not always contain traces of the errors or clues of omissions, which might have an impact on the comprehension. ## reading comprehension tests in a pilot study, @xcite selected three texts of the english mt evaluation version of the corpus of reading comprehension exercises @xcite and set up a reading comprehension test for both human translated and raw mt texts. ninety-nine participants were asked to read the translation very carefully after which they had to answer comprehension questions without having access to the translated text. human translations received the best overall clarity scores, but the reading comprehension tests provided much less unequivocal results. ## comprehensibility of newly invented words in nmt output nmt systems occasionally generate non-existing words, i.e words that are not part of the vocabulary of the target language and were thus invented by the nmt system. in cases in which readers only have access to the mt output without the source text, such non-existing words can affect comprehension. there are several reasons why an nmt system creates new non-existing words. one reason is that, although nmt systems have made huge progress, they sometimes still generate a too literal translation for different types of multiword expressions such as compounds, another reason, specific for nmt systems, is that they operate at sub-word level to reduce vocabulary size. @xcite set up an experiment in which eighty-six participants were given 15 non-existing words (5 single words and 10 noun compounds) and were either asked to describe the meaning of these words or to select the correct meaning from a predefined list. the words were presented either in isolation or in sentence context. non-existing words indeed impaired comprehension as on average in 60% of the cases the participants gave a wrong answer. sentence context, however, made it easier for the participants to determine the meaning of the non-existing word as the percentage of wrong answers is much higher (77%) when the words were presented in isolation. ## nmt for literary translation to assess whether, with the improved quality, nmt systems are able to produce high-quality translations for more creative text types such as literature, we translated agatha christie's novel the mysterious affair at styles with google's nmt system into dutch and applied the two-step error annotation. @xcite found that 44% of the mt sentences did not contain any errors. the accuracy subcategory mistranslation was the most frequent error type encountered in the novel, followed by the fluency subcategories coherence and style & register. @xcite further investigated how the mt version differs from the published professionally human-translated (ht) dutch version of the book. measures of lexical richness (type-token ratio and mean segmental type-token ratio) gave inconclusive results. they also looked at word translation entropy @xcite , which indicates the degree of uncertainty to choose a particular translation from a set of target words based on the number and distribution of different translations that are available for a given word in a given context and found that the average word translation entropy scores were higher in ht than in mt, meaning that there was more variety in the translations in ht. at the syntactic level, the mt generally follows more closely the structure of the source sentence compared to the ht version. ## future work in future work, we will set up eye-tracking experiments and expand the ghent eye-tracking corpus @xcite with eye-tracking data for the nmt version of the novel. this will allow us to analyse to what extent mt impacts the reading process, and which errors impact this reading process the most.