Reading Comprehension of Machine Translation Output: What Makes for a Better Read? (2018)

Abstract: This paper reports on a pilot experiment that compares two different machine translation (MT) paradigms in reading comprehension tests. To explore a suitable methodology, we set up a pilot experiment with a group of six users (with English, Spanish and Simplified Chinese languages) using an English Language Testing System (IELTS), and an eye-tracker. The users were asked to read three texts in their native language: either the original English text (for the English speakers) or the machine-translated text (for the Spanish and Simplified Chinese speakers). The original texts were machine-translated via two MT systems: neural (NMT) and statistical (SMT). The users were also asked to rank satisfaction statements on a 3-point scale after reading each text and answering the respective comprehension questions. After all tasks were completed, a post-task retrospective interview took place to gather qualitative data. The findings suggest that the users from the target languages completed more tasks in less time with a higher level of satisfaction when using translations from the NMT system.

## introduction recently, there has been an increase in neural machine translation (nmt) research as contemporary hardware supports much more powerful computation during the creation process. research c 2018 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. on the translation quality of nmt engines show that, in general, when compared against statistical machine translation (smt) engines, the output quality of nmt systems is higher when measured using automatic metrics @xcite @xcite . however, results are not as positive when human evaluators compare these outputs @xcite @xcite . human evaluation of mt output, although not always implemented in quality evaluation, has been increasingly endorsed by researchers who acknowledge the need for human assessments. some of the most commonly-used manual metrics are fluency and adequacy, error analysis, translation ranking, as well as post-editing effort. despite the considerable focus on mt quality evaluation, the impact of mt on the end user has been underresearched. measuring the usability of mt output allows for identification of the impact that the translation might have on the end user @xcite . with the intention of exploring the cognitive effort required to read texts originating from smt and nmt engines by the end users of those texts, we set-up a pilot experiment that aims to measure the reading comprehension of spanish and simplified chinese users of texts produced by both paradigms using an eye-tracker (using the english users' data as a baseline). the remainder of this paper is organized as follows: in section 2, we survey the existing literature concerning reading comprehension for mt evaluation and the use of eye-tracking techniques for translation assessment; in section 3, we describe the research questions and hypotheses which guide this pilot experiment, as well as the methodology applied to carry out the experiment with english (en), spanish (es) and simplified chinese (zh) native speakers; the results are discussed in section 4, and finally, in section 5, we draw the main conclusions of the pilot study and outline promising avenues for future work. ## methodology hypothesis and research questions as mentioned in section 1, the primary aim of this experiment is to gather more information about the user experience when reading for comprehension machine-translated texts. with this aim in mind, we identified the following research questions: rq1: which mt engine offers better efficiency to participants, i.e. with which one are they able to successfully answer more comprehension questions? or with which one are they able to complete the tasks faster? rq2: to what extent are there differences in participants cognitive processes due to different engines (nmt and smt)? rq3: what is the participants level of satisfaction with smt and nmt when reading for comprehension? content and design in order to answer the research questions, we measured participants reading comprehension according to the number of correct answers (goal completion) to a set of comprehension questions about each text, and task time. eye-tracking fixation count and duration are also computed, as well as satisfaction indexes after each reading task. after all tasks were completed, we interviewed the participants by means of a semi-structured retrospective interview to gauge the understanding of the texts from a qualitative perspective. for this pilot, we recruited two native speakers per language, a total of six participants (english, spanish and simplified chinese languages). in this case, we used a sample of convenience. the participants were part of the student and staff body of dublin city university. there were three female and three male participants, average age was 30.6 years, and all of them had received education to a post-graduate level. half of them had previous experience in reading comprehension tests, either as part of their education or work. the spanish and simplified chinese participants had a university level standard of english as they have taken english proficiency tests and have been working and studying in an english-speaking country for some time. as for the reading texts, two were taken from the international english language testing system (ielts) foot_0 that measures english language proficiency by assessing four language skills: listening, reading, writing and speaking. ielts has two types of tests: general and academic. since we were trying to assess the reception of raw output for a general user, we decided to use the general training ielts, reading modality, which contains a text and comprehension questions about that same text. the total number of words in the source content amounted to 1090 words. the two english texts selected and their accompanying comprehension questions were then translated using microsoft translator try and compare feature foot_1 that allowed one to generate output in both smt and nmt, and compare their quality. the first text (text 2), entitled "beneficial work practices for the keyboard operator", contained seven comprehension questions in which the users were required to choose the correct heading for each section of the text from a list of headings. the second text (text 3), entitled "workplace dismissals", contained five comprehension questions for which the users were required to match each description from a list with a correct term displayed in a box. one short text was also extracted from the ielts website to be used as baseline. this baseline text (text 1) was available in english, spanish and simplified chinese on the ielts website. 3 moreover, ten questions in the style of the test (write true, false or not given) were created in english for this baseline text and translated into spanish by a spanish translator and into simplified chinese by a native speaker. the baseline was used to test participants attention and reading comprehension with a human-translated version. the total number of words in the source baseline text amounted to 229 words. the baseline text was presented first followed by the text 2 and text 3 (smt and nmt) which were randomised. 4 figure 1 shows the set up of the task. after each task (text and comprehension questions), four statements were presented (in english) in a three-point likert scale (1-disagree, 2-neither agree or disagree, 3-agree) for the participants: the eye tracker used was a tobii t60xl with the filter set for i-vt (velocity-threshold identification), as this is the filter recommended by tobii for reading experiments. the participants were recorded during the post-task interview using the flashback application that allows recording of all movements, sounds, and webcam output on the computer. this retrospective post-task interview was designed so that participants could watch their recordings and give their feedback regarding the subject matter, language used, questions, and personal experience when completing the whole task. figure 2: goal completion (%) figure 3: task time (in seconds) 4 results ## conclusions and future work the aim of this pilot experiment was to verify the methodology to measure the impact of the quality of two mt paradigms -nmt and smt -on the end user. for that, we established three research questions regarding efficiency (goal completion), cognitive effort, and satisfaction. regarding rq1 (which mt engine offers better efficiency to participants?), results show that participants (figure 2 ) in the two target languages -spanish and simplified chinese -were able to complete more tasks successfully when using the nmt translated texts when compared to the smt translations, as well as when compared to participants who used the original en texts. regarding the time spent to complete the texts, again, we noted that when using the nmt translations, participants were faster than when using smt translations and, moreover, have task completion times closer to participants who used the english text than the results for smt. regarding rq2 (to what extent are there differences in participants cognitive processes due to different engines?), results for the fd (figure 4 ) and fc (figure 5 ) show that cognitive effort does not seem to differ much for es, and presents a bit of mixed results for zh, were fd are slightly longer for the nmt system, whereas fc are lower. we believe that with a greater number of participants, a clearer tendency would be observed. regarding our last research question (rq3: what is the participants level of satisfaction with smt and nmt when reading for comprehension?), participants rated nmt higher and also commented that the language in nmt texts was easier to understand in the post-task retrospective interviews. it is also necessary to point out that es and zh participants commented on the fact that the language in the human translation (text 1) was easy to understand, while they struggled in certain sections in both nmt and smt texts (texts 2 and 3). this was not the case with en participants that only made slight remarks on the quality of the english, but they did not mention any misunderstandings of the texts. we are aware of the limitations of the results presented here since the number of participants was very low, and there were few texts for each mt system. our next steps are to add more languages, especially those languages which have been showing greater improvement with nmt over the smt paradigm, as well as gathering more participants. another consideration to bear in mind is the nature of the texts; we noted that the combination of difficult text with easy questions and vice-versa could cloud the findings. furthermore, we believe that this research could benefit from computing more eye-tracking measures, such as visit count, which is the number of visits to an area of interest, as the shifts of attention between the questions and the text may be an indicator of cognitive effort @xcite .