With or without you? Effects of using machine translation to write flash fiction in the foreign language (2020)

Abstract: The improvement in the quality of machine translation (MT) for both majority and minority languages in recent years is resulting in its steady adoption. This is not only happening among professional translators but also among users who occasionally find themselves in situations where translation is required or MT presents itself as a easier means to producing a text. This work sets to explore the effect using MT has in flash fiction produced in the foreign language. Specifically, we study the impact in surface closeness, syntactic and lexical complexity, and edits. Results show that texts produced with MT seem to fit closer to certain traits of the foreign language and that differences in the use of part-of-speech categories and structures emerge. Moreover, the analysis of the post-edited texts reveals that participants approach the editing of the MT output differently, displaying a wide range in the number of edits.

## introduction in recent years, the quality of machine translation (mt) has greatly improved, and as a consequence, increasingly more users are adopting the technology. these users can have varying profiles. on the one hand, we find professional translators, and on the other hand, we have users who do not belong directly in the translation industry but still, occasionally, need translations. among the latter, c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. a good few studies have been conducted on the impact of mt for professional translators but still numerous questions remain unanswered. among others, this research has focused on analysing how translating using mt differs from translating from scratch and on ways to optimally provide the automated translation to these professionals. however, little research has been carried out on nonprofessional translators, even when freely available online systems have been providing automated translations for a long time, since 2006 in the case of google translate. this situation leaves us with little insight into what happens when nonspecialists avail of mt. the scarce research carried out on regular users has mainly focused on measuring the usefulness of mt for assimilation, that is, to facilitate comprehension. @xcite reported that people are using mt increasingly more for gisting purposes and that they are prepared to accept different quality levels for comprehension and for publication. @xcite and @xcite focused their efforts on the canadian context. in the former study, the author examined the potential acceptance of mt output by minority communities. she reported a positive attitude towards output that had undergone rapid post-editing for assimilation purposes but the need for at least full post-editing for texts intended for cultural preservation. the latter study analysed the usefulness of machine translation to make the ottawa public library website more accessible to spanish speakers. authors reported that users would be willing to ac-cept mt output, post-edited at different levels, for certain services. focusing on romance languages, another group of researchers studied mt in reference to the concept of intercomprehension, that is, the ability of speakers of different languages to understand one another @xcite . @xcite studied if users perceive mt output, non-native and native texts differently, and examined the usefulness of mt to improve comprehension in cases where native language texts would not be available. they highlighted that the efficiency seems to vary according to the level of specialisation of the texts, their domain and the mt system used. almost no research has been carried out on the effects of using mt to produce texts by regular users. the few efforts made in this area have mostly focused on the use of mt by nonnative english speakers for academic publishing. parra escartín et al. (2017) studied five medical practitioners' papers and o'brien, simard and goulet (2018) examined abstracts of ten scholars. both studies found that whereas these professionals were able to correct and improve the mt output, their final versions still required further editing to be adequate for publication. @xcite provide an overview of this user group and make a first attempt at establishing a framework for mt literacy for scholar communication. further research in this line will prove essential to train different user groups in the optimal use of this technology, as non-language-specialists seem to be willing to accept low quality mt output when translating familiar topics @xcite . within this context, the current work focuses on non-specialist users. we concentrate on using a series of metrics to compare texts produced by those users in the foreign language with and without mt. in particular, we aim to examine the effects of using mt in terms of accuracy, fluency and complexity. in the future this should be complemented with further qualitative analyses to account for word and word-sequence choices and editing. regarding the foreign language, english in this case, the reported level of competence is more widespread even when almost half classify themselves within the b2 level, and almost 40% within the b1 level. as expected, over 75% of the participants report using english less than 25% of the time. all in all, given their reported level of competence in their main and foreign languages, this group of participants proved adequate to study the impact of using mt to produce texts in a foreign language where their competence is low, starting from their language of instruction. therefore, the foreign language is at the independent user level according to the cefr, whereas their main language of instruction is at the proficient user level. ## analysis of results the analysis presented here focuses on using a series of metrics to compare texts produced in both set-ups, (1) when participants write directly in the foreign language, english, and (2) when they start with their main working language, basque, and edit the english mt output provided by google's translator. many researchers in the area of language acquisition have long considered complexity, accuracy and fluency as the three main aspects that capture the foreign language competence (see @xcite for a discussion). in reference to the experiment that concerns us, this means that texts can be classified as better or worse depending on how natural and nativelike they sound, the grammatical and lexical inaccuracies they contain, and the complexity of the structures included. therefore, for this study, we concentrate on the surface closeness, syntactic and lexical complexity, and edit types, while also con-sidering participants' view on usefulness. we report count averages together with the standard deviations, and when possible, calculate statistical significance of the differences (with a 95% confidence interval) using the unpaired student's t-test to compare the two set-ups (significance is marked with a †). ## conclusions given the increase in the translation quality provided by automatic systems, the option of using online freely available systems to produce text in a language in which we are not fully competent by exploiting our main language is more and more appealing. with this in mind, this work analyses the effects of using mt when writing flash fiction in the foreign language. to examine this, we asked participants, who were advanced users of basque (language of instruction) and independent users of english (foreign language), to write two pieces of flash fiction of around 150 words each, with and without using mt. we compared features of the stories produced in each set-up with the aim to examine the effect of starting the writing process in a language in which the participants were competent and having an mt system provide them with a preliminary translation. specifically, we aimed to observe whether mt can help to produce a text that sounds more english, and whether it can increase the complexity of the text. to that end, we compared wordand pos-level perplexities, lexical proportions, diversity and density, and the frequency of semantic relations that involve complex structures. results suggest that using mt participants produced final foreign language texts that followed english word-and pos-sequences more closely, indicating a higher fluency. we also observed that the proportion of pronouns and prepositions or subordinate conjunctions was higher in this set-up, even when no significant difference was observed in lexical variety and density. dependency relations, in turn, revealed that the frequency of noun and prepositional modifiers, as well as coordinating conjunctions was also significantly higher. overall, we can conclude that the texts produced using mt display certain traits that are typical of better quality texts. we also considered the post-editing work of the participants. by examining ter scores and edits counts, we discovered that the participants approach the mt output differently. while it is true that the edit-distance is rather low in general, some make no changes to the output, whereas others change over 20% of the words, with most staying somewhere in between. finally, it was encouraging to learn that participants perceived that the mt system was useful for the task (it obtained a score of 3.9 on average in a 1-5 scale), which shows the advance of mt quality for basque and the positive attitude towards the technology. while this research has revealed a number of interesting features from a quantitative perspective, further research into the actual lexical choice in each of the set-ups is now necessary to highlight differences in terms of lexical precision and difficulty between set-ups. also, what remains to properly account for is the level of proficiency participants show in addressing mt output. interesting results would be provided by research reporting on the elements that prompt users to introduce changes and on the impact these have at a linguistic level but also from the reader's perspective. complementary research on the linguistic characteristics of the texts and user performance could also shed light into second language acquisition processes and teaching opportunities, as well as guide mt development.