MT syntactic priming effects on L 2 E nglish speakers (2020)

Abstract: In this paper, we tested 20 Brazilian Portuguese speakers at intermediate and advanced English proficiency levels to investigate the influence of Google Translate’s MT system on the mental processing of English as a second language. To this end, we employed a syntactic priming experimental paradigm using a pretest-priming design which allowed us to compare participants’ linguistic behaviour before and after a translation task using Google Translate. Results show that, after performing a translation task with Google Translate, participants more frequently described images in English using the syntactic alternative previously seen in the output of Google Translate, compared to the translation task with no prior influence of the MT output. Results also show that this syntactic priming effect is modulated by English proficiency levels.

## introduction machine translation systems (mt), especially google translate, have become popular in the last decades @xcite . the popularity of these systems has grown not only due to technical improvements, but also due to the facilitation of users' access through the proliferation of mobile applications containing a number of functions that allow users to translate from texts, from speech or from a text image @xcite . the rapid development of c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. mt technologies is the result of massive research investment in the field over the past decades focusing on language resources, new methods and techniques with the aim of improving the quality of the mt output. consequently, the progress made in mt technology has changed the way people are engaging with these systems @xcite . in the past, mt systems were used mainly for gisting purposes, but nowadays they are also being used as a tool supporting writing skills, grammar skills and language production in a second language (l2) @xcite . however, research in the mt field focusing on end-users is limited so that, currently, little we know about what the users' interaction with an mt system could bring to the mental processing of a second language. in this paper, we aim at investigating the role mt systems play, especially, the role that the popular mt system google translate plays on the processing of english as a second language. specifically, we investigate the influence of google translate on the processing of english as l2 by testing whether the mt output can influence the way mt users process english syntax. we address the following research questions: to the best of our knowledge, these questions remain unaddressed in the mt literature and deserve further scrutiny. to accomplish this paper's goal, we carried out a syntactic priming study, an experimental paradigm commonly used by researchers in the field of psycholinguistics, as an way to understand aspects of the representation and processing of language syntax @xcite . the syntactic priming approach enabled us to understand the influence of the mt in participants' linguistic behaviour after exposition to the mt output. due to differences in syntactic structures, we focused in particular on portuguese-english (pt-en) translation. before presenting our methodology in detail, we highlight the progress that has been made in previous research focusing on syntactic processing adopting the syntactic priming methodological paradigm. ## related work the syntactic priming effect, known as syntactic alignment or structural alignment, occurs when people, in a communicative context, repeat the same syntactic structure previously seen, heard or read @xcite . the first priming effect was reported by @xcite who observed a repetition in grammatical structure in a question-and-answer telephone experiment in which merchants were asked either (a) "at what time does your store close?" or (b) "what time does your store close?". the researchers observed that the merchants were much more likely to respond to (a) with a sentence also starting with a preposition, such as "at 6 o'clock", and to questions such as (b) with a noun-phrase, such as "6 o'clock". @xcite was the first to implement a laboratory study to investigate this repetition effect and developed an experimental paradigm to understand its characteristics in a controlled and naturalistic manner at the syntactic level. the experiment consisted of reading a sentence and asking participants to repeat out loud the same sentence. listening and repeating the sentence was considered the "prime phase" of the experiment as the experimenter could control participants' exposure to different syntactic structures. following the "prime phase", participants were requested to describe an image so that the researcher could observe if participants would use the same structure they had just produced in the prime phase in the subsequent utterance. @xcite noticed that in their subsequent utterance participants tended to use the same syntactic structure previously heard and repeated. after this seminal study, a number of studies have shown syntac-tic priming evidence between human interlocutors in l1 @xcite @xcite @xcite and l2 @xcite interactions. these studies have also revealed that less frequent syntactic structures prime more than more frequent structures @xcite . for example, passive structures which are less frequently used by english and dutch speakers prime more than active structures that are more frequently used @xcite . some researchers call this effect as the "inverse preference effect" and they claim that the most uncommon structures drive the priming effect @xcite . this repetition effect @xcite has also been shown between humans and computers @xcite . @xcite found syntactic alignment in human-computer speech-based interactions for both dative structures (e.g. give the waitress an apple vs. give the apple to the waitress) and noun phrase structures (e.g. a purple circle vs. a circle that is purple) evidencing that a computer system can also influence a speaker's grammatical choices in speech-based interactions. virtual reality studies have also demonstrated syntactic alignment between humans and computer avatars. @xcite observed a priming effect for passives and actives, although the priming effect was stronger for passives than for actives. @xcite have also found prosodic alignment between humans and computers. in their experiment, people exhibited alignment of loudness and response latency in their speech in response to computer-generated speech. @xcite found that children talking to computer partners spontaneously adapt several basic acoustic and prosodic features of their speech by 10-50%, with the largest adaptations involving utterance pause structure and amplitude. in addition, both naturalistic @xcite and laboratory research @xcite investigating speech-based interactions between humans and computers have also shown that people tend to align syntactically with computers. from the findings presented above, it is possible to conclude that, in an interactive context, speakers tend to syntactically align with their interlocutors both in first and second languages as well as with computer partners. importantly, these studies show that syntactic priming plays a central role in successful communication since it can promote mutual understanding through the semantic and structural representations shared by interlocutors @xcite . in this paper, we report the preliminary results of a study that investigates whether human participants, specifically l2 speakers of english, syntactically align with mt output after performing a translation task. in other words, we investigate whether users are primed by the mt output when speaking in a second language. we expect that if, after performing a translation task using an mt system, the syntactic structure from the translation task is observed in the speakers' subsequent utterance, then mt output can influence the syntactic processing of english as a second language. despite a number of studies demonstrating syntactic priming between humans and computers, to the best of our knowledge, this is the first time syntactic alignment between an mt system and a human partner has been investigated. the present study also expands previous syntactic priming studies. for the first time syntactic priming is tested from text (comprehension) to speech (production) by means of a crosslinguistic task (translation task) using a computer tool. based on previous findings, we hypothesize that when translating a text using an mt, the user's syntactic choices when producing speech in a second language will mirror the syntax of the mt output. to test these hypotheses, we carried out a behavioural syntactic priming study using an experimental paradigm commonly used in syntactic priming studies in the field of psycholinguistics @xcite . based on @xcite , we adopted a pretest-priming design as it enabled us to study the influence of the mt on participants' performance in a picture description priming task as compared to the pretest baseline. in addition to the two general research questions presented above, by adopting the syntacting priming methodological paradigm we aim at answering the following specific research question: in the following sections, we describe the methodology employed in detail. ## analysis and coding verbal responses were transcribed and manually coded for the syntactic structure used to create the factorial dependent variable prime. we coded a sentence as "1" if participants produced np constructions mirroring google translate syntactic alternative and as "0" if they produced pnp constructions or any other syntactic construction such as the office's table is full or the window in the house is broken that have not appeared in the output of the mt. after coding participants' responses (20 pretest and 20 priming test), we obtained a dataset containing 760 data points (19 x 40 = 760). the dataset was modelled using mixed-effects logit model and the glmer function of the lme4 package version 1.1.-4; @xcite . the mixed-effects logit model is a linear regression model used to handle the repeated measures nature of a dataset whose dependent variable is binomial. as fixed effects, we included the english proficiency test score (continuous) to investigate the influence of this variable on participants' responses and a factorial predictor test with two levels: baseline pretest and priming test. following @xcite , we used a maximal random-effects structure. we began with a maximal model and then performed a step-wise reduction procedure to find the simplest model that did not differ significantly from the full model in terms of variance explained. the numeric predictor (en-figure 2 : example of trial presented in the priming test. participants google translated the sentences of the first and second primes in the two conditions and, in the target item, were instructed to describe the images out loud using the words provided above each image. glish test score) was centered and the factorial predictor (test) was dummy coded (all means compared to a reference group). p-values reported here were obtained by means of anova type 3, test chisquare. in order to make sure that the inclusion of the random slopes and random intercepts are justified we used likelihood ratio tests which allowed us to compare the models @xcite . based on this process, the final model included by participant random slopes for test and by-item random intercepts as random effects. ## results figure 3 below shows the influence of language proficiency on priming effect. participants at higher english proficiency levels tended to produce more np constructions during the priming test phase than participants at lower english proficiency levels. in the priming test condition, more np constructions were produced compared to the baseline (@xmath0.001). this indicates that, after performing a translation task using google translate, participants tended to use significantly more np constructions, thus mirroring the syntactic structure previously seen in the google translate output when speaking in english. results also show a significant effect of language proficiency (@xmath1.001), demonstrating a difference between participants' responses at higher and lower levels of english proficiency as well as a significant interaction (p<.05) between factors test (baseline vs. priming test) and englscore (participants' english proficiency test score) suggesting that participants at higher levels of english proficiency produced more np constructions during the priming test than participants at lower levels of english proficiency. ## general discussion and conclusions to test the influence of mt output on the syntactic processing of english, we measured the effect of priming when performing a translation task using google translate. to this end, we used a common syntactic priming experimental design allowing us to compare the priming magnitude in the participant's speech before and after the task. in line with our predictions and the literature, our results show that, after interacting with an mt system, participants tend to use the same syntactic alternative previously seen in the output of the mt system more frequently in their subsequent speech. that is, they tended to use more np constructions after the translation task with google translate. importantly, after being primed by the google translate output, participants used noun phrase structures without a preposition, as a direct result of exposure to sentences with these structures in english. this structure represents the syntactic alternative more likely to elicit processing difficulties due to word ordering differences between portuguese and english. this result suggests that an mt system is capable of facilitating the syntactic processing of a second language by allowing users to access a structure that poses a challenge to the syntactic processing mechanism of brazilian portuguese native speakers. in contrast, this processing facilitation triggered by the mt system was more evident at higher english proficiency levels. we hypothesize that the effect of english proficiency observed is related to the participants' focus of attention when processing the second language. a number of studies (e.g. @xcite ) have shown that language learners at lower levels of proficiency are less sensitive to syntax than more proficient bilinguals because they focus their attention more on resolving semantic ambiguities than on resolving parsing problems. however, at higher proficiency levels it is possible to observe more automaticity in second language parsing. accordingly, our results suggest that mt systems can be useful in english language learning as it can facilitate endusers to access or construct problematic syntactic structures in english due to the structural differences between the languages. moreover, based on some psycholinguistic studies (e.g. @xcite ) evidencing that people tend to syntactically align more often with people they like, the priming effect observed may suggest that people enjoy interacting with google translate. although this study provides evidence for the influence of mt on english syntactic processing as a second language, especially np structures, it also reveals the possiblity that other syntactic structures may show the same effect as well as other language pairs. therefore, in follow-up studies, we will test whether the same effect can be observed for other challenging syntactic structures for portuguese speakers such as dative constructions with and without prepositions. in future studies, we also aim at increasing the number of participants, testing whether the popularity of the system plays a role on the magnitude of the effect and, finally, testing whether the participants mimic the google translate syntactic structure consciously or unconsciously. nonetheless, we claim that this first attempt to check syntactic alignment between an mt system and mt users provides an important investigation as the methodology has been tested and produced results comparable to results found in the literature in the field of psycholinguistics, second language learning and human-computer interaction.