Writing in a second Language with Machine translation ( W i LM a) (2022)

Abstract: The WiLMa project aims to assess the effects of using machine translation (MT) tools on the writing processes of second language (L2) learners of varying proficiency. Particular attention is given to individual variation in learners’ tool use.

## pilot study in this project's main experiment, we want to study the mt-assisted writing process across proficiency levels. to allow for this comparison, we need a reliable and valid instrument that quantifies learners' l2 proficiency level. we selected two tests that assess l2 learners' levels of swedish: the swedish levels test @xcite and a standardized placement test developed by folkuniversitetet. nine learners completed both tests. the internal consistency of the tests was high, as was the correlation between the learners' scores on the two tests. this correlation supports the criterion validity of the tests. moreover, we want to compare l2 learners' writing processes in two conditions: with access to an mt tool and with access to an obd. to this end, the writing prompts the learners respond to should fulfill two criteria. first, they should elicit equivalent products and processes (i.e., be comparable). second, they should be attainable to the least proficient learners, as well as challenging to the most proficient ones (i.e., be 'multilevel'). therefore, we had 5 learners of varying proficiency respond to 4 prompts. in each prompt, we asked them to describe 3 images, choose the one that appealed to them the most, and explain why. this way, the texts contained both descriptive and argumentative elements, blending genres of varying difficulty. using the linguistic profiling tool profiling-ud foot_0 and the keystroke analysis program inputlog, foot_1 we analyzed the equivalence of the prompts by comparing product and process measures across the 4 tasks. the prompts elicited texts of similar length and complexity. the amount of time the participants spent on the tasks was comparable, as were their pausing and repair patterns. the participants consulted the tools that were allowed equally often and spent a similar amount of time consulting them. we also did not find any patterns in how difficult the learners perceived the different prompts. the suitability of the prompts for use with multilevel learners was assessed by conducting similar analyses, but this time across levels instead of prompts. the least proficient learners still managed to comfortably exceed the threshold of 100 words in the given time, which is needed to perform reliable automated analyses on the texts. the consultation measures show that even the most proficient learners relied heavily on the tools when responding to the prompts, indicating that the tasks were still challenging for them. the data of this pilot study will be made available on osf, licensed under cc by-nc-sa. ## future work in the future, we will collect data on the writing processes and products of multilevel learners, by having them respond to the piloted prompts in two conditions: with access to deepl (mt) and with access to van dale (obd). we will register their online behaviours with screen capture, keystroke logging, and eye-tracking, and their underlying cognitive processes with stimulated recall.