Tracking the Evolution of Written Language Competence in L 2 S panish Learners (2020)

Abstract: In this paper we present an NLP-based approach for tracking the evolution of written language competence in L2 Spanish learners using a wide range of linguistic features automatically extracted from students’ written productions. Beyond reporting classification results for different scenarios, we explore the connection between the most predictive features and the teaching curriculum, finding that our set of linguistic features often reflect the explicit instructions that students receive during each course.

## introduction in the last few years, research on language acquisition has benefited from the use of natural language processing (nlp) technologies applied to large-scale corpora of authentic texts produced by learners, in both the first and second language context. the empirical evidence acquired from learner corpora, complemented with the increased reliability of linguistic features extracted by computational tools and machine learning approaches, has promoted a better understanding of learners' language properties and how they change across time and increasing proficiency level @xcite . a first line of research has focused on providing automatic ways of operationalizing sophisticated metrics of language development to alleviate the laborious manual computation of these metrics by experts @xcite . a second line of research has taken the more challenging step of implementing completely data-driven approaches, which use a variety of linguistic features extracted from texts to automatically assign a learner's language production to a given developmental level @xcite . a great amount of work has been carried out in the field of second language acquisition where the study of l2 writings is seen as a proxy of language ability development @xcite . in this respect, much related work is devoted to predicting the degree of second language proficiency according to expert-based evaluation @xcite or to modelling the evolution of grammatical structures' competence with respect to predefined grades, such as the common european framework of reference for languages (cefrl) @xcite . given the difficulty of defining a unique indicator of linguistic complexity in the context of l2 language development, a great variety of features from all linguistic levels have been used as input for supervised classification systems trained on authentic learner data for different l2s. such is the case e.g. of @xcite and @xcite , dealing with l2 german and l2 estonian, respectively, and of @xcite , who also provided a features analysis focused on predictive features extracted from both receptive and productive texts in swedish l2. this paper adopts this framework and presents an innovative nlp-based stylometric approach to model writing development in learners of spanish as a second and heritage language. our approach relies on a wide set of linguistically motivated features extracted from students' essays, which have already been shown relevant for a number of tasks related to modelling the 'form' of a text rather than the content. while the majority of previous studies on the evolution of language proficiency in l2 uses cross-sectional data, this study is the first, to our knowledge, using a longitudinal corpus of spanish l2 essays to model writing development. interestingly, a similar approach resulted in the successful prediction of the development of writing competence in a l1 acquisition scenario for the italian language @xcite . contributions in this paper: (i) we present, to the best of our knowledge, the first data-driven study which uses linguistic features from student data to model the evolution of written language competence in spanish as a second language (ssl); (ii) we show that it is possible to automatically predict the relative order of two essays written by the same student at different course levels using a wide spectrum of linguistic features; (iii) we investigate the importance of linguistic features in predicting language growth at different course levels and whether they reflect the explicit instruction that students receive during each course. ## motivation and approach studies of l2 writing have focused on linguistic complexity as an indicator of writing development @xcite . this construct, however, is still ill-defined, as evidenced by the divergent measures of complexity utilized in different studies. typical measures of complexity have been the length of the t-unit @xcite , the number of subordinate clauses in a text, or type to token ratios, among others. instead of considering the construct as being multidimensional @xcite and, thus, encompassing an array of different features, most studies have selected one or two of these measures and used them as single indicators of complexity @xcite . this has prevented the development of much needed research that associates different steps of linguistic and written development with specific sets of characteristics. this situation has also prevented the formation of an indepth picture of how those specific aspects develop in relation to the grammatical, lexical or stylistic content taught in classes at different language course levels. this second objective of characterizing writing at different proficiency levels may provide useful insights into how writing samples could be used for placement tests or other assessments to determine which language course is best suited to further develop a student's linguistic skills. in the concrete case of ssl, the literature indicates that one of the most difficult aspects to master for learners is the language's complex verbal morphology @xcite , given that verbal inflections express a complex cluster of person, number, tense, aspect and mood. therefore, ssl courses tend to propose a step-bystep introduction to these different aspects of verbal morphology, generally following this order: (1) per-son and number in the present indicative, (2) past tenses (i.e., imperfect vs. preterite vs. pluperfect), and (3) mood (subjunctive vs. indicative). if this typical instructional sequence had to influence students' writing, it would be expected that learners show an increase in the variety of inflections that they are able to use over time. nonetheless, several studies also indicate that a linguistic feature that has been learned in class may be mastered in exercises that focus on explicit knowledge but take additional time to unfold in tasks that require more implicit knowledge, such as free writing @xcite . this means that a simple classification of students' proficiency based on the presence or absence of features studied in a particular course may not be accurate, as some students may explicitly know the rules for a specific inflectional distinction but still be unable to use them accurately in writing. taking lack of use in writing as evidence for lack of explicit knowledge could entail that students be mistakenly invited to enroll in courses where those features that do not show in their writing are unnecessarily explained to them again. a better approach would thus be to know what students are able to do when they are enrolled in different courses and, only then, compare those abilities to see which match, or mismatch, the contents seen in that particular class. by using a large set of linguistic features, it is possible to understand which phenomena change across proficiency levels and whether they are explicitly related to the teaching guidelines. this study aims at tackling some of the still open methodological issues in the literature on spanish acquisition by decomposing the problem into two main research questions: (i) verify if it is possible to predict the relative order of two essays written by the same student at different course levels using a wide set of linguistic predictors automatically extracted from spanish l2 written productions; (ii) understand which typologies of language phenomena contribute more to the identification of writing skills' evolution and whether such properties reflect the teaching guidelines of the courses. following the approach devised in @xcite we addressed the first research question as a classification task: given a pair of essays written by the same student and ordered according to the course level (d 1 , d 2 ), we classify whether c(d 2 ) > c(d 1 ), where c(d 1 ) and c(d 2 ) correspond respectively to the course levels during ## experiments we train a linearsvm that takes as input pairs of essays written by the same students according to all the possible pairs of course levels (e.g. spa 1 -spa 2, spa 2 -spa 3, etc.). specifically, we extract for each pair the linguistic features corresponding to the first and second essays and the difference between them. we standardize the input features by scaling each component in the range [0, 1]. to test the actual efficiency of the model, we perform the experiments with a 5-cross validation using different students during the training and testing phases. in order to provide our system with negative samples, we expand our datasets by adding reversed samples. since the students were asked to write essays responding to different prompts, we devise two set of experiments, pairing all the essays written by the same students that have: (i) the same prompt; (ii) both same and different prompts. also, because of the small number of training samples for certain pairs of course levels we also decide to perform the experiments on a sentence-level, extracting the linguistic features for each sentence in the longitudinal subset of the cows-l2h corpus and pairing them on the basis of the previously defined criteria. in order to obtain reliable results both on the document and sentence configurations, we consider only datasets at different pairs of course levels that contain at least 50 and 20 samples (including negative pairs) respectively. all the classification experiments are performed using the majority class classifier as baseline and accuracy as the evaluation metric. ## conclusion we present a first study aimed at modeling the evolution of written language competence in spanish as a second and heritage language, using data from the cows-l2h corpus. we have described a rich set of linguistic features automatically extracted from student writing, and have demonstrated that it is possible to automatically predict the relative order of two essays written by the same student at different course levels using these features, especially when considering students enrolled in beginner-level spanish courses. finally, we have shown that the linguistic features most important in predicting essay order often reflect the explicit instruction that students receive during each course. this work can help instructors and language researchers better understand the specific linguistic factors which contribute to improved writing proficiency. additionally, the appearance of features in the linearsvm ranking helps clarify the effect of instruction on writing performance, specifically on effects such as the known delay between students being taught a concept and that concept appearing in the students' writing. we also believe that this work may contribute to the development of better language assessment and placement tools. in future work we intend to explore the influence of student l1 on feature rankings, as l1 (and l2) transfer and interference effects may influence the rate at which students acquire specific linguistic features. additionally we plan to conduct a crosslingual analysis, investigating how the feature rankings we see in spanish writing development relate to those seen in the acquisition of other languages.