Becoming Linguistically Mature: Modeling E nglish and G erman Children’s Writing Development Across School Grades (2020)

Abstract: In this paper we employ a novel approach to advancing our understanding of the development of writing in English and German children across school grades using classification tasks. The data used come from two recently compiled corpora: The English data come from the the GiC corpus (983 school children in second-, sixth-, ninth- and eleventh-grade) and the German data are from the FD-LEX corpus (930 school children in fifth- and ninth-grade). The key to this paper is the combined use of what we refer to as ‘complexity contours’, i.e. series of measurements that capture the progression of linguistic complexity within a text, and Recurrent Neural Network (RNN) classifiers that adequately capture the sequential information in those contours. Our experiments demonstrate that RNN classifiers trained on complexity contours achieve higher classification accuracy than one trained on text-average complexity scores. In a second step, we determine the relative importance of the features from four distinct categories through a Sensitivity-Based Pruning approach.

## introduction there is growing recognition among researchers, educators and policymakers that literacy and the language of schooling (other terms include academic language, language of education, scientific language) are key to children's overall educational success and academic achievement (see, e.g., @xcite . children are expected to acquire the ability to comprehend and produce complex clause and sentence structures, sophisticated vocabulary and informationally dense texts characteristic of language of schooling as they progress through their school career (see, e.g., @xcite . however, this ability is acquired gradually and for many school children only with diffi-culty @xcite . given the key role of academic language, it is somewhat surprising that relatively little empirical research has been conducted on the development of academic language skills across school grades in children's first language, in particular in the area of writing (for exceptions, see, @xcite . this paper contributes to and expands the scant literature by investigating the development of linguistic complexity in children's writing from second-, sixth-, ninth-and eleventh-grade in english schools and fifth-and ninth-grade in german schools. we employ a novel approach to the automatic assessment of text complexity. in this approach, a series of scores for a given complexity measure is obtained through a sliding window technique, tracking the progression of complexity within a text, captured in what we refer to as 'complexity contours' @xcite . these contours are then fed into recurrent neural network (rnn) classifiersadequate to take into account the sequential information in the contours -to perform grade-level classification tasks. we demonstrate the utility of the approach by comparing the performance of 'contour-based' rnn models against those of 'means-based' rnn models trained on text-average performance scores. in a second step, we determine which features drive classification accuracy through a sensitivity-based pruning (sbp) approach. the remainder of the paper is organized as follows: section 2 provides a concise overview of related work on automated assessment of text complexity in combination with machine learning techniques in the language learning context. section 3 presents the two data sets representing english and german children's school writing. section 4 introduces our approach to assessment of text complexity based on a sliding-window technique, whereas section 5 introduces the features investigated in the paper. sections 6 describes the model architecture and the training procedure (section 6.1) and the sbp method used to determine the relative feature importance (section 6.2). sections 7 presents the results and concluding remarks follow in section 8. ## related work in recent years, there has been an increased interest in automated assessment of text complexity in authentic contextualized language samples in combination with machine learning techniques @xcite , for a recent overview). as a valuable complement to experimental research, this research has the potential to advance our current understanding of (both first and second) language learning and development @xcite . important steps have been made in this direction through both language input and language output perspectives: regarding the former, a number of studies have examined whether and to what extent learning materials show an adequate level of linguistic complexity considered to be of crucial importance for successful learning outcomes (see, e.g., franc ¸ois and fairon, 2012; @xcite @xcite . for example, @xcite employ a text classification approach to examine to whether and to what extent reading complexity of school textbooks differ systematically across grade levels in line with the so-called 'systematic complexification assumption'. they build text classification models using a sequential minimal optimization (smo) algorithm trained on a wide range of lexical, syntactic, morphological, and cohesionrelated features to predict the grade level (fifth to tenth grade) and school track (high vs. low). the best performing model reached a grade-level classification accuracy of 53.7%, corresponding to a 20.7% over the random baseline, providing only partial support for thee systematic complexification assumption. in addition, they report significant differences across grade levels and tracks for some of the ten linguistic features. regarding the latter, a rapidly growing body of research has focused on language output aiming to determine to what extent l2 writing and speaking differs from that of their l1 peers and expert writers/speakers, to differentiate levels of language proficiency, to predict human ratings of the quality of learner productions, and to examine to the relationship between l1 and l2 writing complexity and speaking fluency (see, e.g., @xcite @xcite . much research in this area has focused on english and on populations of upper intermediate to advanced l2 learners (but see @xcite @xcite . two recent studies are particularly relevant for the purposes of the present study. @xcite zoom-in on the development of vocabulary sophistication in english children's writing across second-, sixth-, ninth-and eleventh grade. their corpus (also used in the present paper) consists of 2,898 texts of children's writing produced by 983 children. through a mixed-effects regression modeling approach, they assess the effects of grade level and genre on lexical sophistication -measured through children's use of low-frequency words and register appropriate words. their analysis reveals no significant differences with regard to the average frequency of the lexical words used by younger and older children. however, with increasing age children's writing display a shift from a more fictionlike vocabulary to a more academic-like vocabulary, reflecting a development towards more register appropriate word use. @xcite focus on german children's writing development through a text classification approach based on a broad range of complexity and accuracy measures. their dataset includes 1,633 texts of writing from 727 german elementary school children from first to fourth grade and 906 secondary school students from fifth to eighth grade, who attended either a basic or an intermediate school track. using smo classifiers with a linear kernel, their best performing classification model employed a combination of linguistic complexity features, error rate and meta information on topic and school track to reach an accuracy of 72.68% in classifying four grade level categories. their analysis further revealed a shift in the primary locus of development from accuracy to complexity within elementary school and an increasing linguistic complexity in secondary school, in particular in the lexical complexity domain. ## data the data used in this study come from two recently compiled corpora representing school writing: the english data come from the the growth in grammar corpus (gig, https://gigcorpus.com/ ) that comprises 2,898 texts produced by 983 children in 24 different schools from 14 cities in great britain. the texts in the gig corpus were sampled at four points that mark 'key stages' of the english school system: the ends of key stage (ks) 1 (year 2, when children are 6-7 years old) and ks2 (year 9, when children are 10-11 years old), encompassing the primary phase of the school system, and the ends of ks3 (year 9, when children are 13-14 years old) and ks4 (year 11, when children are 15-16 years old), encompassing the secondary stage. the texts were classified into two text types (literary and non-literary texts) on the basis of their overall purpose. approximately 13% of the texts were written by children categorized as speaking english as an additional language. the german data come from the forschungsdatenbank lernertexte (fd-lex; https://fd-lex.uni-koeln.de/ ), a research database of learner texts compiled in joint project of the mercator institute for language promotion and german as a second language. it contains a total of 5,628 texts from two text types (report and argumentation) collected from a total of 930 school children in grades five (when children are 10-11 years old) and nine (when children are 14-15 years old) at comprehensive and grammar schools in two german cities. these texts were elicited using a narrative and an argumentative writing prompt. the database contains information on a number of learner background variables, including the learners language background distinguishing monolingual german students from students who have german as a their first language (l1) and know at least one additional language and students for whom german is not their first language. ## automatic assessment of text complexity through a sliding window technique text complexity of the writing samples in the two corpora is automatically assessed using the complexity contour generator (cocogen), a computational tool that implements a sliding-window technique to generate a series of measurements for a given complexity measure (cm) @xcite @xcite . this approach enables a 'local assessment' of complexity within a text, in contrast to the standard approach if the window size is specified to be greater than one sentences, the denominators and numerators of the fractions from the first to the last sentence in the window are added up to form the denominator and numerator of the resulting complexity score of a given window (see figure 3 in the appendix). the size of the window is user-defined parameter whose value depends on the goals of the analysis: when windows is set to the minimum, i.e. complexity is measured at each sentence of a text, the resulting complexity contour will typically exhibit many sharp turns. by increasing the window size, i.e. the number of sentences in a window, the complexity contour can be smoothened akin to a moving average technique (see figure 4 in the appendix). to compute the complexity scores, cocogen uses the stanford corenlp suite @xcite for performing tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic parsing using the probabilistic context free grammar parsers for english @xcite and german @xcite . ## features in its current version cocogen features 57 complexity measures (cms) for english of which 13 are also available for german. 2 these features cover (1) surface measures, (2) measures of syntactic complexity, (3) measures of lexical richness, (4) information theoretic measures, and (5) register-based n-gram frequency measures. the operationalizations of the syntactic and lexical cms follow those given in @xcite and @xcite . for details on the operationalization of the information theoretic cms, see @xcite . 3 ) lexical sophistication, i.e. the proportion of relatively unusual or advanced words in the learner's text (words from the new academic word list (nawl), words from the new academic formula list (nafl), words that are not part of the new general service list (ngsl), lexical sophistication bnc (ls.bnc), lexical sophistication anc (ls.anc)). the three information-theoretic measures are kolmogorov deflate (koldef), kolmogorov deflate syntactic (koldefsyn), kolmogorov deflate morphological (koldefmor) (see @xcite , for the benefits of using these measures in the assessment of text complexity in the context of language learning). these measures, use the deflate algorithm @xcite to compress a given text and obtain performance scores by relating the size of the compressed file to the size of the original file @xcite . the fifth group of register-based n-gram frequency measures was based on list of the top 100,000 most frequent ngrams (for n 2 [1, 5]) from the five register subcomponents of the coca corpus 3 (spoken, magazine, fiction, news, academic language). the general definition of these cms is given in (1) and 69 (2): let a n,s be the list of n-grams (n 2 [0, 5]) contained within a sentence s, b n,r the list of n-grams on the n-gram frequency list of a register r (r 2 {acad, acad, fic, mag, news, spok}) and c n,s,@xmath0,s \ b n,r the intersection list. furthermore, u n,s denotes the list of unique n-grams in s, and freq n,r (a) the frequency of n-gram a in the n-gram frequency list of register r. the score of a given n-gram-based cms is thus obtained by multiplying the number of n-grams in a sentence that are on the n-gram list with the log of the product of the corresponding frequencies of those n-grams divided by the number of distinct n-grams in the sentence. ## results we report the results of classification with 10-fold cross-validation (see figures 5 and 6 in the appendix for a visualization of model accuracy for the means-based and contour-based models over 200 epochs across the 10 cross-validation folds). we first present the results of the experiments on the english data, before moving to results for the german data. the performance metrics of the classification models for english (global accuracy, precision, recall and macro f1 scores per grade level) are presented in the results of the feature importance analysis reveal that classification is mainly driven by features related to vocabulary (the feature importance statistics for the top 30 measures can be found in ## conclusion and outlook in this paper, we demonstrated how the automatic assessment of text complexity through a sliding window approach in combination with machine learning techniques can provide valuable and unique insights into the development of children's writing as they progress through their school education. such an approach has the added advantage of capturing the progression of complexity within a text. in classification tasks on two data sets representing children's school writing in l1 english and german, we showed that the inclusion of this sequential information can substantially increase classification performance across grade-levels. we also show that sensitivity-based pruning is a viable complementary approach to other approaches aimed at assessing feature importance to identify 'criterial features' that are characteristic and indicative of language competencies at a given level @xcite . more generally, the type of research presented in this paper has the potential to advance our understanding of the development of literacy skills in children during adolescent years, a key stage that is still not well understood. in future work, we intend to extend the approach presented here to larger cross-sectional data sets covering additional school grades in search of valid and reliable benchmarks and norms that can be used to inform school curricula and educational standards.