Automated Classification of Written Proficiency Levels on the CEFR -Scale through Complexity Contours and RNN s (2021)

Abstract: Automatically predicting the level of second language (L2) learner proficiency is an emerging topic of interest and research based on machine learning approaches to language learning and development. The key to the present paper is the combined use of what we refer to as ‘complexity contours’, a series of measurements of indices of L2 proficiency obtained by a computational tool that implements a sliding window technique, and recurrent neural network (RNN) classifiers that adequately capture the sequential information in those contours. We used the EF-Cambridge Open Language Database (Geertzen et al. 2013) with its labelled Common European Framework of Reference (CEFR) levels (Council of Europe 2018) to predict six classes of L2 proficiency levels (A1, A2, B1, B2, C1, C2) in the assessment of writing skills. Our experiments demonstrate that an RNN classifier trained on complexity contours achieves higher classification accuracy than one trained on text-average complexity scores. In a secondary experiment, we determined the relative importance of features from four distinct categories through a sensitivity-based pruning technique. Our approach makes an important contribution to the field of automated identification of language proficiency levels, more specifically, to the increasing efforts towards the empirical validation of CEFR levels.

## introduction the common european framework of reference (cefr) is an internationally recognized standard for describing language proficiency based on six reference levels -a1, a2, b1, b2, c1 and c2 -the same letter pairs corresponding to a three level distinction between beginner, intermediate and advanced @xcite . each proficiency level is related to specific linguistic features and skills, establishing a progression from rudimentary language to varied and sophisticated language. the cefr descriptors, available for all four fundamental language skills (receptive skills: reading & listening and productive skills: writing & speaking), describe the expected competencies in terms of functional can-do statements. for example, a learner at a vantage or upper intermediate b2 level in the domain of writing is expected to have "a sufficient range of language to be able to give clear descriptions, express viewpoints and develop arguments without much conspicuous searching for words, using some complex sentence forms to do so" @xcite . the can-do descriptors formulated for each of the six cefr proficiency levels are typically vague and subjective and are only useful for orientation purposes. thus, there is an urgent need for research on empirical validation of cefr levels at the interface between areas of language learning, testing and assessment, and natural language processing and machine learning @xcite . a closely related line of research has been directed toward automated essay scoring (aes) (for overviews see @xcite @xcite ). this line of research has benefited from the increasing availability of publicly accessible learner corpora of l2 writing, such as clc-fce @xcite , toefl11 @xcite , merlin @xcite and efcamdat @xcite . supervised approaches to aes have recast the task as (1) a regression task aimed at predicting the score of an essay @xcite , (2) a classification task aimed at classifying a text as belonging to one of a specified number of classes, e.g. the three score levels (low, medium or high) in the toefll11 corpus or the six cefr levels (a1-c2) in the merlin corpus @xcite @xcite ; caines and buttery, 2020) 1 or (3) a ranking task aimed at ranking two or more texts based on their quality @xcite . previous work on aes has taken both feature-based approaches and neural approaches (see @xcite for a discussion of (dis)advantages of these two approaches). the features employed are diverse, ranging from the use of descriptive metrics of the text related to word or sentence length to more abstract features related to proficiency development in the area of (second) language learning ( @xcite for a recent overview). the existing studies that have used a feature-based approach have typically relied on text averages of a given feature. however, the use of such aggregate scores may obscure the considerable degree of variation in distribution of feature values within the text. in this paper, we present experiments geared towards the automated assessment of written language proficiency of non-native learners (l2) of english. for the experiments, we take advantage of the ef-cambridge open language database (ef-camdat, @xcite ), a large-scale learner corpus consisting of 1.8 million texts labeled with the six cefr proficiency levels (a1-c2). the aim of the paper is twofold: (1) to apply a sliding window technique in a feature-based modeling approach to automated proficiency classification and (2) to determine what features contribute the most to the classification accuracy. the features employed in this paper are derived from numerous studies in the field of l2 acquisition centering around the notion of 'complexity' 2 (see e.g. @xcite @xcite ). the inclusion of such features is further motivated by the fact that, according to the cefr descriptors, learners are expected to acquire the ability to produce increasingly varied and sophisticated written language, as they progress through the six proficiency levels. such diverse and sophisticated language use should be evident not only in vocabulary growth, but also in the choice of individual 1 the latter paper has been published in the context of a recent shared task on language proficiency scoring at the lrec 2020 -reprolang task d.2 https://lrec2020.lrecconf.org/en/reprolang2020/selected-tasks/ 2 complexity -commonly defined as "the range of forms that surface in language production and the degree of sophistication of such forms" @xcite -is one of the three dimensions of the 'complexity-accuracy-fluency' triad that has emerged as a prominent conceptual framework for l2 assessment (see e.g. @xcite words and multi-word phrases, and in the complexity of sentence, clause, and phrase structures. through the sliding window technique we obtain a series of measurements for a given feature tracking the progression of complexity within a text in a sentence-by-sentence fashion. we refer to such series of measurements as 'complexity contours'. these contours are then fed into recurrent neural network (rnn) classifiers -adequate to take into account the sequential information in the contours -to perform grade-level classification tasks. we demonstrate the utility of the approach by comparing the performance of 'contour-based' rnn models against those of 'means-based' rnn models trained on text-average performance scores. in a second step, we determine what features drive classification accuracy through a sensitivity-based pruning (sbp) technique. the approach taken in this paper was already successfully applied in the area of first language (l1) writing development. @xcite showed that rnn classifiers trained on complexity contours achieve higher classification accuracy in predicting secondary school children's grade levels in both english and german (second-, sixth-, ninth-and eleventh-grade in english schools and fifth-and ninth-grade in german). here we set out to extend the approach to automated proficiency classification in l2 english. the remainder of the paper is organized as follows: in section 2, we provide a concise overview of related work. section 3 presents the dataset and section 4 the features used in the experiments. section 5 describes the sliding-window approach to generating complexity contours. sections 6 presents the architecture of the rnns and the training procedure. sections 7 introduces the sbp method used to determine the relative feature importance. section 8 reports the results before conclusions are drawn in section 9 along with indications of future research directions. ## related work in this section, we present two types of previous work: l2 studies that have investigated the relationship between certain linguistic features and proficiency levels, and those that have used supervised machine learning approaches to predict learner proficiency on the cefr scale. numerous studies reveal that syntactic complexity can be considered as one of the key skills that strongly influence l2 proficiency (see e.g. @xcite for a synthesis of twenty-five studies, see also @xcite for a recent special issue). these studies have measured this multidimensional construct along both global features, such as length measures and subordination ratios, as well as more specific features pertaining to the usage of particular structures. for example, @xcite conducted an evaluation of 14 features of syntactic complexity in a corpus of english essays written by chinese l1 students and found that the complexity measures that best discriminated between proficiency levels were the number of complex nominals per sentence and the mean sentence length. another series of studies indicate the importance of lexical complexity (aka lexical richness) subsuming its three sub-dimensions (lexical density, sophistication and variation) in the assessment of l2 proficiency (see @xcite ). for example, @xcite showed 47.5% of the variance in holistic scores of lexical proficiency and 48.7% of the variance in holistic scores of speaking proficiency can be explained using a range of lexical sophistication indices. this study also introduced the use of multi-word sequences (mws) as an indicator of language proficiency, operationalized in terms of register-specific n-gram measures (bigrams and trigrams). the inclusion of such features reflects the growing interest of mws in language learning and development. this interest stems from an extensive body of evidence demonstrating that both child and adult populations, including adult second-language learner populations, can develop the sensitivity to the statistics of mws and rely on knowledge of such statistics to facilitate their language processing and boost their acquisition (for overviews see e.g. @xcite ). @xcite , for instance, investigated the relationship of the usage of mws and human judgments of writing proficiency based on the cefr-graded yonsei english learner corpus @xcite and found that essays from higher cefr levels include a greater proportion of frequent academic trigrams and more strongly associated spoken trigrams. finally, in recent years the l2 literature has introduced information-theoretic features based on kolmogorov complexity @xcite . @xcite investigated essays written by advanced learners of english from international corpus of learner english @xcite and showed that more advanced learners use considerably more complex texts than beginner learners, although this tendency is not always reflected in a clear, linear relationship between proficiency and complexity. studies that have employed supervised machine learning approaches to predict proficiency on the cefr scale for different l2s have used numerous linguistic features in combination with a host of classifiers @xcite @xcite @xcite . the classification accuracy reported in these studies ranged between 62.7% and 83.8%. @xcite reached a classification accuracy of 62.7% in predicting five (out of six) cefr levels of professionally rated free text essays from the merlin database comprising cefr exams taken by second language learners of german based on a total of 3821 lexical, morphological, and syntactic features using the sequential minimal optimization (smq) algorithm implemented in weka. using the same smq algorithm, @xcite achieved an accuracy of 67% in correctly identifying the cefr level of l2 swedish learner essays on the basis of 61 count-based, lexical, syntactic, morphological, and semantic features extracted from the linguistic annotation available in the swell corpus foot_0 . @xcite achieved 70% accuracy in predicting cefr-levels of l1 french and spanish l2 english users on the basis of manually annotated errors in the l1 french and spanish subsets of the efcamdat corpus. @xcite reported a classification accuracy of 79% in an experiment on the estonian interlanguage corpus 4 . in another study, @xcite performed experiments with cross-lingual and multilingual classifiers on individual language classification. the data used in their study included 2,286 manually graded texts (five levels, a1 to c1) from the merlin learner corpus (german, 1,029 texts; italian, 803 texts, and czech, 434 texts). trained on a wide range of feature, such as word and pos n-grams, task-specific word and character embeddings, dependency n-grams, features pertaining to lexical richness and error features, their classification models obtained an accuracy of 0.68 for german, 0.84 for italian and 0.73 for czech for monolingual classification. for multilingual classification, their models reached classification accuracy up to 0.73. the data set and findings ob-tained in this study have served as the baseline for the reprolang 2020 shared task on 'language proficiency scoring' foot_2 . in the context of this task, caines and buttery (2020) reproduce and extend the finding described in @xcite reaching a classification accuracy of up to 83.8% for the italian component. their results indicate that feature-based approaches perform better than neural network classifiers for text datasets of the given size. ## data the data come from the efcamdat, an open access corpus compiled at cambridge university in collaboration with ef education first, an international school of english as a second/foreign language @xcite . the corpus consists of writing assignments submitted to the englishtown, the online school of ef education first, summing up to a total of 1,180,309 individual writing samples by 174,743 l2 learners. the curriculum of englishtown covers all six proficiency levels, from cefr a1 to c2 organized along 16 ef teaching levels with each level subsuming 8 teaching units and ending with an open-ended writing task (128 distinct writing assignments). the length of the writing samples in the corpus increases monotonically with englishtown levels, ranging from an average of 30.1 words at the lowest level (1) to an average of 170 words at the highest level (16). since one of our main aims is to demonstrate the usefulness of the sliding window technique and the inclusion of a set of measurements per individual feature (complexity contours), we filtered the original dataset to obtain texts containing at least 100 words. this resulted in a total of 163,657 writing samples. in addition, we removed texts that had received exceptionally low scores on writing performance, since their inclusion would add bias and variance and may skew the results @xcite . specifically, texts whose writing score fell more than 1.5 times the interquartile range below the first quartile, corresponding to a threshold score of 75%, were removed, which resulted in a loss of 7% of the data. the final dataset comprised a total of 152,314 individual learner texts. whose distributions across cefr levels along with associated text length statistics are shown in ## features the 57 features used in this paper fall into four distinct groups: (1) measures of syntactic complexity, (2) measures of lexical richness, (3) measures pertaining to the usage of multi-word sequences (mws) and (4) information-theoretic measures. the first group consists of 16 features used in the past to measure syntactic complexity in writing and its relation to writing proficiency reviewed in section 2. these features are implemented based on descriptions in @xcite and using the tregex tree pattern matching tool @xcite with syntactic parse trees for extracting specific patterns. the second group subsumes 13 features pertaining to lexical richness: five measures of lexical variation, one measure of lexical density, seven measures of lexical sophistication. the operationalizations of these measures follow those described in @xcite and @xcite . the third group includes 25 n-gram frequency features that are derived from the five register sub-components of the contemporary corpus of american english (coca, @xcite ): spoken, magazine, fiction, news and academic language foot_3 . our frequency n-gram measures differ from those used in the earlier studies reviewed in section 2. instead of using only bigrams and trigrams, we extend them to include longer word combinations (four-and fivegrams) and use a more nuanced definition to operationalize the usage of such combinations given in equation ( 1 ): a total of 25 measures results from the combination of (a) a 'reference list' containing the top 100,000 most frequent n-grams and their frequencies from one of five register subcomponents of the coca corpus and (b) the size of the n-gram (n 2 [1, 5]). the fourth group includes three information-theoretic measures that are based on kolmogorov complexity. these measures use the deflate algorithm @xcite to compress a text and obtain complexity scores by relating the size of the compressed file to the size of the original file (for the operationalization and implementation of these measures see @xcite ). ## a sliding-window approach text complexity of the writing samples was automatically assessed using the cocogen, a computational tool that implements a sliding-window technique to generate a series of measurements for a given complexity measure (cm) @xcite . in contrast to the standard approach that represents text complexity as a single score, providing a 'global assessment' of the complexity of a text, the use of a sliding-window technique enables a 'local' (sentence-level) assessment of complexity within a text. a sliding window can be conceived of as a window of size ws, which is defined by the number of sentences it contains. the window is moved across a text sentence-by-sentence, computing one value per window for a given cm. the series of measurements generated by cocogen captures the progression of linguistic complexity within a text for a given cm and is referred here to as a 'complexity contour' (see figure 1 ). to compute the complexity score of a given window, a measurement function is applied to each sentence in the window. the size of the window (ws) is user-defined parameter whose optimal value depends on the goals of the analysis: when complexity is measured for each sentence, i.e. @xmath0, the resulting complexity contour will typically exhibit many sharp turns. by increasing the window size, i.e. the number of sentences in a window, the complexity contour can be smoothened akin to a moving average technique. 7 in this paper, the window size parameter was set to one sentence, meaning that no smoothing of the curve was performed. figure 1 illustrates complexity contours on three randomly selected texts across three cefr levels (a2, b2 and c2) for eight selected complexity measures. cocogen uses the stanford corenlp suite @xcite for performing tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic parsing (probabilistic context free grammar parser @xcite ). ## classification models we used a recurrent neural network (rnn) classifier, specifically a dynamic rnn model with gated recurrent unit (gru) cells @xcite . a dynamic rnn was chosen as it can handle sequences of variable length 8 . as shown in figure 2 , the input of the contour-based model is a sequence @xmath1, x 2 , . . . , x l , x l+1 , . . . , x n ), where x i , the output of cocogen for the ith window of a document, is a 57 dimensional vector, l is the length of the sequence, n 2 z is a number, which is greater or equal to the length of the longest sequence in the data and x l+1 , • • • , x n are padded 0-vectors. the input of the contour-based model was fed into a rnn that consists of two layers of gru cells with 200 hidden units for each. to predict the class of a sequence, the last output of the rnn, i.e. the output of the rnn right after the feeding of x l , is transformed through a feed-forward neural network. the feed-forward neural-network consists of three fully connected layers, whose output dimensions are 512, 256, and 6 respectively. the rectifier linear unit (relu) was used as an activation function. before the final output, a softmax layer was applied. for the mean-based model, we used the same neural network as in the contour-based model, except that the network was trained with vectors of text-average complexity scores. the models were implemented using pytorch @xcite . we evenly split our data into 10 folds and applied a 10-fold cross validation, i.e. each time a fold (10% of data) is taken out as test set and the rest (90% of data) are used as the training set. in both 7 when the window size is specified to be greater than 1, cocogen returns complexity scores for a given measures as fractions (wnm/wdm). in this case, the denominators and numerators of the fractions from the first to the last sentence in the window are added up to form the denominator and numerator of the resulting complexity score of a given window. 8 the lengths of the feature vector sequences depends on the number of sentences of the texts in our corpus. as the loss function for training cross entropy was used: in which c is the true class label of the current observation, c is the number of classes, (p(y 1 ), . . . , p(y c )) is a one-hot vector with p(y i ) = ## feature ablation to determine the relative importance of the individual features, we conducted feature ablation experiments for the contour-based rnn. classical forward or backward sequential selection algorithms that proceed by sequentially adding or discarding features require a quadratic number of model training and evaluation in order to obtain a feature ranking @xcite . in the context of neural network model training a quadratic number of models can become prohibitive. to alleviate this problem, we used an adapted version of the iterative sensitivity-based pruning algorithm proposed by @xcite . this algorithm ranks the features based on a 'sensitivity measure' @xcite and removes the least relevant variables one at a time. the classifier is then retrained on the resulting subset and a new ranking is calculated over the remaining features. this process is repeated until all features are removed (see algorithm 1). in this fashion, rather than training n(n+1) 2 required for sequential algorithms, the number of models trained is reduced to n m , where m is the number of features that can be removed at each step. we report the results obtained with @xmath2, i.e. the removal of a single feature at each step. the procedure of finding the rank order of feature importance is de-scribed as following. to increase the robustness of the feature importance rank order, k-fold crossvalidation is applied. at step t, neural network models m t,n , n 2 {1, . . . , k} are trained on the training sets of a k-fold cross-validation, where n is the fold id. the training sets at step t consist of instances with feature set f @xmath3, f 2 , . . . , f dt } where f 1 , . . . f dt are the remaining features at the current step, whose importance rank is to be determined. we define x t,n as the test set of the nth fold with feature set f t and x i t,n as the same dataset as x t,n except we set the i th feature f i of each instance within the dataset to its average. furthermore, we define g(x) as the classification accuracy of m t,n for a dataset x. the sensitivity of feature f i on the nth fold at step t is obtained from: the most important feature at step t can be found by: then we set the rank for feature f î: rank î = t in the end, feature f î is dropped from f t and the corresponding columns in training and test dataset are also dropped simultaneously: ## results and discussion an overview of the performance statistics of the models in terms of precision, recall and f1 scores is presented in 3 . as is evident in this table, most classification errors appeared in adjacent categories, with few classification errors occurring between distant categories. the top 20 features that contributed most to the classification accuracy of the contour-based rnn model are shown in ## conclusion and outlook in this paper, we applied a sliding window technique in a feature-based modeling approach to automated classification of written proficiency levels on the cefr-scale (a1-c2 levels). we made use of 'complexity contours' obtained through this technique to represent the distribution of scores per linguistic feature within a text in combination with rnn classifiers that exploit the sequential information in those contours. we demonstrated that an rnn classifier trained on complexity contours achieved higher classification accuracy across all six cefr proficiency levels compared to one trained on text-average scores with an increase in performance of up to 14% in terms of precision and 15% in terms of recall. we also showed that iterative sensitivity-based pruning approach is a viable way of assessing relative feature importance in text classification tasks performed with neural network models. this approach taken in our paper has the potential to provide a valuable contribution to increasing efforts to identify 'criterial features', i.e. features that are characteristic and indicative of language proficiency at each level @xcite . in our future work, we intend to include additional sets of features of language use based on crowd-sourced language metrics entitled word prevalence @xcite as well as liwc-style features that relate language use with behavioral and self-reported measures of personality, social behavior, and cognitive styles @xcite . we also intend to take into account the effects of task type on the features of language use investigated in this paper @xcite .