Mitigating Learnerese Effects for CEFR Classification (2022)

Abstract: The role of an author’s L1 in SLA can be challenging for automated CEFR classification, in that texts from different L1 groups may be too heterogeneous to combine them as training data. We experiment with recent debiasing approaches by attempting to devoid textual representations of L1 features. This results in a more homogeneous group when aggregating CEFR-annotated texts from different L1 groups, leading to better classification performance. Using iterative null-space projection, we marginally improve classification performance for a linear classifier by 1 point. An MLP (e.g. non-linear) classifier remains unaffected by this procedure. We discuss possible directions of future work to attempt to increase this performance gain.

## introduction the need for automated methods in establishing both the readability of a piece of text and the level of linguistic proficiency of its author has been recognised decades before most students started writing essays, compositions and other homework assignments on computers. motivations for creating such automated methods are diverse. seminal work by @xcite focused on alleviating work load of language teachers and fast turn-around of writing feedback to language students. since then, much progress has been made, and a comprehensive overview of original and still standing challenges in this field is presented by @xcite . related to this is the line of research on grammatical error correction @xcite , accompanied by a number of shared tasks @xcite @xcite . much of the work in this sub-field of nlp is usually aggregated under the label automated essay scoring 1 . scoring an essay, however, depends on a 1 or variations thereof: automated essay grading, automated writing evaluation, etc. while the nature of the influence of one's native language (l1) on second language acquisition (sla) is a topic of ongoing debate (richards and rodgers, 2014) and the terms being used are dependent on the assumed framework (interference @xcite , transfer @xcite , influence @xcite )) the fact that there is interaction is uncontroversial. this l1 interaction is problematic in the sense that a classifier trained on texts written by native speakers of chinese may perform poorly on texts written by native speakers of portuguese, for example. inspired by recent successes in debiasing embeddings-based representations for particular traits @xcite @xcite @xcite , we set out to dispose the representations that feed into the classifier of traits that can be taken as signs of l1 influence, to train a single cefr classifier -devoid of l1 features (i.e. learnerese)-that improves its performance when trained on aggregated data from different native speaker groups. the rest of this paper is structured as follows: section 2 discusses earlier work on both cefr classification and debiasing strategies. section 3 explains the data we used in our experiments. section 4 explains the classification setup. section 5 discusses our results and provides pointers to future work. finally, section 6 sums up our main findings. ## related work the task of automated essay scoring itself has received a fair amount of attention over the last decades, see beigman klebanov and madnani (2020) for a comprehensive overview of the current state of the art. individual sub-tasks that can be taken as indicative for proficiency in a given language, such as grammatical error correction (gec), have been accompanied by a number of popular shared tasks @xcite @xcite . the task of cefr classification itself however, seems to have received fewer attention. among the studies that address this problem for various languages are @xcite (italian), hancke and meurers (2013) (german), vajjala and lõo (2014) (estonian) and @xcite (swedish). earlier work on english (our language of interest) is represented by @xcite , who create their own annotated corpus and experiment with automated classification using several classification algorithms. in this paper, we interpret the influence of l1 as an issue of bias in the embeddings-based representation of the english texts. particular word order, article-or gender-based preferences or errors that can be traced back to the native language of the author, are likely to be more ubiquitous within the same group of native speakers. to the best of our knowledge, the cefr classification problem has not been combined before with methods attempting to debias embeddings for l1 features. bias in nlp has attracted a lot of interest recently @xcite @xcite , and the specific mitigation approach that we follow in our work is that of @xcite , who propose inlp -an iterative nullspace projection algorithm to debias gender stereotypes in text. unlike previous approaches @xcite that solely rely on a contrastive wordlist to identify a linear direction for debiasing, inlp debiases all linearly present gender directions in a data-driven manner. considering that a classification task relies on a certain feature that we want to remove, inlp iteratively trains a series of probing (linear) classifiers to predict that feature until the probing classifier is confounded. for more details, we encourage the readers to read the original paper. in @xcite , the potential of this approach was explored for debiasing translation artifacts (which carry similar stylistic differences as learnerese) in human/machine-translated documents. building upon this work, we employ the algorithm for our use-case. ## data our first experiments were done on the international corpus network of asian learners of english (icnale) @xcite , a data set comprising essays from 2.800 authors from over 10 different native speaker groups, annotated for different metrics indicating skill levels (toeic, toefl, ielts, etc.), including cefr labels. when aggregating all data from non-native english speakers, using a vanilla bert @xcite classifier, we obtained a classification accuracy of 0.51, for 2.600 essays foot_1 . for individual native language groups, however, we achieved comparable performance while using considerably fewer training instances (for example, 0.50 on just 200 essays whose authors are from indonesia). at the same time, some native speaker groups in icnale are heavily imbalanced, resulting in simple majority vote classifiers outperforming the trained classifier for those native speaker groups. while these preliminary findings initially inspired us to apply debiasing strategies, we decided to use a larger, less imbalanced corpus for the majority of the experiments reported on in this paper. we extracted a subset of the ef-cambridge open language database (efcamdat) @xcite , consisting of 191,969 texts from authors from china, japan and korea. since all texts in efcamdat are from language learners, we combined this with 200 texts from native english speakers from icnale to get debiasing directions. furthermore, efcamdat only provides information on the author's country of origin. information on native language would be more accurate, but unfortunately is not specified in the corpus. for the purpose of this paper, we will assume the country of origin and native language to align. ## method & results as mentioned in section 3, we were initially inspired by the fact that adding more training data did not seem to improve classification performance. in addition, earlier work indicated that classifying the country of origin of an author based on their english text provides good results, with @xcite reporting an accuracy of 87% on all of ic-nale for this task. we argue that this points at signals of l1 in the english learner texts that a classifier can pick up on, and that consequently, finding a way to make input text more homogeneous to a classifier through debiasing (section 4.1) can lead to cefr classification performance gains (section 4.2). ## discussion for the efcamdat_native_en dataset, we observe a marginal performance gain when using a linear classifier (lr), but not when using a nonlinear classifier (mlp). this can be explained from the results in as illustrated in compared to earlier work, the overall performance of our system scores well. @xcite also work on english and report an accuracy of 53% on their data set 5 . in other related work however, performance seems to depend highly on the specific data set (and language), with reported accuracy figures between 64.5% @xcite and 79% @xcite . 5 moreover, they aggregate the c1 and c2 levels, resulting in 5-way classification, compared to 6-way in our setup. from ## conclusion in this paper, we experiment with compensating for l1 influence in cefr classification by applying a debiasing approach, the idea being to debias the embeddings for learnerese features in any specific l1-related direction. by doing so, we obtain a small performance improvement with a linear classifier. cefr classification performance seems to be highly dependent on the particular corpora/data used, with earlier work reporting accuracy figures between 53% and 79%. on the ef-camdat dataset, results look promising -best weighted f1-score of 83 via logistic regression and even higher (96) with mlp classifier without any debiasing. our code is available on github foot_3 .