TCFLE -8: a Corpus of Learner Written Productions for F rench as a Foreign Language and its Application to Automated Essay Scoring (2023)

Abstract: Automated Essay Scoring (AES) aims to automatically assess the quality of essays. Automation enables large-scale assessment, improvements in consistency, reliability, and standardization. Those characteristics are of particular relevance in the context of language certification exams. However, a major bottleneck in the development of AES systems is the availability of corpora, which, unfortunately, are scarce, especially for languages other than English. In this paper, we aim to foster the development of AES for French by providing the TCFLE-8 corpus, a corpus of 6.5k essays collected in the context of the Test de Connaissance du Français (TCF - French Knowledge Test) certification exam. We report the strict quality procedure that led to the scoring of each essay by at least two raters according to the CEFR levels and to the creation of a balanced corpus. In addition, we describe how linguistic properties of the essays relate to the learners’ proficiency in TCFLE-8. We also advance the state-of-the-art performance for the AES task in French by experimenting with two strong baselines (i.e. RoBERTa and feature-based). Finally, we discuss the challenges of AES using TCFLE-8.

## introduction automated essay scoring (aes) aims to develop algorithms that can assess the quality of essays similarly to humans. the field may be traced back to the seminal work of @xcite . since then, several publications have been studying aes. 2 in the late 1990's, several functional aes systems were already available, either relying on latent semantic analysis (e.g., @xcite ), nlp-extracted features combined with multiple regression (e.g., @xcite or bayesian text classification (e.g., @xcite ). as noted by @xcite , a small amount of essays (less than 1000) could be enough for training such systems in some contexts. however, even collecting such a small corpus was difficult, as the essays need to be manually rated, and essays reliable assessment is a notoriously difficult task for humans @xcite . recent advances in aes have been made possible by deep learning (dl) approaches and large language models @xcite . prominent studies used embeddings @xcite , recurrent neural network @xcite , attention @xcite , and bert-based architectures @xcite . these approaches have also led to a growing need for large corpora. consequently, aes teams have turned their attention to learner corpus research, a branch of corpus linguistics providing large-scale, computerized, naturalistic learner production. pioneering works such as the international corpus of learner english (icle) @xcite and the european science foundation l2 database @xcite demonstrated the potential of such learner data collections for second language acquisition (sla) research, but it is only recently that more learner corpora fitted for aes, i.e., large enough and annotated with different proficiency levels, were developed for various languages @xcite @xcite @xcite . unfortunately, there is no such large corpus for french, making the situation for french aes far from encouraging. the first systems thus relied on unsupervised approaches: @xcite used latent semantic analysis to compare native language (l1) of student essays with textbook passages, whereas auto-eval @xcite automatically captured several l1 essay features, which are heuristically combined. more recently, @xcite trained a naive bayes classifier on a very small corpus of 200 essays written in foreign language (fl). finally, @xcite were the first to fine-tune bert for fl french aes on more data, but they did not release it and only a single l1 is represented. therefore, in order to support the development of aes solutions for french, the need for a large and reliable corpus of written french essays becomes apparent. in this paper, we make two main contributions. first, we provide the community with the tcfle-8 corpus 3 , composed of 6,569 learner essays, with 8 different languages of habitual use, scores, from at least 2 raters, for the 6 levels of the common european framework of reference for languages (cefr) (council of @xcite , and automatically annotated with +5k features. these essays were collected in the context of the official french knowledge test (tcf) exam, one of the main certification exams for french. second, we provide solid baselines for future research in aes using tcfle-8. this is the largest aes modeling study that has been done for french. this paper is organized as follows. section 2 presents the main characteristics and uniqueness of corpora used for aes. we then present the corpus compilation process, presenting the official certification exam which our corpus is based on and the essay selection process (section 3). next, in section 4, we present the tcfle-8 corpus, discussing its size, metadata and annotation. an exploration of tcfle-8 for aes systems is presented in section 5. finally, final remarks are presented in section 6. tcfle-8 is freely available for research purposes. ## related work developing a french corpus for aes means taking part in the field of learner corpus research, which, since its emergence in the late 1980s, gave rise to more than 200 learner corpora around the world 4 . reviews on learner corpora (gilquin, 3 test de connaissance du français (french knowledge test); fle stands for français langue étrangère (french as a foreign language) and 8 refers to the eight different languages of habitual use included in the corpus. among this large body of written learner corpora, we will focus on two types relevant for this work: corpora used for aes and learner corpora targeting french as a foreign language. ## the tcfle-8 corpus at the end of the compilation process, the final tcfle-8 corpus comprises 6,569 essays (581,333 words). some figures about the corpus size by cefr levels are shown in tables 1 , 2 , 3 and 4. it is expected that beginner-level essays tend to be shorter than the other ones @xcite . moreover, the extreme levels (a1 and c2) are less represented in the corpus. this might be caused by two factors: (1) few a1-level learners seek an official language exam since this level is rarely sufficient for official purposes (e.g., employment and visa requirements), and (2) reaching the c2 level in a foreign language is extremely difficult. ## aes for french in this section, we analyze the applicability of the tcfle-8 corpus for training aes systems. for this purpose, we explore two approaches: deep learning, since most aes systems relied on neural networks @xcite , and feature-based machine leaning. we split the anonymized corpus with 80% for training, 10% for validation, and 10% for testing, stratifying by score and language. in addition, to explore the impact of model initialization, we performed 5 repetitions of the training process; in each one, we adjusted the test set so that it does not overlap with the others. we performed a hyperparameter exploration using the accuracy on the validation set. for the deep learning model, we used camem-bert @xcite , a roberta-based model for french. as for the hyperparameters 14 , we use a learning rate of 5e-5 and an early stop of 5. for machine learning, we use the xgboost 15 and logistic regression 16 as a feature-based baseline. these were trained using the 119 features extracted using the method described in section 4.2. the evaluation of these models is shown in in order to characterize human level performance on the task, we report standard aes evaluation metrics for the human raters (column "raters" in ## final remarks in this work, we presented tcfle-8, a corpus of 6,569 candidates' essays written during the french knowledge test (tcf), with 8 different languages of habitual use. this paper described the data gathering by france education international (fei), data cleaning, anonymization, and annotation performed to compile this corpus, which is the largest french corpus targeting french as a foreign language for aes. this corpus, along with its metadata (i.e., essays, metadata and annotation) is available to the community. we also described the learners' proficiency in the corpus using numerous linguistic variables related to sla. this description confirms that these linguistic features could capture developmental patterns in tcfle-8 in a similar fashion to other learner corpora. exploring tcfle-8 for aes, we applied different machine learning algorithms. camembert appears to be more accurate and xgboost, a featurebased model, achieved similar results at beginner level. this raises a question about what features should explored for better describing the intermediate and advanced levels. interestingly, part of this answer may come from the transformer model ## limitations several normalization steps were applied in order to develop a coherent corpus for aes, aiming to compile a high quality corpus illustrating the proficiency levels with learner productions on which professional raters would agree. as a consequence, some potentially interesting cases were removed. this concerns for example texts on the borderline between two levels. although they are interesting cases as they could support studies on understanding of the level boundaries, we opted for a corpus that represents the texts of each level. tcfle-8 is a corpus designed for supporting the research in french as a foreign language, including aes. in this work, the focus is on the corpus compilation. despite the initial tests performed here, our goal does not include an exhaustive verification of the corpus' applications nor an evaluation of various aes approaches. finally, we do not intent nor recommend using tcfle-8 for a fully-automated evaluation environment but to improve writing assessment in french as a foreign language. the results of this evaluation is shown in