Cross-corpora experiments of automatic proficiency assessment and error detection for spoken E nglish (2022)

Abstract: The growing demand for learning English as a second language has led to an increasing interest in automatic approaches for assessing spoken language proficiency. One of the most significant challenges in this field is the lack of publicly available annotated spoken data. Another common issue is the lack of consistency and coherence in human assessment. To tackle both problems, in this paper we address the task of automatically predicting the scores of spoken test responses of English-as-a-second-language learners by training neural models on written data and using the presence of grammatical errors as a feature, as they can be considered consistent indicators of proficiency through their distribution and frequency. Specifically, we train a feature extractor on EFCAMDAT, a large written corpus containing error annotations and proficiency levels assigned by human experts, in order to extract information related to grammatical errors and, in turn, we use the resulting model for inference on the CLC-FCE corpus, on the ICNALE corpus, and on the spoken section of the TLT-school corpus, a collection of proficiency tests taken by Italian students. The work investigates the impact of the feature extractor on spoken proficiency assessment as well as the written-to-spoken approach. We find that our error-based approach can be beneficial for assessing spoken proficiency. The results obtained on the considered datasets are discussed and evaluated with appropriate metrics.

## introduction automatic scoring of language proficiency is becoming a point of growing interest and importance in the field of second language (l2) assessment because the number of english-as-a-second-language (esl) learners has been steadily increasing worldwide @xcite . a common issue in this field is the lack of publicly available data specifically designed and an-notated for automatic assessment, especially as regards spoken data. another typical problem is the lack of consistency and coherence in human assessment, as it frequently relies on proficiency indicators that often have biases and are not clearly generalizable, therefore not easily transferable into automatic scoring systems @xcite . although l2 proficiency cannot be assessed on the mere basis of the presence of errors in learners' written and spoken productions, this aspect is highly consistent and plays a major role in language assessment by human experts @xcite . nevertheless, to the best of our knowledge, the impact of errors on automatic spoken language assessment has not been thoroughly investigated yet, whereas other types of feature-based assessment have been more widely studied and explored @xcite . in this paper, we address the task of automatically predicting the scores of spoken responses of esl learners leveraging written data and exploiting the presence of grammatical errors, thus tackling both the aforementioned problems: the issue related to the scarce availability of spoken data and the problem of inconsistency in human assessment. in order to do so, we design a ranking of grammatical error gravity based on the frequency of each human-annotated error in the ef-cambridge open language database (efcamdat), modelling it across 15 proficiency levels aligned with the cefr (common european framework of reference) levels ranging from a1 to c1 (council of @xcite ; as our purpose is scoring spoken language proficiency, we discard spelling, punctuation and orthographic errors and we group errors into 5 categories. subsequently, we train a feature extraction model feeding the learners' texts of the efcam-dat as inputs and setting the 5 classes of errors as targets for our predictions and we use this model as an error feature extractor (efex) for inference on the cambridge learner corpus -first certificate in english (clc-fce) and on the international corpus network of asian learners of english (ic-nale), thus generating 5 labels corresponding to the aforementioned 5 classes of errors; then, we train a scoring model on the clc-fce injecting the 5 error labels generated by efex and we test it on the spoken annotated section of icnale. likewise, we use efex for inference on the tltschool corpus. subsequently, we train a scoring model on the written section of the corpus injecting the 5 error labels generated by efex and we test it on the spoken section. figure 1 shows the proposed pipeline. finally, we fine-tune our model on a small spoken subset. the structure of the paper is as follows: in the next paragraphs, we briefly illustrate the theoretical framework and literature related to automatic scoring and assessment; in section 2, we describe the data used in our experiments and our ranking of grammatical error gravity; in section 3, we show the model architectures; in section 4, we show the results of our experiments on the models; finally, in section 5, we illustrate the conclusions of the study and reflect upon next steps. ## model architectures we build our models using a bert architecture @xcite in the version provided by the huggingface transformer library @xcite . in both the feature extractor and the scoring models bert layers are frozen. ## conclusions and future work in this work we presented a promising approach to automatic proficiency assessment of spoken responses based on the presence of errors across proficiency levels, extracted with an error feature extractor that we developed using a bert-based architecture. furthermore, we proposed to use models previously trained on written data in order to tackle the problem related to limited availability of spoken data. first, we tried our error-based approach on some publicly available datasets, training our models on the clc-fce and testing them on the icnale. in this case, our efex-enriched model managed to modestly improve the prediction of the dialogues scores in terms of pcc. specifically for this experiment, one also has to consider the difference in domain and scoring metrics between the two corpora, albeit they are approximately around the same proficiency levels. subsequently, we discovered that the use of efex labels shows a more interesting improvement in scoring the spoken section of tlt-school after training our models on written data, suggesting that these additional features can mitigate the impact of asr errors and some typical phenomena of the spoken modality. an example drawn from the data could be the following: "in fact when a person does a lot of movement and moves a lot and goes out in the in the nature then his his body is in more healthy". the repetitions 'in the' and 'his' as well as what appears to be a wrongly inserted preposition 'in' would be considered actual errors if they occurred in written productions, but not necessarily so in spoken texts. our assumption is that bert models, as they are trained on large written corpora, already possess written grammatical knowledge and are sensitive to grammatical violations to a certain extent. therefore, when evaluating written proficiency, they do not need to be warned with explicit indications with regard to errors, but error-related features can be beneficial to understand and decode the typical phenomena of oral language and learn spoken and conversational grammar. considering that in spoken responses the scoring module could take advantage of a distinction of errors made by the speaker or introduced by asr @xcite , we assume that there is still room for improvement in the approaches that detect errors as additional features. further work should be undertaken starting from the first step of our pipeline, i.e. the error feature extractor, since, despite the good results shown in considering that we removed spontaneous speech phenomena such as hesitations and fragments of words from the data for our experiments, we envisage a combination of the approach presented in this paper and the use of error-related features derived from audio recordings, such as phonological errors as well as repetitions and other types of disfluency. moreover, we plan to investigate the impact of models trained on written data and tested on spoken data also for other cefr levels. finally, we acknowledge that the presence of errors cannot be the only feature to be taken into account when assessing l2 proficiency at higher levels, but, if properly weighted and balanced with other proficiency indicators, it might improve consistency and objectivity in assessment.