Harnessing GPT to Study Second Language Learner Essays: Can We Use Perplexity to Determine Linguistic Competence? (2024)

Abstract: Generative language models have been used to study a wide variety of phenomena in NLP. This allows us to better understand the linguistic capabilities of those models and to better analyse the texts that we are working with. However, these studies have mainly focused on text generated by L1 speakers of English. In this paper we study whether linguistic competence of L2 learners of Swedish (through their performance on essay tasks) correlates with the perplexity of a decoder-only model (GPT-SW3). We run two sets of experiments, doing both quantitative and qualitative analyses for each of them. In the first one, we analyse the perplexities of the essays and compare them with the CEFR level of the essays, both from an essay-wide level and from a token level. In our second experiment, we compare the perplexity of an L2 learner essay with a normalised version of it. We find that the perplexity of essays tends to be lower for higher CEFR levels and that normalised essays have a lower perplexity than the original versions. Moreover, we find that different factors can lead to spikes in perplexity, not all of them being related to L2 learner language.

## introduction in the past couple of years we have seen a fast development in the capabilities of decoder-only language models, such as @xcite , llama @xcite , and bloom. 1 these models have been increasingly deployed in a wide variety of applications such as machine translation @xcite and financial @xcite and legal applications @xcite . in the context of second language (l2) educational applications, these models have been deployed to different subtasks, with varying degrees of success @xcite . even though they excel at a multitude of nlp tasks, the inner workings of large language models are obscure. this means that it is complicated, if not impossible, to verify that a model has learned actual linguistic features instead of making spurious correlations. the same issue is true when attempting to determine how the model arrived at specific decisions @xcite . this can be tricky, especially in high-stakes situations such as educational applications. one such example is the evaluation and assessment of second language performance, as the result of such assessment can alter the life opportunities a person has access to (education, job offers, among others). when dealing with text, we want to be able to understand how systems interact with text from second language learners. this would allow us to properly build models for tasks such as second language assessment, for grammatical error correction, among others, while complying with the right to an explanation @xcite . one way to do so is to analyse how much text diverges from what a language model expects. this can be done using perplexity, a statistical concept that measures the probability of a sequence given an estimator and has been interpreted in an intuitive manner as a measure of "surprisal" @xcite ). however, it has been mostly used to study texts of first language speakers of english. to address this gap, we aim to study how perplexity interacts with texts generated by second language learners of swedish. we hypothesise that the perplexity of a decoderonly model is related to the complexity of the text in an l2 speaker's essay. in this paper we aim to answer the following research questions: we have used gpt-sw3 @xcite ) as our language model for this study. it is based on the gpt series @xcite and trained on data of several nordic languages. we give more details about it in section 3.1. for the l2 learner essays we used swell @xcite , a collection of corpora of l2 learner essays in swedish. a more in-depth explanation of its contents and how the essays were collected can be found in section 3.2. we describe perplexity and some of its intuitive interpretations in section 3.3. we ran two sets of experiments. in section 4 we show the perplexity of the essays and see how it is distributed both across levels and within the essays. this analysis is done both in a statistical manner in section 4.1 and in a linguistic manner in section 4.2. the second set of experiments is a comparison between the perplexity from original essays written by l2 students and normalised versions of these essays, described in section 5. ## experiment 1: perplexity and cefr levels in this section we analyse the perplexities of the essays given by gpt-sw3. we begin by doing statistical analyses of the perplexities by level in section 4.1. we then do a linguistic analysis of some of the essays of each level in section 4.2 with the aim of identifying patterns in how the perplexities are distributed within the essay texts. ## experiment 2: perplexity and text normalisation in this section we analyse whether the perplexity of an essay given by gtp-sw3 is reduced when dealing with a normalised version of it. the idea is to establish whether non-standard language correlates with perplexity and to what degree. when looking at the perplexities in the swellgold dataset in figure 3 and when comparing the original and the normalised versions of the essays, we see two noticeable tendencies. the first is that, even though the original essays seem to have different distributions depending on their level, the normalised ones have pretty much the same distribution regardless of it, as seen in tables 4 and 5 . the other tendency is that the perplexity between the original and the normalised versions of the essays go down in all of them. even though we have suspected this when first looking at figure 3 , the fact that there is an overlap in the boxplots should not be ignored. however, this is confirmed when looking at the figures in the remaining spikes in perplexity in normalized essays indicate use of rare words, potentially register switches, citation marks, among others. ## conclusions one of the issues with large language models tends to be their lack of interpretability and explainability. this keeps true with generative models such as those based on the gpt architecture despite them being able to generate text "justifying" their reasoning @xcite . in this work we aimed to explore the relationship between the perplexity from a decoder-only model of swedish and the complexity of the text of an l2 speaker's essays. we found that there is an inverse relationship between the cefr level of an essay and its perplexity. however, due to the overlap between the values of each level means that they are not a strong indica-figure 3 : boxplots for the perplexities of the different proficiency levels. even though there does not seem to be an obvious pattern between the levels of the texts and their perplexities, the normalised texts show a much lower perplexity than the original texts. moreover, the distributions of the normalised texts are much more similar to each other than to their original versions. we also found that there are perplexity effects through the essays that are not exclusive to l2 language, such as placement within a text, punctuation, frequency of the tokens, among others. despite that, some of the more prevalent effects are characteristic of l2 language, such as errors, non-idiomatic use of the language, and multi-word expressions. there is a correlation between the use of nonstandard language as an l2 language learner. this conclusion can be drawn by the fact that the perplexity for every essay became lower after being normalised. one of the possible applications of these could be done through the use of these features to help guide human graders with the assessment of learner language. the idea of this is to take either a humanin-the-loop @xcite or a hybrid intelligence approach to evaluation @xcite . however, it would be of essence to disentangle the perplexity effects that are specific to l2 speakers from those effects that are not. this would allow us to have a more reliable and fair estimation. this, however, remains to be explored in the future. cefr are categorical classes used to describe language proficiency for teaching and assessment convenience. despite that, language development itself works as a continuum, where essays within each particular level are not homogeneous with regards to their linguistic complexity. this continuum of linguistic complexity of learner language has rather vague and arbitrary cut-off points between one level and another @xcite @xcite . given the context of our experiments, we hypothesise that the perplexity score per essay can help place each essay on a scale between one level and another and that it may be an indirect way of grading essays within the same level. however, this is a hypothesis that needs to be explored in another paper.