Rating Short L 2 Essays on the CEFR Scale with GPT -4 (2023)

Abstract: Essay scoring is a critical task used to evaluate second-language (L2) writing proficiency on high-stakes language assessments. While automated scoring approaches are mature and have been around for decades, human scoring is still considered the gold standard, despite its high costs and well-known issues such as human rater fatigue and bias. The recent introduction of large language models (LLMs) brings new opportunities for automated scoring. In this paper, we evaluate how well GPT-3.5 and GPT-4 can rate short essay responses written by L2 English learners on a high-stakes language assessment, computing inter-rater agreement with human ratings. Results show that when calibration examples are provided, GPT-4 can perform almost as well as modern Automatic Writing Evaluation (AWE) methods, but agreement with human ratings can vary depending on the test-taker’s first language (L1).

## introduction automated writing evaluation (awe) systems are commonly used to evaluate test-taker writing. awe systems are deployed on large-scale, highstakes writing assessments used for admissions to higher education institutions, and for lower-stakes us state writing assessments that provide information about k-12 students' academic writing performance. these systems typically use featureengineering approaches that include rule-based and statistical natural language processing (nlp) methods. nlp is used to extract features from essay writing responses that are characteristic of writing quality. features may include errors in grammar and spelling, discourse structure, discourse coherence, vocabulary usage, and sentence variety. features may be rule-based or statistically derived. statistical model methods, such as straightforward linear regression, are used to train (build) awe scoring models for high-stakes scoring of writing assessments. detailed descriptions of systems are avail-able for major systems, including e-rater®, intelligent essay assessor™, intellimetric®, and peg (shermis and @xcite , and cambium's automated essay scoring system @xcite . recent advances in language modeling with neural transformer architectures @xcite have the potential to revolutionize awe. these large language models (llms) demonstrate an incredible potential to analyze and evaluate text which has implications for the future of awe. in addition, gpt's intuitive, text-based interface lowers barriers for use, potentially increasing accessibility and adoption of these tools for awe. the assumptions about how llms -specifically gpt-4 -can be used for awe tasks, such as automated scoring and feedback need to be evaluated to determine how we can use them beneficially, and particularly to ensure that they can be used in a fair and ethical manner @xcite . previous research evaluated gpt-3.5 for essay scoring tasks in an l2 context @xcite . in this paper, we evaluate gpt-4 for a similar task, comparing it to gpt-3.5, human judgement, and a strong baseline using current awe methods. we also explore various aspects that affect the accuracy of gpt's ratings, and its fairness across gender and l1. ## data for our experiments, we used a human-rated dataset consisting of short essay responses collected as part of the duolingo english test, a highstakes test of english for l2 learners. for this essay task, test-takers are given a short written prompt randomly selected from an item bank of about 700 items. test-takers have 5 minutes to provide their essay response to the prompt. two human raters used a scoring rubric aligned with the common european framework of reference (cefr) @xcite . we started by sampling 10,000 responses from test sessions that took place over a 10-month period, controlling for l1 and gender. for l1, we limited responses to 7 of the most common l1 languages for the test, which also captures a broad range of language families: arabic (ara), mandarin chinese (cmn), telugu (tel), english (eng) foot_0 , spanish (spa), gujarati (guj), and bengali (ben). to ensure all cefr levels were well represented in the final dataset foot_1 , we used a simple cefr classifier that uses logistic regression and nlp features to roughly estimate the cefr level of each response. for the final dataset, we randomly sampled an equal number of responses for each combination of l1, gender, and estimated cefr level from the 10,000 test sessions. the scoring rubric was aligned to the cefr scale and assessed each response based on its content, coherence, vocabulary, and grammar. the rubric instructed raters to assign each essay one of eight rating categories: six based on the cefr scale, and two "unscorable" categories for minimal responses (e.g., provides no response or says they can't answer the question) and bad-faith responses (e.g., off-topic or nonsensical). the full rubric is provided in appendix b. based on this rubric, two assessment researchers developed a set of calibration examples by collectively rating 676 essays, 180 of which were rated by both. the rubric and calibration examples were provided to two new human raters, who collectively rated 1,961 new essays, including a random sub-sample of 389 essays that were rated by both. both new human raters were trained by one of the original assessment researchers and inter-rater agreement was routinely checked. raters were provided feedback to help with calibration when necessary. the final quadratic weighted kappa (qwk) between the two raters was 0.87. ratings were roughly normally distributed (see figure 1 ), with ∼53 % of essays receiving a rating of b1 or b2 and only ∼12 % getting a rating of a1 or c2. ## experiments we conducted three experiments. the first evaluates both gpt-3.5 and gpt-4 with a minimal rubric and up to the maximum number of calibration examples that fit within the gpt model's token limit. the second experiment evaluates various prompt engineering strategies for improving performance. the third experiment explores gpt-4's fairness properties across gender and l1. ## conclusion we showed that unlike gpt-3.5, gpt-4 is able to attain performance similar to conventional automated writing evaluation (awe) models when rating short l2 essays. gpt-4 only required one calibration example per rating category to achieve near optimal performance, but other prompt engineering techniques we tried were not very helpful. furthermore, when assessing fairness with respect to the test-taker's gender or l1, we found that while gpt-4 did not show bias in favor of any one group, it showed significantly less agreement with human ratings for some l1s. it is unclear whether this is due to the reliability of gpt-4 or that of the human ratings themselves. more research is needed to understand this discrepancy and its implications for fairness. future research may also explore other prompt engineering strategies for improving gpt-4's performance at this task, or potentially finetuning gpt-3.5, enabling one to leverage dramatically more training data than what can be provided in a prompt. perhaps most excitingly, future work may explore gpt-4's potential for providing feedback aligned to essay scoring: a task for which gpt-4 seems particularly well suited.