Can GPT -4 do L 2 analytic assessment? (2024)

Abstract: Automated essay scoring (AES) to evaluate second language (L2) proficiency has been a firmly established technology used in educational contexts for decades. Although holistic scoring has seen advancements in AES that match or even exceed human performance, analytic scoring still encounters issues as it inherits flaws and shortcomings from the human scoring process. The recent introduction of large language models presents new opportunities for automating the evaluation of specific aspects of L2 writing proficiency. In this paper, we perform a series of experiments using GPT-4 in a zero-shot fashion on a publicly available dataset annotated with holistic scores based on the Common European Framework of Reference and aim to extract detailed information about their underlying analytic components. We observe significant correlations between the automatically predicted analytic scores and multiple features associated with the individual proficiency components.

## introduction automated essay scoring (aes) of second language (l2) proficiency is a well-established technology in educational settings, involving the automatic scoring and evaluation of learners' written productions through computer programs @xcite . originating in the 1960s, the roots of aes can be traced back to the development of project essay grade (peg) @xcite ), an automatic system which evaluated writing skills based only on proxy traits: hand-written texts had to be manually entered into a computer, and a scoring algorithm then quantified superficial linguistic features, such as essay length, average word length, count of punctuation, count of pronouns and prepositions, etc. across the following decades, as natural language processing (nlp) technologies have advanced and increased their power @xcite , the field of aes has expanded and improved, and more significant studies have been conducted from the 1990s and early 2000s. the most widely known automated scoring systems for essays include the e-rater®, developed by educational testing service @xcite , intellimetric™ by vantage learning @xcite , and the intelligent essay assessor™, built at pearson knowledge technologies @xcite . in recent years, deep neural network (dnn) approaches have brought significant improvements @xcite , and especially the advent of transformer-based architectures @xcite , such as bert @xcite which took the world of nlp and, consequently, aes by storm, outperforming classic feature-based systems @xcite . yet, the most recent breakthrough has been brought by large language models (llms), such as the gpt models @xcite , which might revolutionise the world of aes, not only from the nlp experts' and language testers' perspective, but also considering the users' point of view due to gpt's extremely accessible and intuitive interface. in the context of l2 writing assessment, previous studies have employed @xcite and gpt-4 @xcite , obtaining promising results. although llms have been employed for holistic scoring (i.e., assessing the overall quality of a composition as a whole, considering various aspects such as vocabulary, grammar, coherence, etc. altogether), to the best of our knowledge, so far they have not been investigated for the task of analytic scoring (i.e., breaking down a composition into specific components or criteria and assigning separate scores or ratings to each component). 1 offering l2 learners specific analytic proficiency scores is crucial for delivering insightful and effective feedback, emphasising both their strengths and weaknesses to facilitate improvement. for holistic scoring, previous works have shown that state-of-the-art automatic techniques can reach near-human results @xcite or even outperform them @xcite . this is, at least in part, ascribable to the fact that holistic scores are generally easier to obtain for human evaluators (see section 2). conversely, assessing analytic aspects of language proficiency is generally considered to be more difficult, time-consuming, and cognitively demanding for human evaluators, and, as a result, "noisy" ground truth scores are harder to learn and predict for automatic systems (see section 2). starting from these premises, in this paper, we conduct a series of exploratory experiments on a publicly available dataset annotated with holistic scores according to the common european framework of reference (cefr) @xcite using gpt-4 in a zero-shot fashion, and aim to extract specific information about their underlying analytic components. although ground truth analytic scores are not available, we find significant correlations between the analytic scores predicted by the model and several features related to the analytic scores. when we consider assessment strictly from a human perspective, holistic assessment is considered highly practical as it is more time-efficient per se and in relation to rater training @xcite , less cognitively demanding @xcite , and generally has a higher inter-annotator agreement @xcite than analytic assessment. on the other hand, holistic scoring may suffer from lack of clarity regarding how different aspects are prioritised, which may vary among evaluators @xcite , the risk that evaluators might primarily concentrate on candidates' strengths rather than their weaknesses @xcite , and the potentially erroneous assumption that various aspects of proficiency develop uniformly over time @xcite . analytic assessment allows for a more detailed and systematic evaluation and is supposed to provide much more detailed feedback to l2 learners, by highlighting their fortes and their weaknesses @xcite in addition to enhancing scoring validity. however, it is not a panacea. analytic scores may be psychometrically redundant @xcite due to a halo effect @xcite , whereby raters fail to distinguish between different aspects of learners' performances but assess all or some of them with similar scores. for example, when assessing grammatical accuracy, raters might be influenced by the score previously assigned to vocabulary range. on top of this, raters might confuse analytic criteria in the phase of assessment due to high cognitive load @xcite or, more simply, to indefiniteness of the analytic criteria @xcite . the difficulty in providing analytic scores -especially for a large number of written productions -is evident in the total absence of publicly available l2 english learner datasets annotated in this way 2 and the fact that the primary emphasis in aes research has been on holistic scoring. ## conclusions in this paper, we have conducted an initial study on the use of gpt-4 for assessing 9 individual aspects of l2 writing underlying the cefr proficiency levels in a zero-shot fashion. to do this, we used a holistic grading system on the essays of the w&i validation set and, subsequently, fed them with their respective holistic scores into gpt-4, asking to assess one individual aspect at a time. although the ground truth analytic scores are not available, we have obtained significant correlations between the predicted analytic scores and various features linked to the componential aspects of the cefr levels. beyond its immediate implications for computer-assisted language learning applications, we believe that our exploratory experiments may hold promise as valuable contributions to theoretical studies on construct validity in the broader field of language testing and assessment, given the inclusion of cefr descriptors in our study. in order to collect further evidence to support our findings, we plan to deploy this system, use it in educational settings, and evaluate its effectiveness by monitoring learners' progress in relation to each specific aspect of proficiency. future work will also explore the use of multi-modal systems, such as the one presented in @xcite , for assessing l2 speech in a similar fashion.