Assessing Essay Fluency with Large Language Models (2024)

Abstract: “With the development of education and the widespread use of the internet, the scale of essay evaluation has increased, making the cost and efficiency of manual grading a significant challenge. To address this, The Twenty-third China National Conference on Computational Linguistics (CCL2024) established evaluation contest for essay fluency. This competition has three tracks corresponding to three sub-tasks. This paper conducts a detailed analysis of different tasks,employing the BERT model as well as the latest popular large language models Qwen to address these sub-tasks. As a result, our overall scores for the three tasks reached 37.26, 42.48, and 47.64.”

## introduction ccl points out that with the development of education, the quantity of essay texts is gradually increasing. the cost and efficiency of manually grading essays have become a major challenge. many research institutions aim to provide objective, accurate, and timely scoring and feedback by analyzing the language, content, and structure of essays. among these factors, the fluency of expression is an important aspect of essay evaluation for teachers. therefore, this competition revolves around evaluating the fluency of essay texts. there are three tracks in this competitionincluding: track 1: identification of sentence types in primary and secondary school essays. the identification of incorrect sentence types in primary and secondary school essays is a multi label classification problem, which predicts which types of incorrect sentences a sentence is. the sick sentence type label contains both lexical and syntactic errors. in this evaluation task, a total of 5 coarse-grained error types and 14 fine-grained error types were defined. track 2: revision of sentence types in primary and secondary school essays. the task of rewriting incorrect sentences in primary and secondary school essays is a text generation task, which involves inputting incorrect sentences and outputting modified ones. track 3: evaluation of fluency in primary and secondary school essays. the fluency rating task for primary and secondary school essays is a multi classification task, which involves inputting an essay and outputting its level of fluency. this evaluation task defines three fluency levels: excellent, average, and failing. three tracks provide more basis for evaluating the fluency of primary and secondary school essays and offering higher-quality assessments. ## experiments during the experiment, we used the transformer library to load the qwen-7b and bert-large-chinese pre-training model, trained the model with the datasets announced by the competition organizer, and optimized the model parameters through backpropagation and gradient descent algorithms. the hyperparameter settings during model training are shown in table1 and table2. ## conclusion in this experiment, we conducted a comprehensive assessment of the fluency of essays and validated it through a competition format. the results indicate that large language models perform exceptionally well in text correction for evaluating essay fluency, while bert models are more effective in essay fluency classification. in future research, we will further explore evaluation methods for essay fluency, integrating theories from linguistics, psychology, and related fields, in order to seek more scientifically effective evaluation metrics and methods.