Predicting Initial Essay Quality Scores to Increase the Efficiency of Comparative Judgment Assessments (2024)

Abstract: Comparative judgment (CJ) is a method that can be used to assess the writing quality of student essays based on repeated pairwise comparisons by multiple assessors. Although the assessment method is known to have high validity and reliability, it can be particularly inefficient, as assessors must make many judgments before the scores become reliable. Prior research has investigated methods to improve the efficiency of CJ, yet these methods introduce additional challenges, notably stemming from the initial lack of information at the start of the assessment, which is known as a cold-start problem. This paper reports on a study in which we predict the initial quality scores of essays to establish a warm start for CJ. To achieve this, we construct informative prior distributions for the quality scores based on the predicted initial quality scores. Through simulation studies, we demonstrate that our approach increases the efficiency of CJ: On average, assessors need to make 30% fewer judgments for each essay to reach an overall reliability level of 0.70.

## introduction the comparative judgment (cj) method is utilized in diverse educational assessments, and specifically, some educational institutions employ it for the assessment of student essays. as shown in figure 1 , this approach involves presenting two essays in a web-based tool, where assessors compare them to determine the best one. after a sufficient number of judgments, all pairwise comparisons are used to calculate a quality score for each essay. in contrast to rubric marking, cj provides distinctive advantages. assessors can apply their expertise and experience flexibly, without strict adherence to rubrics @xcite . additionally, cj enhances the reliability and validity of scores by incorporating multiple judgments from various assessors @xcite . despite the advantages of cj, it still requires many judgments from assessors before quality scores become reliable enough, typically requiring between 10 and 14 judgments per essay to achieve a reliability level of 0.70 @xcite , rendering the assessment method rather inefficient @xcite . a cause of its inefficiency is that, at the start of the assessment, there is no information about the quality scores, as no judgments have been made yet. in adaptive learning systems, this problem is commonly referred to as cold-start problem @xcite . a solution to alleviating this cold-start problem, and subsequently increasing the efficiency of cj, would be to introduce a 'warm start' in the assessment by automatically predicting initial quality scores for essays. although the prediction of essay quality has already been extensively explored in automated essay scoring (aes) (see a review by @xcite , these studies have mostly focused on what could be defined as non-comparative, or absolute @xcite , essay scoring, where each essay is scored as a standalone piece without comparison to other essays. to the best of our knowledge, there have been few to no studies that explored the automatic prediction of essay quality scores obtained through cj assessments. to address this research gap, we studied the extent to which essay quality scores, resulting from a cj assessment, can be automatically predicted and used to alleviate the cold start of cj with the goal of increasing the efficiency of cj for assessing essay quality. we focused on dutch essays written for argumentative assignments. firstly, we conducted a machine learning experiment in which deep learning models were trained on data collected from cj assessments to predict quality scores of essays. secondly, we ran simulations where we used the predicted quality scores as initial quality scores to alleviate the cold start of cj. these steps were conducted to answer the following research questions: ## discussion our findings underscore the ability of current deep learning models, particularly transformer models, to predict initial quality scores that provide valuable information on the argumentative writing quality of essays. furthermore, incorporating the assignment prompts for fine-tuning enhances the re-liability of predicted quality scores, which aligns with prior research in aes @xcite @xcite . we posit that prompt information is especially important for the prediction of initial quality score, since, in this study, the essays in the training set were written for different assignments than the essays in the test set. when warm-starting cj assessments with these predicted initial quality scores, the necessary number of comparative judgments to obtain reliable quality scores decreases significantly. this suggests that less effort from assessors is required while upholding high levels of reliability of the quality scores. furthermore, our approach to increase the efficiency of cj avoids any undesirable effects with respect to the reliability measures, which have been noted when employing an adaptive selection rule @xcite @xcite . additionally, our method demonstrates a more substantial improvement in efficiency compared to the approach of @xcite , who devised a more efficient selection rule based on similarities in essay texts. ## conclusion we successfully improved the efficiency of cj assessments by introducing a warm start for the estimation of the quality scores. this involved predicting essay quality scores, which were then used to form informative prior distributions within a bayesian btl model. through an extensive simulation study, we demonstrated that our approach led to a reduction, ranging between 15% and 41%, in the number of comparative judgments needed to reach a reliability of 0.70 and produced more accurate rankings of essays at the start of an assessment. furthermore, our findings indicate that these efficiency gains can be measured in practical settings, as the ssr θ approximates the reliability well.