Are the Reasoning Models Good at Automated Essay Scoring? (2025)

Abstract: This study investigates the validity and reliability of reasoning models, specifically OpenAI’s o3-mini and o4-mini, in automated essay scoring (AES) tasks. We evaluated these models’ performance on the TOEFL11 dataset by measuring agreement with expert ratings (validity) and consistency in repeated evaluations (reliability). Our findings reveal two key results: (1) the validity of reasoning models o3-mini and o4-mini is significantly lower than that of a non-reasoning model GPT-4o mini, and (2) the reliability of reasoning models cannot be considered high, with Intraclass Correlation Coefficients (ICC) of approximately 0.7 compared to GPT-4o mini’s 0.95. These results demonstrate that reasoning models, despite their excellent performance on many benchmarks, do not necessarily perform well on specific tasks such as AES. Additionally, we found that few-shot prompting significantly improves performance for reasoning models, while Chain of Thought (CoT) has less impact.

## introduction the development of large language models (llms) has marked a significant breakthrough in artificial intelligence, showing remarkable progress and versatility across various fields @xcite @xcite . these advances have made substantial impacts in education, where llms are being actively adopted and tested in different learning contexts @xcite @xcite . one of the notable applications in this domain is automated essay scoring (aes). aes represents a well-established research field with over fifty years of continuous development and improvement @xcite @xcite . in recent years, fine-tuned deep neural networks, especially those based on bert architectures, have shown superior performance in this task, setting new standards for automated assessment accuracy. the application of llms in aes has gained significant attention from researchers worldwide @xcite @xcite @xcite @xcite . for instance, a study by @xcite evaluates the validity and reliability of llms for aes in language education, finding gpt-4 exhibited the best performance with excellent intrarater reliability and good validity. meanwhile, recent advancements in llms include reasoning models, which have been enhanced through reinforcement learning and demonstrate superior performance across various benchmarks @xcite @xcite . however, while these models are expected to show improved capabilities in aes, their actual performance in this specific domain remains unclear. we investigate validity and reliability in aes for reasoning models, specifically openai's o3-mini and o4-mini. through evaluation using six prompting strategies combining zero-shot and fewshot prompting with chain of thought (cot), we ensure robust findings. for validity, we measure agreement between expert and model ratings on the toefl11 dataset. for reliability, we evaluate intra-rater consistency through repeated evaluations. our findings demonstrate two key results: (1) reasoning models o3-mini and o4-mini show significantly lower validity than nonreasoning model gpt-4o mini, and (2) reasoning models exhibit moderate rather than excellent reliability. these results reveal that reasoning models, despite excellent performance on many benchmarks, do not necessarily perform well on different tasks such as aes. ## discussion our study found that newer models o3-mini and o4-mini performed significantly worse than gpt-4o mini. this underperformance persisted across all six prompting strategies tested, suggesting this stems from fundamental model characteristics rather than prompting techniques. the low qwk was caused by reasoning models rating expertrated medium essays as high. detailed analysis of reasoning processes in correctly versus incorrectly evaluated essays may help us understand these patterns. in our experiments, the highest qwk of 0.628 achieved by gpt-4o mini is below the state-of-theart score of 0.782 achieved by @xcite , who introduced the dual-scale bert + cnn model for multi-trait aes. although our prompts did not reach state-of-the-art performance with llm-based aes, considerable potential remains for further improvement through fine-tuning, advanced prompt-engineering techniques, or ensemble methods. for reasoning models, few-shot performance was significantly superior to zero-shot performance. while earlier research (deepseek-ai, 2024) and openai's best practices (openai, 2025c) recommend zero-shot over few-shot approaches for reasoning models, our results contradict these recommendations. we found that providing examples improves performance even in reasoning models, suggesting genuine potential for prompt engineering. there were no significant differences in fewshot prompts regarding reasoning provision or placement (sf, srf, rsf), showing cot has less impact than few-shot prompting. since reasoning models perform reasoning internally, they could already use a form of cot, explaining the lack of differences. reliability evaluation revealed reasoning models achieved moderate consistency (icc≈0.7), considerably lower than gpt-4o mini's excellent reliability (icc≈0.95) with a deterministic parameter setting (temperature 0). this 0.25-point icc gap has important implications: reasoning models require 56% more evaluations to achieve equivalent reliability in high-stakes assessments. score variability likely originates from reasoning models generating different reasoning paths for the same essay, causing fluctuations that do not occur in deterministic non-reasoning models. low reliability and limited validity reinforce each other: unstable scores lead to worse agreement with human raters. this suggests unstable scoring in reasoning models contributes to lower validity scores, creating challenges for aes applications. our experiments found that reasoning models o3-mini and o4-mini, despite strong benchmark performance (openai, 2025a; 2025b), showed lower validity and reliability in aes than gpt-4o mini. this demonstrates that benchmark performance doesn't guarantee effectiveness across all tasks. previous research (yoshida, 2024) made similar observations, and our results confirm this pattern applies to reasoning versus non-reasoning models. therefore, evaluating each model on specific tasks is essential rather than assuming universal effectiveness. additionally, reasoning models' improved performance with few-shot approaches shows potential for prompt engineering, representing one contribution of this research. ## conclusion in this study, we investigated the validity and reliability of reasoning models (o3-mini and o4mini) in aes tasks using the toefl11 dataset. through comprehensive evaluation using qwk for validity and icc for reliability, we found that reasoning models demonstrated lower performance compared to the non-reasoning model gpt-4o mini despite their superior performance on general benchmarks. our key findings revealed that (1) reasoning models achieved qwk values of 0.539-0.542, significantly lower than gpt-4o mini's 0.628, and (2) reasoning models showed moderate reliability (icc≈0.7) compared to gpt-4o mini's excellent reliability (icc≈0.95) with a deterministic parameter setting (temperature 0). additionally, we discovered that few-shot prompting significantly improved reasoning model performance, while cot had minimal impact. these findings contribute to our understanding that state-of-the-art models may not universally excel across all tasks, highlighting the critical importance of task-specific evaluation rather than relying solely on general benchmark performance. this research provides essential guidance for practitioners in selecting appropriate models for aes and underscores the need for continued investigation into model performance across diverse applications.