Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction (2024)

Abstract: Large Language Models (LLMs) have been reported to outperform existing automatic evaluation metrics in some tasks, such as text summarization and machine translation. However, there has been a lack of research on LLMs as evaluators in grammatical error correction (GEC). In this study, we investigate the performance of LLMs in GEC evaluation by employing prompts designed to incorporate various evaluation criteria inspired by previous research. Our extensive experimental results demonstrate that GPT-4 achieved Kendall’s rank correlation of 0.662 with human judgments, surpassing all existing methods. Furthermore, in recent GEC evaluations, we have underscored the significance of the LLMs scale and particularly emphasized the importance of fluency among evaluation criteria.

## introduction large language models (llms) have surpassed existing systems in various nlp tasks, showcasing their high capabilities of language understanding and generation @xcite . these llms, which have had a significant impact on recent nlp research, also demonstrate the ability to produce high-quality corrections in grammatical error correction (gec) @xcite @xcite @xcite . in recent years, several studies have been conducted on the use of llms as an evaluator. in text summarization, dialogue generation, and machine translation, gpt-4 has demonstrated superior performance compared to existing automatic evaluation metrics @xcite . while there is very little research on gec evaluation, considering gpt-4's ability to explain grammatical errors with 90% accuracy in human evaluations @xcite , it holds potential for evaluating corrections. sottana et al. (2023) conducted meta-evaluation using a limited number of systems, but there has been no comprehensive analysis using dozens of systems like traditional approaches such as @xcite and @xcite . therefore, we aim to explore the extent to which llms operate as evaluation models in english gec. specifically, we conduct gec evaluations using llms with prompts at different evaluation granularities to investigate how evaluation capabilities change with the presence of evaluation criteria and the scale of llms, as shown in figure 1 . @xcite 's work on the evaluation of metrics (i.e., meta-evaluation) has revealed that conventional metrics lack the resolution to capture performance differences in high-performing gec systems. given this current state, to facilitate proper gec evaluation moving forward, we investigate the potential of llms by comparing them with conventional metrics through meta-evaluation. our contributions are summarized as follows. (1) we conducted a comprehensive investigation into the performance of llms as evaluators in gec, and the results showed that gpt-4 achieved stateof-the-art performance, indicating the usefulness of considering evaluation criteria in prompts (especially fluency). (2) it was suggested that as llm scales decrease, the correlation with human evaluations decreases, and the ability to capture fluency in corrected sentences diminishes. smaller llms tend to avoid extreme scores, while larger llms tend to assign higher scores. ## experiment setup in this section, we explain the considered gec metrics ( §2.1) and meta-evaluation methods ( §2.2). ## results in this section, we analyze the performance of llms as gec evaluators in system-level ( §3.1) and sentence-level meta-evaluations ( §3.2). additionally, we conduct further analysis by changing the system set to investigate the impact of the considered systems in the meta-evaluation( §3.3). ## related work several studies have investigated the evaluation performance of llms. chiang and lee (2023) conducted the first investigation into llm evaluation performance, demonstrating that gpt-3.5 can achieve expert-level evaluation in tasks such as open-ended story generation and adversarial attacks. in the summarization task, @xcite revealed that gpt-4 has state-of-the-art evaluation performance by leveraging their proposed methods like auto-cot (chain-of-thought) and weighted scores. in the machine translation task, @xcite demonstrated that only larger models exceeding gpt-3.5 can perform translation quality evaluation, with gpt-4 slightly inferior to existing metrics at the segment level. @xcite utilized llms to evaluate second language writing proficiency through essay grading, discovering that gpt-4 exhibits performance equivalent to modern automated writing evaluation methods. ## conclusion in this work, we investigated the capability of llms as evaluators in english gec, and gpt-4 demonstrated significantly higher correlations compared to traditional metrics. future work should delve into the impact of few-shot learning and optimize prompt engineering for enhanced evaluation performance. furthermore, we plan to explore the possibility of document-level evaluation, considering the expansion of the gpt's context window, which is not currently focused on by existing metrics.