Prompting open-source and commercial language models for grammatical error correction of E nglish learner text (2024)

Abstract: Thanks to recent advances in generative AI, we are able to prompt large language models (LLMs) to produce texts which are fluent and grammatical. In addition, it has been shown that we can elicit attempts at grammatical error correction (GEC) from LLMs when prompted with ungrammatical input sentences. We evaluate how well LLMs can perform at GEC by measuring their performance on established benchmark datasets. We go beyond previous studies, which only examined GPT* models on a selection of English GEC datasets, by evaluating seven open-source and three commercial LLMs on four established GEC benchmarks. We investigate model performance and report results against individual error types. Our results indicate that LLMs do not always outperform supervised English GEC models except in specific contexts – namely commercial LLMs on benchmarks annotated with fluency corrections as opposed to minimal edits. We find that several open-source models outperform commercial ones on minimal edit benchmarks, and that in some settings zero-shot prompting is just as competitive as few-shot prompting.

## introduction grammatical error correction (gec) of second language learner english text is an important task in educational ai. its main applications include: i) enabling learners to receive instant feedback on their written work, ii) providing features for automarking, and iii) profiling learners' grammatical knowledge in such a way as to facilitate personalised learning @xcite @xcite . there is a long history of gec research in the field of computational linguistics, developing from rule-based to statistical approaches to neural network models, as has happened with other tasks in natural language processing @xcite . given the recent emergence of large language models (llms), such as openai's gpt * and meta's llama llms, it is natural to ask how well they can perform at gec and how they compare to existing state-of-the-art supervised approaches @xcite ). to answer this question, we aim to elicit minimal edit style corrections from llms through zeroshot and few-shot prompting. minimal edit correction of text aims to resolve any grammatical errors in a text while staying as close as possible to the phrasing and lexical choices of the original. this is sometimes held up as a distinct alternative to fluency correction -where texts are rewritten for naturalness -though in reality the two annotation methods are not completely separable from each other @xcite . the tendency so far has been to annotate gec datasets with minimal edits only, and so that is the way past systems have been trained. this is an important point to note, as llms by default will output a transformative fluency correction of ungrammatical text @xcite @xcite . in this paper, we attempt to prompt llms to perform minimal edit correction rather than fluency correction, so that the outputs are comparable to previous systems. minimal edit corrections are additionally valuable within an educational setting, for example displaying grammatical errors to a learner @xcite . we evaluate three commercial and seven open source llms on four publicly available gec benchmarks: conll 2014 @xcite , the fce corpus @xcite , jfleg @xcite , and write&improve + loc-ness (w&i) from the bea-2019 shared task @xcite . this builds on previous work, which involved gpt * models only and at most three gec benchmarks @xcite @xcite . we find that some of our chosen open-source models outperform gpt-3.5 turbo on english gec benchmarks which have been annotated in a minimal edit fashion. in contrast, gpt-3.5 performs best on a test set annotated with fluency corrections. we evaluate several zero-shot and few-shot prompts, but find that different models require different styles of prompting. some models appear to respond better to few-shot prompting than others, and certain prompt templates work best with a specific llm and dataset rather than universally across the board. we provide strong empirical evidence that llms do not always outperform existing state-of-the-art, supervised gec models -though the search space over llms, prompt templates, and few-shot learning is so great that our results can only be considered a building block in the full picture of llm evaluation on english gec. our investigations can serve as a comparison point for future work on gec with llms. we make our code, prompt templates, few-shot examples, and model predictions publicly available. foot_0 2 related work grammatical error correction (gec). gec is the task of returning an edited version of an input text such that any errors are corrected. it is a longstanding task in research on nlp for educational applications. thanks to large-scale annotation projects and the public release of labelled data, gec systems can be built aiming at general correction of all error types. machine translation gec systems pioneered this general purpose approach, at first with statistical and later neural models @xcite @xcite . more recently, edit-based approaches have been proposed in which corrections are applied on a sequence labelling @xcite or sequence-to-sequence basis @xcite . @xcite offers a comprehensive survey of the history and current state of gec. gec with large language models (llms). the term 'large language model' is currently used to refer to a variety of neural networks developed by a number of organisations and businesses. these models reached the mainstream media through gpt-3 and chatgpt, and as a result there is now a widespread awareness of 'generative ai' -in particular relating to text generation -amongst the general public. openai's gpt * models feature in this paper, alongside others for comparison. we have selected ten open-source and proprietary llms for reasons described in section 4. recent studies indicate that llms from open-ai can be prompted to generate corrected versions of ungrammatical inputs. @xcite compares chatgpt to grammarly and gector @xcite on a sample of 100 sentences from the conll-14 test set. @xcite compares gpt-3.5 and gpt-4 foot_1 to two gec systems on english benchmarks @xcite , whilst @xcite compare chatgpt with multiple baselines including taggec @xcite and t5 @xcite . finally @xcite compare chatgpt and gpt-3.5 (text-davinci-003) with models trained on synthetic data @xcite . @xcite ; @xcite ; @xcite all perform evaluation of english gec on some combination of the jfleg, conll-14 and w&i+locness test sets. they find that the gpt * models set new state-of-the-art (sota) performance on the jfleg dataset in which the annotators were permitted to carry out naturalistic fluency rewrites @xcite . however, they perform worse than sota on conll-14 and w&i+locness, which are much larger, more popular datasets that were annotated on the basis of minimal edit corrections. @xcite and @xcite found through further investigation that the gpt * models have a tendency to over-correct and make extraneous fluency edits. this explains why it is that they can score so highly on jfleg but not on minimal edit data. @xcite meanwhile investigated the possibility of prompting for minimal edits rather than fluency rewrites, and obtained promising improvements which motivate further work. 2023 ) additionally carried out human evaluation to rate the output from each system and found a preference amongst human raters for the gpt * outputs because they were considered to be more fluent. they also found instances of under-correction in the reference sentences derived from human annotators: in other words the llms were able to catch and correct errors which had not been corrected by the original annotators. these human evaluations are tentative only, since they involve only small samples of 100 sentences at a time from each test set. while more fluent corrections may be preferred by human evaluators, they may not aid language learners if they diverge too greatly from the original text. existing annotation guidelines for error correction state that edits should be as minimal as possible so that the learner can be helped to express what they are trying to say, rather than told how to express it differently (which may otherwise discourage them); i.e. how to amend an error rather than avoid it @xcite . consequently, although both minimal and fluent corrections may be valuable to different user groups, we focus on minimal corrections for educational applications in this paper. ## datasets we compare model performance on four publicly available and well-known english language gec datasets: conll-14 test @xcite , jfleg dev and test @xcite , fce dev and test @xcite , and w&i+locness dev 3 @xcite . we additionally sample 2,000 sentences uniformly from fce train and w&i+locness train to con3 the test set for w&i+locness is not publicly available. struct a development set in order to filter the set of prompt templates. the conll-14 test set contains 50 essays written by undergraduate students at the national university of singapore on one of two topics. it has featured in multiple gec studies, and new sota performance was reported by @xcite in a recent paper describing decoding interventions. jfleg dev and test contain approximately 1.5k sentences randomly sampled from essays by learners of english of unknown proficiency levels, and corrected by crowdworkers. annotators were permitted to make fluency corrections to the sentences: not just minimal edits for grammaticality. @xcite achieved the current sota performance on jfleg for a single system with their guided approach to synthetic generation of training data based on error type distributions found in annotated corpora. fce dev and test feature essays written by intermediate learners of english (cefr levels b1 and b2). it is a subset of the cambridge learner corpus and has also been used in multiple gec studies. current sota was established by @xcite with a multi-encoder model which encodes a given sentence and the preceding one separately, integrating them in the decoder. w&i+locness is a hybrid dataset made up of native speaker essays written by undergraduate students (locness; @xcite ) and essays submitted to the write&improve learning platform by learners of english at varying levels of proficiency (w&i). it was prepared for the bea 2019 shared task on gec @xcite , and sota was achieved by @xcite with system combination across multiple gec models. each dataset was processed with errant @xcite , an automatic error annotation tool, in order to be standardised into a common format. consequently the datasets are in tokenised m2 format, and we first need to detokenise them as llms expect untokenised inputs. to carry out this task, we use the moses detokeniser 4 and a rule-based heuristic to combine negative contractions which are not fully handled by the detokeniser 5 . name prompt min make minimal changes to the following text such that it is grammatically correct. {text} elt † you are an english language teacher. a student has sent you the following text. \n{text}\nprovide a grammatical correction for the text, making only necessary changes. do not provide any additional comments or explanations. if the input text is already correct, return it unchanged. tool * † you are a grammatical error correction tool. your task is to correct the grammaticality and spelling in the input sentence. make the smallest possible change in order to make the sentence grammatically correct. change as few words as possible. do not rephrase parts of the sentence that are already grammatical. do not change the meaning of the sentence by adding or removing information. if the sentence is already grammatically correct, you should output the original sentence without changing anything. \n\ninput sentence: {text}\noutput sentence: dn please correct the following text. do not attempt to rewrite it into perfect english or to interpret the text. often, things could be expressed better by paraphrase, but the task is to make minimal changes to correct the text. do not change anything that is correct. please make no changes if there are no errors. ## models we evaluate three commercial and seven opensource llms. we include more open-source than commercial models as we assume that the latter will have a performance advantage and wish to investigate whether open-source models can perform in comparable ways. if so, this would be positive news from an open-science perspective. for the commercial llms, we include openai's gpt-3.5-turbo and gpt-4 models (openai, 2023), and cohere's command model. 6 many more are available but due to budget constraints we work only with these three and we only evaluate gpt-4 in the zero-shot setting. we choose not to work with chatgpt as it has been engineered to function as a chatbot. for the open-source models, we select instruction-tuned models because the majority of our prompt templates contain instructions, and we evaluate the largest model from each model type that fits on a server with two a100 80gb nvidia gpus. our upper bound on model size relates to the computing resources available to us at the time of writing. the open-source models are: opt-iml-max-30b @xcite , llama-2-70b-chat @xcite , stable beluga 2 @xcite , falcon-40b-instruct @xcite , flan-t5-xxl @xcite , bloomz-7b1 @xcite , instructpalmyra-20b @xcite . this is a representative sample of the models available, with a range of sizes and architectures. approximate model sizes are given in ## results contrary to expectations set by previous work, adding few-shot examples to the three zero-shot prompts does not always lead to an improvement in performance. indeed for fce dev, zero-shot prompts perform best for most models. the picture is mixed for jfleg dev, whilst the majority of models benefit from few-shot learning for w&i dev. it remains a matter for future work to investigate whether more dynamic approaches to data sampling (as opposed to a fixed selection of examples) will aid with few-shot gec prompting. our llm results are well short of sota performance, established by task-specific supervised models, for fce test and conll-14 test -the corpora annotated in minimal edit fashion -whereas the performance of gpt 3.5 turbo is much closer to the sota on jfleg test. these findings reinforce initial experiments by @xcite , ## discussion we set out to investigate the performance levels of llms on the task of english gec. previous work has shown that gpt * models could perform gec with mixed success: outdoing existing sota models on the jfleg dataset, which contains fluency corrections, whilst performing poorly on benchmarks annotated with minimal edit correctionsnamely conll-14, the fce and w&i+locness @xcite @xcite . we aimed to elicit minimal edit corrections through exploration of different prompting strategies, and evaluated models other than gpt * -including more open-source than commercial llms. our findings echo those in previous papers: our chosen llms perform well on jfleg test -above all falcon-40b-instruct, stable beluga 2 and gpt-3.5 -though not outdoing sota, possibly because our prompts were designed to discourage fluency style corrections. based on experiments with jfleg dev, gpt-4 might perform best on jfleg test, but full investigation of this question requires additional funding as gpt-4 is currently an order of magnitude more expensive than gpt-3.5 turbo. in contrast, the llms perform poorly on the fce and conll-14 test sets, lagging far behind sota in both cases. for these datasets, open-source models outperform or compete with the commercial models: the best performing model is flan-t5 on the fce, and stable beluga 2 matches gpt 3.5 turbo in the case of conll-14. again, performance on the fce and w&i+locness dev sets suggests that gpt-4 could outperform the other llms on the test sets. we narrowed down our initial 11 zero-shot prompts to the 7 which performed best on a sample of sentences from the fce and w&i+locness training sets. we created few-shot prompts from the 3 best performing zero-shot prompts and varied the number of examples from 1 to 4. the re-sults for zero-shot versus few-shot learning do not clearly show a best method for prompting. the open-source models which perform best on the test sets are flan-t5, falcon-40b-instruct, and stable beluga 2: of these, flan-t5 and falcon-40b-instruct work best with few-shot learning, whereas stable beluga 2 is best with a zero-shot prompt. for gpt-3.5 turbo, zero-shot is best for fce test, few-shot is best for jfleg and conll-14 test. in terms of prompt wording, the tool and cyn prompts are best for the three best open-source models: flan-t5, falcon-40b-instruct, and stable beluga 2. for gpt-3.5 turbo, the elt prompt is best for the fce test set and the tool one is best for jfleg and conll-14 test. note that two of the three best performing prompts are those in which a role is clearly specified to the llm -either as an english language teacher or a grammatical error correction tool ( the fact that the other best performing prompt, the one from @xcite , replicates the strong results from that paper is evidence for convergence around optimal prompt crafting. further exploration of the huge prompt search space is possible, but we show that the cyn prompt holds up another provision we make for replication in future studies is to supply the list of examples we used in few-shot learning ( finally, we note that the comparison between commercial and open-source llms is not entirely even, as the former sit behind apis and a black box processing pipeline. we recognise that gpt-3.5 turbo shows great promise for english gec, at least for fluency corrections, but we also find that several open-source models perform relatively well -in fact better than gpt-3.5 on benchmarks annotated with minimal edits. this is a boon for open science, because models which researchers can obtain and work with directly lead to greater transparency in gec and beyond. ## conclusion we have shown that llms do not always outperform existing sota models for english gec: for minimal edit style datasets such as the fce, conll-14 and w&i+locness, their performance is far below that of supervised gec systems. we attempted to elicit minimal edit corrections from llms through prompt crafting, but it may be that llms are still biased towards fluency rewrites as has been shown in previous work @xcite @xcite . this is consistent with our finding, echoing that of others, that llms perform best on jfleg, which was annotated with a fluency correction style. we arrive at the following conclusions: (i) supervised models are still best for english gec with minimal edit corrections; (ii) further explorations of prompt crafting, few-shot learning, and dynamic sampling are justified, as is work with open-source models as opposed to commercial ones; (iii) methods for improving llm performance on specific error types could be explored. other potential areas for future work include document-level gec and human evaluation of proposed corrections. we worked with sentence-level gec, but this deviates from the greater amount of essay context given to annotators. document-level gec has been proposed and recommended in previous work @xcite . exploratory work by @xcite showed that chatgpt could not perform documentlevel gec well, and speculated that it may not be able to handle long inputs requiring "high levels of coherence and consistency between sentences". we notice that llms are better at gec on beginner and intermediate texts, rather than advanced or native-speaker ones (appendix initial human evaluation studies suggest a preference for the corrections generated by llms over the reference corrections contained in gec corpora @xcite . it may be that human raters prefer to read the more fluent llm-derived corrections but minimal edit corrections are actually more helpful for language learning since they are more faithful to the original intended meaning of the writer. investigating learning benefits from receiving minimal edit grammatical feedback as opposed to fluency rewrites is a matter for future work which will involve longitudinal data collection, a focus on different feedback styles, and tracking how learners respond. ery (csd3) operated by the university of cambridge research computing service, provided by dell emc and intel using tier-2 funding from the engineering and physical sciences research council (capital grant ep/t022159/1), and dirac funding from the science and technology facilities council. we acknowledge the cohere for ai research grant that supported our evaluation of the cohere command model. we thank yulong lin for his exploratory work on gec with llms.