Exploring Effectiveness of GPT -3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods (2023)

Abstract: Large-scale pre-trained language models such as GPT-3 have shown remarkable performance across various natural language processing tasks. However, applying prompt-based methods with GPT-3 for Grammatical Error Correction (GEC) tasks and their controllability remains underexplored. Controllability in GEC is crucial for real-world applications, particularly in educational settings, where the ability to tailor feedback according to learner levels and specific error types can significantly enhance the learning process. This paper investigates the performance and controllability of prompt-based methods with GPT-3 for GEC tasks using zero-shot and few-shot setting. We explore the impact of task instructions and examples on GPT-3’s output, focusing on controlling aspects such as minimal edits, fluency edits, and learner levels. Our findings demonstrate that GPT-3 could effectively perform GEC tasks, outperforming existing supervised and unsupervised approaches. We also showed that GPT-3 could achieve controllability when appropriate task instructions and examples are given.

## introduction grammatical error correction (gec) is an essential application of natural language processing (nlp) in educational settings, as it significantly enhances learners' language skills and writing performance @xcite . in real-world applications, controlling specific gec settings, such as minimal and fluency edits and learner level-based corrections, is crucial to address diverse learning needs and scenarios @xcite @xcite . although recent gec approaches based on supervised learning have achieved remarkable progress, they heavily rely on large training datasets comprising both genuine and pseudo data @xcite @xcite @xcite @xcite @xcite @xcite @xcite . collecting such data for each specific setting is challenging and time-consuming, which limits the scalability of these methods in various learning situations. prompt-based methods utilize large-scale pretrained language models (plms), such as gpt-3, and have demonstrated promising results in numerous nlp downstream tasks. these tasks include natural language inference, question answering, and summarization @xcite . given the demand for control in gec tasks across various settings, prompt-based methods are appealing because they deliver exceptional performance without needing extensive labeled data. despite the success of prompt-based methods in multiple nlp tasks, their application to gec remains under-explored. although @xcite and @xcite have recently assessed prompt-based methods on select gec benchmarks, a comprehensive analysis has yet to be conducted. this study aims to bridge this gap by concentrating on in-depth analyses of prompt-based methods and their controllability, aspects that have not been thoroughly investigated in previous research. our research seeks to address the following questions: 1) to what extent can plms using promptbased methods solve gec tasks? and 2) is it possible to control gec settings with prompts written in natural language using prompt-based methods? in this work, we demonstrate that prompt-based methods with gpt-3 @xcite achieve outstanding performance in gec tasks (section 3). in addition, the approach provides better control over the gec process using task instructions and examples (section 5). we conduct analyses to examine the impact of different types of task instructions on gpt-3's performance in both zero-shot and few-shot setting, which emphasizing the importance of appropriate task instructions for gec tasks (section 4.1). additionally, we investigate the effect of varying the number of examples in few-shot setting, and reveal that performance improves as the number of examples increases, albeit not strictly linearly (section 4.2). furthermore, we explore the model's controllability in various gec scenarios, more specifically, its ability to concentrate on either minimal or fluency aspects (section 5.1) and edits based on learner levels (section 5.2). experimental results indicate that task instructions alone may be sufficient to control editing without examples. however, we found that combining task instructions with examples resulted in more effective controlling performance. this indicates the importance of both task instruction and examples for better control of gec settings using prompt-based methods, although the example set tends to have more importance. ## overall experimental settings in this study, we designed a series of experiments using the prompt-based method with gpt-3 to evaluate the performance in gec tasks. we utilized the gpt-3 model (text-davinci-003) through the api provided by openai 1 . our experiments were conducted in two settings: zero-shot and few-shot. ## general performance to address research question 1) mentioned in section 1, we investigated the overall performance of the prompt-based method with gpt-3 in gec tasks. this investigation is particularly relevant given the increasing prevalence of gpt-3 in various nlp applications and the need to assess its potential capabilities for gec tasks specifically. ## investigation on prompt in this section, we analyze the impact of different factors in prompt on the performance of gpt-3 in gec tasks. we focus on two factors: (1) the type of task instructions used and (2) the number of examples used in the few-shot settings. our primary objective is to comprehend the influence of various factors in prompts to the models' output, which will enable us to optimize gpt-3 more effectively for gec tasks. ## controllability through prompt in this section, we explore gpt-3's controllability for gec tasks through prompt-based methods. our experiments focus on two settings: (1) comparing the model's performance when instructed to make minimal edits versus emphasizing fluency, and (2) tailoring the editing to different learner levels, including beginner, intermediate, advanced, and native speakers. we aim to gain insights into gpt-3's flexibility and controllability under various conditions. we also analyze the relative influence of task instruction and examples to identify the factor that significantly impacts the model's output. ## conclusion in conclusion, this study demonstrates the potential of using prompt-based methods with gpt-3 for gec tasks, achieving competitive performance compared to traditional supervised and unsupervised methods. by carefully crafting task instructions and examples, we show that gpt-3 can be effectively controlled to focus on different aspects of the gec process and adapt to diverse learning needs. our findings highlight the importance of optimizing task instructions and example selection to enhance the performance and controllability of gpt-3, paving the way for further research on refining prompt engineering techniques and exploring their applicability to other nlp tasks and language models.