GEE ! Grammar Error Explanation with Large Language Models (2024)

Abstract: Existing grammatical error correction tools do not provide natural language explanations of the errors that they correct in user-written text. However, such explanations are essential for helping users learn the language by gaining a deeper understanding of its grammatical rules (DeKeyser, 2003; Ellis et al., 2006).To address this gap, we propose the task of grammar error explanation, where a system needs to provide one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences. The task is not easily solved by prompting LLMs: we find that, using one-shot prompting, GPT-4 only explains 40.6% of the errors and does not even attempt to explain 39.8% of the errors.Since LLMs struggle to identify grammar errors, we develop a two-step pipeline that leverages fine-tuned and prompted large language models to perform structured atomic token edit extraction, followed by prompting GPT-4 to explain each edit. We evaluate our pipeline on German, Chinese, and English grammar error correction data. Our atomic edit extraction achieves an F1 of 0.93 on German, 0.91 on Chinese, and 0.891 on English. Human evaluation of generated explanations reveals that 93.9% of German errors, 96.4% of Chinese errors, and 92.20% of English errors are correctly detected and explained. To encourage further research, we open-source our data and code.

## introduction grammatical error correction (gec) is a practical and valuable application of natural language processing that facilitates both proofreading of text and language learning. recent advances in large language models (llms) have significantly improved the capabilities of gec systems (wang et al., 2021; figure 1 : an illustration of the two-step pipeline of grammar error explanation (gee). given a pair of sentences with corrected errors, the gee system first extracts linguistically meaningful edit units as errors. the extracted errors are then paired with the sentences as the input for gee generation. note: the error in einen can be caused by gender or case or both. without guessing the mental state of a language user, both are offered as the reasons in the explanation. bryant et al., 2023); however, they are unable to explain errors in natural language alongside providing correction. error explanation is crucial to language learning and teaching (ellis, 2010): while corrections are a form of implicit feedback, they are not as impactful as explicit feedback (dekeyser, 2003; ellis et al., 2006), which involves pointing out errors and providing meta-linguistic information to the user (e.g., rules of writing well-formed phrases or sentences). in this work, we propose a new task-grammar error explanation (gee)-for which a model must generate natural language error explanations that help language learners acquire and enhance grammar knowledge. as shown in figure 1 , given a pair of sentences in which one sentence has grammar errors and the other one is corrected, a model needs to generate an explanation for each corrected grammar error. given the capabilities of modern llms, one might ask whether llms can solve this task simply via prompting. we show in section 3 that one-shot gpt-4 (openai, 2023) prompting detects only 60.2% of the true errors and correctly explains only 67.5% of the errors it does detect. given this result, we develop a pipeline for gee generation that features an essential intermediate step-atomic token edit extraction. as shown in figure 1 , given an erroneous sentence and its corrected counterpart (source and target), we first extract atomic edits at the token level by prompting or fine-tuning llms such as gpt-4, which also label the edits with one of four operation-level edit types: insert, delete, replace, and relocate. @xcite in the second step, we append the extracted edits to the source and target sentences and use them as the input to a gee system. we utilize the few-shot learning ability of llms (brown et al., 2020) to generate error explanations using carefully crafted language-specific prompts. we validate our gee pipeline on german, chinese, and english, three very different languages (fusional vs. analytical). we recruit language teachers to evaluate the correctness of the german and chinese explanations. the english outputs are evaluated by the authors manually. for the first step in the pipeline, our atomic edit method extracts 92.3% of the true edits for german, which is 32.1% higher than the one-shot approach in section 3. for the final gee outputs in german, 93.9% of the generated explanations are judged as correct by two german teachers. similar performance is observed in chinese with a 96.4% correctness rate and 92.20% in english. the results suggest that our two-step pipeline together with carefully crafted language-specific prompts generalizes well for the three different languages. in summary, our contributions are the following. first, we propose a new task on grammar error explanation to enhance the utility of current grammatical error correction systems. second, we propose a two-step pipeline and study its performance in german, chinese, and english with detailed error analysis. third, we publicly release our atomic edit extraction datasets for german, chinese, and english as well as all llm-generated gee outputs @xcite these types describe a general relationship between the source and target rather than precise edit operation of the source. with the goal of enabling future research on gee and facilitating the development of more effective gee systems. ## gee task definition while most gec models provide viable grammar error corrections (bryant and ng, 2015; bryant et al., 2023), they do not provide natural language explanations alongside the corrections, which are critical for language learners in mastering grammar (ellis et al., 2006; ellis, 2010). in this section, we propose and define the task of grammar error explanation, which aims to fill this gap. we assume that a gee model has access to the outputs of an existing gec model, which produces the corrected form of an ungrammatical input sentence. ## has gpt-4 already solved gee? a natural question one might ask is whether stateof-the-art llms can solve the gee task in an endto-end manner. this section demonstrates that gpt-4 in its current form is error-prone. it has low error coverage and hallucinates frequently. based on this observation, we experiment with an approach which provides gpt-4 with a list of manuallyextracted gold atomic edits. results show that access to this gold edit list improves the performance greatly, indicating substantial headroom with more structured prompting as we describe in section 4. one-shot prompting of gpt-4. we run an experiment using german grammar error correction data (details in section 5). we randomly sample 30 data points foot_2 and generate explanations using the one-shot prompt in appendix a. ## pipeline for generating gee in section 3, we observed that including a list of gold atomic edits to gpt-4's prompt greatly improves error coverage. we thus propose a two-step pipeline for gee that uses atomic edit extraction as the intermediate step. the pipeline is illustrated in figure 1 . given an input sentence pair defined in section 2.1, we first extract atomic edits from the pair following section 2.2. the edits are then appended to the sentences to form the input for the final step, where gpt-4 is prompted to generate an explanation and an error type. ## german, chinese, and english datasets this section introduces the datasets that are used in our experiments. statistics of the sampled data subsets are reported in ## experimental results this section presents the results of the gee pipeline in german, chinese, and english. we first present the results of the fine-tuned and prompted models on atomic edit extraction in each language. the fine-tuned gpt-3.5-turbo achieved the best performance on edit extraction for german and english but gpt-4 works the best for chinese. section 6.2 presents the human evaluation results of gee outputs generated by gpt-4. among the german gee outputs, 93.9% are judged as correct by two german teachers. for chinese gee outputs, 96.4% of the outputs are correct according to two chinese teachers. for the english gee outputs, 92.20% of the outputs are correct according to our manual evaluation. ## related work our gee task is built on the actively studied gec task, which is often formulated as a neural machine translation task (boyd, 2018; bryant et al., 2023; yuan and bryant, 2021; zhang et al., 2022). researchers have explored various aspects of gec. we identify three of them which the gee task can be built on and benefit from. after that, we compare our task to a related task, feedback comment generation, and show how gee is different from it. gec with multi-reference and context. research has been building gec models on data which have only one gold reference for each source input. however, there is an urge to use multiple references for source inputs (bryant and ng, 2015; zhang et al., 2022; xu et al., 2022). in the context of gee, a capable model should generate well-suited explanations for any valid error corrections, which requires reasoning of word relations and recovering correction rationales. such ability of gee models also need to go beyond the sentence level. wang et al. (2022) has shown that even when only one sentence is added to the input as the context, a gec model's performance can be significantly boosted. if some errors can only be better corrected in context, they can only be better explained in context as well. gec with auxiliary grammar information. existing works have shown improvement of gec models by adding edit types, dependency information, or grammatical error type into the training process (omelianchuk et al., 2020; ma et al., 2022; yang et al., 2023). fei et al. (2023) study the influence of adding evidence words for errors and error types into the pipeline of gec. they found that such information can significantly increase model performance in english gec. for the gee task, it is an interesting direction to explore whether adding those extra information to a gee system can improve its explanations' usefulness. gec with retrieved examples. kaneko et al. (2022) propose to involve retrieved examples into the gec task, which not only improve the gec model's performance but also increase the end users' confidence in deciding whether or not to accept a correction. gee explanations with examples would be a good combination for end users to generalize what they learn from the current error, hence enhancing the knowledge. feedback alongside grammar error detection nagata et al. (2021) proposed a shared task called feedback comment generation for language learners ( fcg). the task differs from our gee task in three important aspects. first, the inputs in fcg are erroneous sentences only, which have spans marked as errors. hence, the fcg task does not need to extract meaningful atomic edits. second, the fcg task focuses solely on preposition words, which are a closed set of function words whose occurrences and usages are limited. third, the fcg task focuses on generating comments as hints for language learners to correct errors themselves (e.g., look up the use of @xmath0.). ## conclusion we present a new task grammar error explanation to provide natural language explanations to grammatical errors. we develop a pipelined approach using llms and atomic token edit extraction. our llm-based pipeline gets a high score of 93.9% in german, 96.37% in chinese, and 92.20% in english error explanation. while we assume a grammar error correction system as the foundation of our gee system, further work are encouraged to explore gee generation alongside gec.