Interpretability for Language Learners Using Example-Based Grammatical Error Correction (2022)

Abstract: Grammatical Error Correction (GEC) should not focus only on high accuracy of corrections but also on interpretability for language learning. However, existing neural-based GEC models mainly aim at improving accuracy, and their interpretability has not been explored.A promising approach for improving interpretability is an example-based method, which uses similar retrieved examples to generate corrections. In addition, examples are beneficial in language learning, helping learners understand the basis of grammatically incorrect/correct texts and improve their confidence in writing. Therefore, we hypothesize that incorporating an example-based method into GEC can improve interpretability as well as support language learners. In this study, we introduce an Example-Based GEC (EB-GEC) that presents examples to language learners as a basis for a correction result. The examples consist of pairs of correct and incorrect sentences similar to a given input and its predicted correction. Experiments demonstrate that the examples presented by EB-GEC help language learners decide to accept or refuse suggestions from the GEC output. Furthermore, the experiments also show that retrieved examples improve the accuracy of corrections.

## introduction grammatical error correction (gec) models, which generate grammatically correct texts from grammatically incorrect texts, are useful for language learners. in gec, various neural-based models have been proposed to improve the correction accuracy @xcite @xcite @xcite . however, the basis on which a neural gec model makes corrections is generally uninterpretable to learners. neural gec models rarely address correction interpretability, leaving language learners with no explanation of the reason for a correction. interpretability plays a key role in educational scenarios @xcite . in particular, presenting examples is shown to be effective in improving understanding. language learners acquire grammatical rules and vocabulary from examples @xcite . presenting examples of incorrect sentences together with correct ones improves the understanding of grammatical correctness as well as essay quality @xcite . recently, example-based methods have been applied to a wide range of natural language processing tasks to improve the interpretability of neural models, including machine translation @xcite , part-of-speech tagging @xcite , and named entity recognition @xcite . these methods predict labels or tokens by considering the nearest neighbor examples retrieved by the representations of the model at the inference time. @xcite showed that in machine translation, examples close to a target sentence in the representation space of a decoder are useful for translating the source sentence. inspired by this, we hypothesized that examples corrected for similar reasons are dis-figure 2 : an illustration of how eb-gec chooses examples and predicts a correction. the model predicts a correction "they have /a tremendous problem ." by using the example "this has /a tremendous problem ." hidden states of the decoder computed during the training phase are stored as keys, and tokens of the output sentences corresponding to the hidden states are stored as values. a hidden state of the decoder (blue box) at the time of inference is used as a query to search for k-neighbors (yellow box) of hidden states of the training data. eb-gec predicts a distribution of tokens for the correction from a combination of two distributions of tokens: a vanilla distribution computed by transforming the hidden state of the decoder; and a knn distribution by the retrieved k-neighbors. in this paper, we introduce an example-based gec (eb-gec) 1 that corrects grammatical errors in an input text and provides examples for language learners explaining the reason for correction (figure 1 ). as shown in figure 2 , the core idea of eb-gec is to unify the token prediction model for correction and the related example retrieval model from the supervision data into a single encoderdecoder model. eb-gec can present the reason for the correction, which we hope will help learners decide whether to accept or to refuse a given correction. experimental results show that eb-gec predicts corrections more accurately than the vanilla gec without examples on the three datasets and comparably on one dataset. experiments with human participants demonstrate that eb-gec presents significantly more useful examples than the baseline methods of example retrieval @xcite @xcite . these results indicate that examples are useful not only to the gec models but also to language learners. this is the first study to demonstrate the benefits of examples themselves for real users, as existing studies @xcite @xcite only showed example utility for improving the task accuracy. 1 our code is publicly available at https://github. com/kanekomasahiro/eb-gec 2 eb-gec eb-gec presents language learners with a correction and the related examples it used for generating the correction of the input sentence. k-nearest-neighbor machine translation (knn-mt; @xcite was used as a base method to consider example in predicting corrections. knn-mt predicts tokens by considering the nearest neighbor examples based on representations from the decoder at the time of inference. eb-gec could use any method @xcite @xcite to consider examples, but knn-mt was used in this study because it does not require additional training for example retrieval. figure 2 shows how the eb-gec retrieves examples using knn-mt. eb-gec performs inference using the softmax distribution of target tokens, referred to as vanilla distribution, hereafter, obtained from the encoder-decoder model and the distribution generated by the nearest neighbor examples. nearest neighbor search is performed for a cache of examples indexed by the decoder hidden states on supervision data (knn distribution). eb-gec can be adapted to any trained autoregressive encoderdecoder gec model. a detailed explanation of retrieving examples using knn-mt is provided in section 2.1, and of presenting examples in section 2.2. ## experiments this section investigates the effectiveness of the examples via manual evaluation and accuracy on the gec benchmark to show that the eb-gec does, in fact, improve the interpretability without sacrificing accuracy. we first describe the experimental setup and then report the results of the experiments. ## conclusion we introduced eb-gec to improve the interpretability of corrections by presenting examples to language learners. the human evaluation showed that the examples presented by eb-gec supported language learners' decision to accept corrections and improved their understanding of the correction results. although existing interpretive methods using examples have not verified if examples are helpful for humans, this study demonstrated that examples were helpful for learners using gec. in addition, the results of the gec benchmark showed that eb-gec could predict corrections more accurately or comparably to its vanilla counterpart. future work would include investigations of whether example presentation is beneficial for learners with low language proficiency. in addition, we plan to improve the datastore coverage by using pseudo-data @xcite and weight low frequency error types to present diverse examples. we explore whether methods to improve accuracy and diversity @xcite @xcite are effective for eb-gec.