A Closer Look at k-Nearest Neighbors Grammatical Error Correction (2023)

Abstract: In various natural language processing tasks, such as named entity recognition and machine translation, example-based approaches have been used to improve performance by leveraging existing knowledge. However, the effectiveness of this approach for Grammatical Error Correction (GEC) is unclear. In this work, we explore how an example-based approach affects the accuracy and interpretability of the output of GEC systems and the trade-offs involved. The approach we investigate has shown great promise in machine translation by using the $k$-nearest translation examples to improve the results of a pretrained Transformer model. We find that using this technique increases precision by reducing the number of false positives, but recall suffers as the model becomes more conservative overall. Increasing the number of example sentences in the datastore does lead to better performing systems, but with diminishing returns and a high decoding cost. Synthetic data can be used as examples, but the effectiveness varies depending on the base model. Finally, we find that finetuning on a set of data may be more effective than using that data during decoding as examples.

## introduction grammatical error correction (gec) is the task of identifying and correcting grammatical mistakes in ungrammatical text. while it can be used to assist native speakers as well, it is frequently applied to text written by language learners, and can be used pedagogically to help them improve their writing skills. providing feedback on grammatical errors in a learner's writing allows them to learn from their mistakes and improve their writing over time. for this feedback to be effective, it must be interpretable to the learner. gec models are often based on neural machine translation (nmt) models and treated as similar to sequence-to-sequence (seq2seq) tasks @xcite . unfortunately, transformer-based seq2seq models produce corrections that are uninterpretable, because they simply output a corrected sentence without any indication of how or why elements of the sentence were corrected. this lack of interpretability can make it difficult for learners to understand the nature of their mistakes and how to avoid them in the future. in contrast, example-based approaches to gec can provide a motivating example for each correction, making the results more interpretable and therefore more helpful for learners. this can make the difference between a learner simply correcting a mistake and actually understanding why it is a mistake and how to avoid it in the future. example-based, or instance-based, methods have recently been applied to tasks across the field such as named entity recognition @xcite , summarization @xcite , and machine translation @xcite . in their recent work, @xcite presented their findings on the interpretability of gec corrections using human evaluation and three example selection methods: token-based retrieval, bert-based retrieval, and their example-based grammatical error correction (eb-gec) system. the study found that presenting examples is more useful to learners than providing none, with eb-gec providing the most useful examples for language learners' understanding and acceptance of the model corrections. eb-gec is based on the k-nearest neighbors approach to machine translation proposed by @xcite . this method uses a datastore constructed from a set of example sentence pairs during the decoding of the vanilla transformer. at each timestep, the vector being passed into the final feedforward network of the decoder is used to locate the k nearest neighbor examples in the datastore. this vector represents the translation context, which is composed of the ungrammatical sentence plus the prefix of the output. the datas-tore itself is constructed from a corpus, authentic or synthetic, of training examples. one entry into the datastore is made for each token of the corrected sentence of each example pair, using the encoded translation context as the key, and the ground-truth token as the value. during inference, the retrieved values of the k nearest contexts form a distribution of target tokens. the distribution of target tokens collected from the datastore is then interpolated with the distribution from the base transformer. in this way, the output of the vanilla transformer is influenced by the most similar examples from the datastore, and motivating examples are returned for each token of the output. @xcite reports high bleu score gains in resource-rich languages with large databases, but less impressive performance in low-resource languages. treating gec as a lowresource machine translation task was proposed by @xcite and has resulted in many high performing systems @xcite . however, there are key differences between grammatical error correction and machine translation. gec is a monolingual task, where both input and output share a vocabulary, and a large number of tokens from the input sentence remain unchanged in the output sentence. this difference may very well affect the viability of using k-nearest neighbors for grammatical error correction. @xcite found that their eb-gec system improved the f 0.5 score on three out of four test sets, relative to the vanilla transformer. however, there are several factors to consider about these results. the three test sets that performed better using eb-gec came from datasets with training splits used for both training the vanilla transformer and as the datastore of example sentences. the fourth test set that performed better with the vanilla transformer did not have any representation in the datastore or the training. this may indicate that eb-gec is not generalizable, as in a way, the three test sets with better scores can be thought of as indomain, because they had similar sentences used in the datastore. it is possible that using example sentence pairs produced in a different context would produce lower scores on the test sets. it is worth investigating how using different data for the example corrections than during training affects the results. the reported scores of this system are lower than those reported by @xcite , using a vanilla transformer pretrained on synthetic data. it is unclear whether applying the same knn method to a higher performing transformer would yield the same gains. a detailed analysis of the costs and benefits of using the k-nearest neighbor approach to grammatical error correction as proposed by @xcite has yet to be carried out, but the results of the initial experiments are worth investigating further. this work aims to address some outstanding questions about the effectiveness of k-nearest neighbors for grammatical error correction (knn-gec). specifically, we seek to determine whether knn-gec always improves the performance of the base transformer, or whether the impact varies. additionally, we investigate how the size of the datastore affects performance, and whether synthetic data can be used to bolster the datastore. we also explore whether using synthetic data produces the same level of interpretability. furthermore, we examine how the choice of data for the datastore impacts the effectiveness of the model on test sets. finally, we compare the effectiveness of finetuning a transformer on a set of data versus using that data as the datastore for knn-gec. we found that the effectiveness of knn-gec varies depending on the base transformer. higher performing transformers show little to no improvement. using synthetic data does not appear to impact the interpretability of the corrections, and can be used to increase the size of the datastore. however, very large datastores may not improve the system's performance enough to warrant the increase in computational cost. bolstering the datastore with error-targeted example sentences does not seem to be a viable way of improving the system's performance on those error types or in general. we also found that finetuning a transformer on a set of in-domain data can be more effective than using knn-gec for in-domain data. ## discussion overall, knn-gec makes a base gec system more conservative about making corrections, which lowers its recall. this is likely due to the fact that the chosen examples may not contain a similar error to the one being corrected, but are closer in content to the example pair. the retrieval method involves comparing embedded vectors from the decoder, which contain a mixture of information about the syntax and semantics. as a result, there can be times when the closest sentence pair to the one being decoded overlaps more heavily on semantics than syntax. it is likely that the tendency to retrieve example sentence pairs that are similar in content but not grammatically incorrect promotes more copying, or more conservative corrections, as the target word may not even be incorrect in the example pair. this is a key difference between the tasks of machine translation and grammatical error correction. machine translation must generate the appropriate content words for the translation, while gec mostly uses the content words from the source sentence, or a different form of the existing word. mt may benefit from the influence of the knn probability distribution on word choice because success in mt often includes selecting the correct content word for the context. in the case of gec, however, example sentence pairs may not contain the same grammatical errors as the query sentence, which leads to more copying and less correcting. ## conclusion in this work, we investigate how using training examples during decoding with the knn-gec method affects the precision and recall of grammatical error correction. we used three different base models and found that the effectiveness of knn-gec varies greatly depending on the base model. in general, this method makes models more conservative in making corrections, improving precision but lowering recall. synthetic data can be used to increase the size of the datastore, but its effectiveness depends on the base model. while knn-gec using authentic or synthetic datastores increases the interpretability of corrections for learners, this comes with the trade-off of fewer corrections made and a longer decoding time. we also explored the effect of the hyperparameter λ on the performance of the knn-gec method. a value of 0.4 tended to produce the best balance of precision and recall, though many papers use a value of 0.5. finally, we compared the effectiveness of finetuning a transformer on a set of data versus using that data as the datastore for knn-gec. our results showed that finetuning the transformer on the data generally outperformed using the data as a datastore. given that the more conservative corrections indicate that knn-gec is retrieving examples that are more semantically similar than those containing similar corrections, a future direction for knn-gec could be selecting the k nearest neighbors based on similarity of errors rather than similarity of content. this would require a target output that expresses the difference between the input sentence and the hypothesis sentence. during decoding, the necessary edits would be applied to change the example source to the example target token to the input. future work could involve separating the syntax from the semantics of the encoded input sentence to retrieve the nearest neighbors with syntactically similar example sentences. this would help overcome the limitations of the knn system with regards to making new corrections.