Dynamic Negative Example Construction for Grammatical Error Correction using Contrastive Learning (2022)

Abstract: “Grammatical error correction (GEC) aims at correcting texts with different types of grammatical errors into natural and correct forms. Due to the difference of error type distribution and error density, current grammatical error correction systems may over-correct writings and produce a low precision. To address this issue, in this paper, we propose a dynamic negative example construction method for grammatical error correction using contrastive learning. The proposed method can construct sufficient negative examples with diverse grammatical errors, and can be dynamically used during model training. The constructed negative examples are beneficial for the GEC model to correct sentences precisely and suppress the model from over-correction. Experimental results show that our proposed method enhances model precision, proving the effectiveness of our method.”

## introduction grammatical error correction (gec) @xcite @xcite aims at correcting texts with different types of grammatical errors into natural and correct forms. it is an important research topic for both natural language processing and language education. most of the current gec systems are developed for correcting writings by learners of english as a second language @xcite @xcite @xcite . however, gec for native writings is also worth exploring, as texts written by native speakers may also contain grammatical errors that should be corrected for enhancement of writing quality. currently, it is not feasible to train a gec model specifically for correcting native writings because gec data containing native writings are not sufficient. therefore, native writings are often corrected by gec models that are trained on gec data consisting of writings by non-native speakers such as the lang-8 @xcite and nucle @xcite datasets. however, the error type distribution, error density and fluency are inconsistent between the writings by non-native and native speakers. therefore, those gec models may over correct sentences and produce a low precision of error correction @xcite . in terms of this issue, contrastive learning (cl) @xcite @xcite can be incorporated to help alleviate the over correction behaviour of the gec models. the core idea is to take the over-corrected sentences as negative examples, and to effectively avoid or alleviate the problem of over correction by increasing the distance between the anchor sentence and the negative examples. so the focus is on how to construct effective negative examples for training the gec models effectively. previous studies about gec models mainly focus on data augmentation for generating pseudo parallel training pairs as complement for the current insufficient gec training data @xcite , or focus on improving the correction performance with a variety of model architectures @xcite @xcite , few of them focus on improving the performance of the gec models with contrastive learning. to the best of our knowledge, @xcite is the only recent work for that. in their work, they propose two approaches for constructing negative examples. first, they treat the beam search candidates produced by an off-the-shelf gec model as negative examples. they find that many output candidates generated by beam search contain erroneous edits, and the constructed negative examples help suppress the trained gec model from producing erroneous edits. second, the source sentence is treated directly as a negative example if it contains grammatical errors. their intuition is that there should be differences between the corrected output sentence and the source sentence, otherwise the gec model fails to detect any grammatical errors in the source sentence. the negative examples constructed in this way suppress the trained gec model from outputting the erroneous source sentences as they are without any modifications. although the aforementioned study produces good performance, we believe that there are still two points that can be improved: (1) the negative examples constructed with beam search may not be sufficient. many beam search output candidates are the same as the target sentence and cannot be used as negative examples. that leads to a small number of the generated negative examples. in addition, the beam search candidates may contain unrealistic grammatical errors with a small number of error types, limiting the diversity of grammatical errors in the generated negative examples. as a result, the low diversity of the negative examples makes the gec model less easier to learn to distinguish the negative examples from the anchor, which limits the improvement of error correction performance brought by contrastive learning. (2) they construct the negative examples with their negative example construction methods before model training. as a result, the gec model can only be able to see a fixed set of negative examples in each iteration during training, which may reduce the generalization ability of the gec model. to this end, we propose a dynamic negative example construction method for grammatical error correction using contrastive learning. the proposed method contains a negative example construction strategy that makes use of realistic grammatical error patterns produced by humans to generate sufficient negative examples with more diverse grammatical errors. with the constructed negative examples, the gec model can learn to correct sentences precisely and be suppressed from over-correction. moreover, the proposed strategy is simple and lightweight, enabling it to be applied dynamically during the training process. in this manner, our method enhances the generalization ability of the gec model. the main contributions of this work are as follows: ## related work in this section, we briefly review different gec methods, including the early rule-based methods, the widely used methods based on machine translation or bert, and the recently proposed gec methods using contrastive learning. ## our method in this section, we first describe the overall architecture of our method in section 3.1. then, we will detail the proposed negative example construction strategy in section 3.2, and the proposed dynamic mechanism in section 3.3. ## conclusion in this paper, a dynamic negative example construction method for grammatical error correction using contrastive learning is proposed. the proposed method constructs sufficient negative examples with diverse grammatical errors dynamically during model training. the constructed negative examples are beneficial for the gec model to correct sentences precisely and suppress the model from over-correction. experimental results show that our proposed method enhances the correction precision significantly. in this study, positive example construction strategy is not proposed for grammatical error correction using contrastive learning, as it is hard to construct sentences that are morphologically different from but semantically identical to the target sentence. one possible solution for that may be utilizing data augmentation. in future work, we will investigate this topic in depth.