Grammatical Error Correction with Contrastive Learning in Low Error Density Domains (2021)

Abstract: Although grammatical error correction (GEC) has achieved good performance on texts written by learners of English as a second language, performance on low error density domains where texts are written by English speakers of varying levels of proficiency can still be improved. In this paper, we propose a contrastive learning approach to encourage the GEC model to assign a higher probability to a correct sentence while reducing the probability of incorrect sentences that the model tends to generate, so as to improve the accuracy of the model. Experimental results show that our approach significantly improves the performance of GEC models in low error density domains, when evaluated on the benchmark CWEB dataset.

## introduction grammatical error correction (gec) is the task of correcting errors in a source sentence and generating a well-written and grammatically correct target sentence. good results have been achieved by state-of-the-art gec systems based on the seq2seq transformer architecture @xcite @xcite . however, most prior approaches in gec are all targeting english-as-a-second-language (esl) datasets, where gec systems are trained to correct errors made by esl learners. in fact, grammatical and other writing errors are made not only by esl speakers but also by native speakers. therefore, correcting grammatical errors made by native speakers should also be considered, which helps to broaden the application of gec. compared to the errors made by esl learners, native english speakers are less likely to make grammatical errors, so the density of errors in the sentences is much lower. the gec model may end up over-correcting or failing to correct certain errors unique to native speakers. to address the problem mentioned above, it is necessary to improve the ability of the model to discriminate grammatical features from ungrammatical features with minor differences. recently, supervised contrastive learning (cl) was proposed by @xcite , which allows the model to learn discriminative features through pushing the features of positive samples closer together and negative samples further apart. however, since gec is a text generation task, it is not clear how to generate positive sample sentence pairs. to bridge the gap, we instead incorporate cl by increasing the probability of the model generating the right corrections and further reducing the probability of generating the wrong corrections, thereby improving the ability of the model for error correction in low error density domains. more specifically, we use the negative loglikelihood (nll) loss to increase the probability of a model to generate positive samples (the right corrections) and use a margin-based cl loss to increase the gap between the probability of positive samples and the probability of negative samples (the wrong corrections) predicted by the gec model. in this paper, the negative samples are generated in two ways. the first kind of negative samples consists of those wrong corrections generated with high probability by the gec model during beam search. the second kind of negative samples consists of erroneous sentences from the dataset that require some correction. through the above negative sampling method, we make the model avoid over-correcting a correct sentence or neglect to correct an erroneous sentence. the main contributions of this paper are as follows: foot_0 ## method in this section, we will first introduce the background of grammatical error correction in section 2.1, and then introduce our contrastive learning method in section 2.2. ## experiments in this section, we demonstrate the effectiveness of our cl approach. ## conclusion in this paper, we propose a contrastive learning approach and a corresponding negative sampling method to improve the performance of seq2seq gec models in low error density domains. by assigning a higher probability to grammatical corrections and reducing the probability of wrong corrections that the model tends to generate, we improve the performance of gec models in low error density domains.