Mitigating Exposure Bias in Grammatical Error Correction with Data Augmentation and Reweighting (2023)

Abstract: The most popular approach in grammatical error correction (GEC) is based on sequence-to-sequence (seq2seq) models. Similar to other autoregressive generation tasks, seq2seq GEC also faces the exposure bias problem, i.e., the context tokens are drawn from different distributions during training and testing, caused by the teacher forcing mechanism. In this paper, we propose a novel data manipulation approach to overcome this problem, which includes a data augmentation method during training to mimic the decoder input at inference time, and a data reweighting method to automatically balance the importance of each kind of augmented samples. Experimental results on benchmark GEC datasets show that our method achieves significant improvements compared to prior approaches.

## introduction grammatical error correction (gec) is the task of correcting errors in a source sentence and generating a well-written and grammatically correct target sentence. good results have been achieved by state-of-the-art gec systems @xcite based on the sequenceto-sequence (seq2seq) transformer @xcite architecture. generally, the seq2seq models are optimized to predict the next token given all previous context tokens at each time step. during training, the ground truth tokens are used as the context, i.e., the socalled teacher forcing mechanism. however, during inference, the context is the previous tokens predicted by the model. as a result, the contexts at training and inference time are drawn from different distributions. this discrepancy is called exposure bias @xcite , which causes the model to make inferences under conditions that ## related work this section briefly introduces recent related research in the literature. ## method in this section, we first briefly introduce a baseline gec system in section 3.1. then, we introduce our data manipulation method in section 3.2. ## experiments in this section, we report the effectiveness of our data manipulation approach. ## conclusion in this paper, we propose a novel data manipulation approach that includes both data augmentation and data reweighting to mitigate the exposure bias problem for seq2seq gec models. our core idea is to inject realistic augmented sentences into the decoder input and to automatically sample the augmented sentences. we show the effectiveness of our method which achieves significant improvements on both the conll-2014 and bea-2019 test sets.