Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation (2020)

Abstract: The incorporation of data augmentation method in grammatical error correction task has attracted much attention. However, existing data augmentation methods mainly apply noise to tokens, which leads to the lack of diversity of generated errors. In view of this, we propose a new data augmentation method that can apply noise to the latent representation of a sentence. By editing the latent representations of grammatical sentences, we can generate synthetic samples with various error types. Combining with some pre-defined rules, our method can greatly improve the performance and robustness of existing grammatical error correction models. We evaluate our method on public benchmarks of GEC task and it achieves the state-of-the-art performance on CoNLL-2014 and FCE benchmarks.

## introduction grammatical error correction (gec) is a task of detecting and correcting grammatical errors in a sentence. due to the growing number of language learners of english, there has been increasing attention to the english gec in the past few years. considering the outstanding performance of neural network models in machine translation tasks, many studies have tackled gec as a machine translation task. they regard ungrammatical sentences as the source language and grammatical sentences as the target language. this approach allows cutting-edge neural machine translation models to be applied to gec. many encoder-decoder models, such as recurrent neural network (rnn) @xcite , convolutional neural networks (cnn) @xcite , have been widely applied to gec. a challenge in applying neural machine translation models to gec is the requirement of a large amount of training data, i.e., the source-target pairs. to address this problem, many data augmentation methods have been proposed. existing methods, however, are often only able to generate sentences with limited error types, and can only improve the performance of gec model on these few error types, while it is still hard for the model to correct the sentences with other types of errors. to address the above problem, we propose a new data augmentation method to generate synthetic samples by editing the latent representations of grammatical sentences. given a target grammatical error type and the corresponding grammatical error type classifier, we can get a perturbation vector in latent space. then we add the perturbation vector to the latent representation of input sentence, and use a decoder to generate a sentence with target grammatical error type. in this way, diverse errors can be generated by assigning different target error types. to further improve the performance, we adopt some rules to assist the generation of some local grammatical errors, such as spelling errors, wrong punctuation, etc. we apply this data augmentation method to the existing gec model copy-transformer @xcite to evaluate the results. experiments are conducted on the following widely used benchmarks: @xcite , fce @xcite , bea-2019 @xcite . experimental results show the efficacy of our proposed method which outperforms several existing models. our contributions are summarized as follows: ## related work early gec models are mainly based on manually designed grammar rules @xcite @xcite . @xcite pointed out the limitation of rule-based method and proposed a statistical model. later, some researchers proposed solutions based on statistical machine learning method @xcite @xcite . with the development of deep learning, recent works proposed a variety of neural network models to deal with gec task. some treated the gec task as a translation problem and applied neural machine translation model to detect and correct grammatical errors. @xcite used a classical bidirectional recurrent neural network @xcite with attention. @xcite proposed a convolution neural network @xcite to capture the local context. many recent works @xcite made use of the powerful machine translation architecture transformer @xcite . @xcite further applied copying mechanism @xcite to transformer. considering the tremendous performance of pre-trained methods, pretrained language model, such as bert @xcite , can be incorporated into the encoder-decoder model @xcite ). an encoder-decoder gec model requires a large amount of training data, and the available training corpora usually failed to train a good gec model. to address this problem, many data augmentation methods have been proposed @xcite . many works adopted pre-defined rules to generate local grammatical errors. @xcite applied a confusion set built by spellchecker to the corpus. @xcite extracted some common text editing operations from human writing habits and got synthetic samples by these extracted operations. lichtarge et al. @xcite made use of models trained on large amounts of weakly supervised text. inspired by back-translation procedure for machine translation @xcite , @xcite proposed a model that can learn to generate erroneous sentences from correct ones. based on this work, @xcite further studied the data augmentation methods and got some empirical conclusions. ## our data augmentation method our data augmentation method applies noise to the latent space and it can generate sentences with various error types by editing latent representations of grammatical sentences. to further improve the performance, we adopt some rules to assist the generation of some local grammatical errors. synthetic training samples generated from our method are used to train neural gec model and enable the model to detect and correct most error types and improve its performance and robustness. ## gec model in this study, we choose copy-augmented transformer @xcite as gec model to test our data augmentation method. copy-augmented transformer is a kind of transformer that incorporates an attention-based copy mechanism in the decoder. it can generate word from a fixed vocabulary and the source input tokens. considering the similarity between input and output, this copy mechanism leads to great performance of models in gec task. nucle is a collection of essays written by students who are non-native english speakers. professional english instructors were invited to correct the grammatical errors in these essays. there are 28 common grammatical error types. the lang-8 corpus is a cleaned english subset of the language learning websites. fce and w&i+locness are public gec datasets. @xcite use an automatic annotation tool errant to annotate the types of grammatical errors. there are 25 main grammatical error types. evaluation data. we report results on conll-2014 benchmark evaluated by official m2 scorer @xcite , and on bea-2019 and fce benchmarks evaluated by errant. seed corpus following the @xcite , we choose the large english corpus gigaword as seed corpus for data augmentation. the datasets used in the experiments are summarized in ## conclusion in this paper, we propose a data augmentation method to apply noise to latent space. by editing latent representations of grammatical sentences, we can generate synthetic samples with diverse error types. these synthetic training samples can further improve the performance and robustness of the gec model and it enables the model to detect and correct most errors. we evaluate our method on public benchmarks of gec task and it achieves the state-of-the-art performances on conll-2014 and fce datasets.