Improving C hinese Grammatical Error Detection via Data augmentation by Conditional Error Generation (2022)

Abstract: Chinese Grammatical Error Detection(CGED) aims at detecting grammatical errors in Chinese texts. One of the main challenges for CGED is the lack of annotated data. To alleviate this problem, previous studies proposed various methods to automatically generate more training samples, which can be roughly categorized into rule-based methods and model-based methods. The rule-based methods construct erroneous sentences by directly introducing noises into original sentences. However, the introduced noises are usually context-independent, which are quite different from those made by humans. The model-based methods utilize generative models to imitate human errors. The generative model may bring too many changes to the original sentences and generate semantically ambiguous sentences, so it is difficult to detect grammatical errors in these generated sentences. In addition, generated sentences may be error-free and thus become noisy data. To handle these problems, we propose CNEG, a novel Conditional Non-Autoregressive Error Generation model for generating Chinese grammatical errors. Specifically, in order to generate a context-dependent error, we first mask a span in a correct text, then predict an erroneous span conditioned on both the masked text and the correct span. Furthermore, we filter out error-free spans by measuring their perplexities in the original sentences. Experimental results show that our proposed method achieves better performance than all compared data augmentation methods on the CGED-2018 and CGED-2020 benchmarks.

## introduction the goal of grammatical error detection is to detect grammatical errors in texts @xcite . it is useful for many nlp applications such as writing assistant @xcite , search engine @xcite , and speech recognition systems @xcite , etc. grammatical errors may appear in all languages @xcite . in this paper, we only investigate the problem of chinese grammatical error detection (cged). grammatical error detection is usually formulated as a sequence tagging task, where each erroneous token is assigned with an error type, e.g., selection errors and redundant words, as shown in figure 1 . since annotating grammatical errors requires rich linguistic knowledge, it is expensive and time-consuming to annotate a large-scale corpus. therefore, the scarcity of labeled data is one of the main challenges for this task. to handle this problem, previous works proposed various data augmentation methods to automatically generate more training samples @xcite @xcite . the methods of generating erroneous sentences can be roughly categorized into the following two types: (1) rule-based methods. these methods construct erroneous sentences by introducing noises into original texts, e.g., inserting, deleting, or replacing some words @xcite . as the erroneous sentence shown in figure 1 , human grammatical errors are usually context dependent. on the contrary, the randomly introduced errors are context-independent (case i in figure 2 ), therefore these noise-corrupted sentences are quite different from the erroneous sentences made by humans. (2) model-based methods. in order to imitate human errors, many studies utilize neural generative models to generate grammatical errors, such as seq2seq models @xcite , and translation models which obtain erroneous sentences via round-trip translation through a bridge language @xcite . however, considering that the outputs of generative models are not usually faithful to the inputs @xcite , semantic and syntactic ambiguities may arise when the generative models bring too many changes to the original sentences (case ii in figure 2 ). even human can not infer the correct sentences from these generated sentences, so it is also difficult for the detection model to automatically detect grammatical errors. meanwhile, generated sentences may be error-free and become noisy data (case iii in figure 2 ). therefore, these constructed samples have little contributions to improving the performance of detection models. to handle the aforementioned problems, we propose cneg, a novel conditional non-autoregressive error generation (cneg) model for generating chinese grammatical errors. figure 3 illustrates the architecture of the model. specifically, to predict a context-dependent error, we first mask a span of a correct text, and utilize bert @xcite to conduct non-autoregressive span prediction. in order to ensure that the generated sentence will be faithful to the original sentence, we force the model to generate span conditioned on the original span. considering that the correct information is integrated into the model, we further introduce a penalty to encourage the model not to directly reconstruct the correct span. our cneg model is based on bert, which is pretrained on a large scale of chinese corpus. therefore, the model can generate errors that do not appear in the training dataset. finally, in order to filter out the error-free spans, we also utilize a pre-trained bert to measure the perplexities of generated spans. the main contributions of this paper can be summarized as follows: ## related work chinese grammatical error detection (cged) aims at detecting grammatical errors in chinese sentences @xcite . most studies regard it as a sequence tagging task, where each token will be given a correct label or an error-type. sequence labeling methods are widely used for cged, such as feature-based statistical models @xcite , and neural models @xcite . due to the effectiveness of bert @xcite in many other nlp applications, recent studies adopt bert as the basic architecture of cged models @xcite @xcite . @xcite propose a model that combines resnet and bert to achieve state-ofthe-art results on the cged-2020 task. @xcite apply a crf layer on bert to introduce the dependency of tokens. however, neural models usually require a large amount of training data, and manually annotating a large corpus is expensive and time-consuming. therefore, many studies focus on data augmentation methods to automatically generate large-scale training samples to boost the performance of grammatical error detection models @xcite @xcite . @xcite investigated different strategies of the incorporation of pseudo data, including the method of generating the pseudo data, the seed corpus for augmentation, and training strategies with these augmented samples. wang et al. (2019 ) proposed a rule-based editing method that constructs the noise-corrupted text. instead of directly adding noise into the sentence, wan et al. (2020) introduce noise to the representation of a sentence and apply the seq2seq model to generate sentences with various error types. lichtarge et al. (2019) use an intermediate language as a bridge to generate grammatical error samples. zhou et al. (2020) consider that neural machine translation (nmt) model is significantly better than the statistical machine translation (smt) model, then utilize nmt model and smt model to generate correct and erroneous sentences respectively. moreover, @xcite firstly identify the most vulnerable tokens by a seq2seq model, then replace these tokens with the grammatical errors which are collected from the training dataset. ## conclusions in this paper, considering that grammatical errors made by humans are usually context-dependent, we propose a conditional non-autoregressive error generation method (cneg) for data augmentation of cged. by introducing the correct span into the non-autoregressive model, the model will generate errors conditioned on both the context and the correct span. observing that the model may generate correct spans, a filtering strategy is proposed to filter out error-free spans. experimental results show that our method outperforms all compared data augmentation methods on the cged datasets, which demonstrates the effectiveness of our method.