C hinese Grammatical Correction Using BERT -based Pre-trained Model (2020)

Abstract: In recent years, pre-trained models have been extensively studied, and several downstream tasks have benefited from their utilization. In this study, we verify the effectiveness of two methods that incorporate a pre-trained model into an encoder-decoder model on Chinese grammatical error correction tasks. We also analyze the error type and conclude that sentence-level errors are yet to be addressed.

## introduction grammatical error correction (gec) can be regarded as a sequence-to-sequence task. gec systems receive an erroneous sentence written by a language learner and output the corrected sentence. in previous studies that adopted neural models for chinese gec @xcite , the performance was improved by initializing the models with a distributed word representation, such as word2vec @xcite . however, in these methods, only the embedding layer of a pretrained model was used to initialize the models. in recent years, pre-trained models based on bidirectional encoder representations from transformers (bert) have been studied extensively @xcite , and the performance of many downstream natural language processing (nlp) tasks has been dramatically improved by utilizing these pre-trained models. to learn existing knowledge of a language, a bertbased pre-trained model is trained on a large-scale corpus using the encoder of transformer @xcite . subsequently, for a downstream task, a neural network model is initialized with the weights learned by a pre-trained model that has the same structure and is fine-tuned on training data of the downstream task. using this two-stage method, the performance is expected to improve because downstream tasks are informed by the knowledge learned by the pre-trained model. recent works @xcite show that bert helps improve the performance on the english gec task. as the chinese pre-trained models are developed and released continuously @xcite , the chinese gec task may also benefit from using those pre-trained models. in this study, as shown in figure 1 , we develop a chinese gec model based on transformer with a pre-trained model using two methods: first, by initializing the encoder with the pre-trained model (bert-encoder); second, by utilizing the technique proposed by @xcite , which uses the pre-trained model for additional features (bertfused); on the natural language processing and chinese computing (nlpcc) 2018 grammatical error correction shared task test dataset @xcite , our single models obtain f 0.5 scores of 29.76 and 29.94 respectively, which is similar to the performance of ensemble models developed by the top team of the shared task. moreover, using a 4-ensemble model, we obtain an f 0.5 score of 35.51, which outperforms the results from the top team by a large margin. we annotate the error types of the development data; the results show that word-level errors dominate all error types and that sentence-level errors remain challenging and require a stronger approach. ## related work given the success of the shared tasks on english gec at the conference on natural language learning (conll) @xcite , a chinese gec shared task was performed at the nlpcc 2018. in this task, approximately one million sentences from the language learning website lang-8 1 were used as training data and two thousand sentences from the pku chinese learner corpus @xcite were used as test data. here, we briefly describe the three methods with the highest performance. first, @xcite combined a 5-gram language model-based spell checker with subwordlevel and character-level encoder-decoder models using transformer to obtain five types of outputs. then, they re-ranked these outputs using the language model. although they reported a high performance, several models were required, and the combination method was complex. second, ren et al. (2018) utilized a convolutional neural network (cnn), such as in @xcite . however, because the structure of the cnn is different from that of bert, it cannot be initialized with the weights learned by the bert. last, zhao and wang (2020) proposed a dynamic masking method that replaces the tokens in the source sentences of the training data with other tokens (e.g. [pad] token). they achieved stateof-the-art results on the nlpcc 2018 grammar error correction shared task without using any extra knowledge. this is a data augmentation method that can be a supplement for our study. ## methods in the proposed method, we construct a correction model using transformer, and incorporate a chinese pre-trained model developed by @xcite in two ways as described in the following sections. ## discussion case analysis in the first example, the spelling error 持别 is accurately corrected to 特别 (which means especially) by the proposed model, whereas it is not corrected by the baseline model. hence, it appears that the proposed model captures context more efficiently by using the pre-trained model through the wwm strategy. in the second example, the output of the proposed model is more fluent, although the correction made by the proposed model is different from the gold edit. the proposed model not only changed the wrong word 经验 (which usually means the noun experience) to 经历 (which usually means the verb experience), but also added a new word 会 (would, could); this addition makes the sentence more fluent. it appears that the proposed model can implement additional changes to the source sentence because the pre-trained model is trained with a large-scale corpus. however, this type of change may affect the precision because the gold edit in this dataset followed the principle of minimum edit distance @xcite . error type analysis to understand the error distribution of chinese gec, we annotate 100 sentences of development data and obtain 130 errors (one sentence may contain more than one error). we refer to the annotation of the hsk learner corpus 7 and adopt five categories of error: b, cc, cq, cd, and cj. b denotes character-level errors, which are mainly spelling and punctuation errors. cc, cq, and cd are word-level errors, which are word selection, missed word, and redundant word errors, respectively. cj denotes sentence-level errors which contain several complex errors, such as word order and lack of subject errors. several examples are presented in ## conclusion in this study, we incorporated a pre-trained model into an encoder-decoder model using two methods on chinese gec tasks. the experimental results demonstrate the usefulness of the bert-based pretrained model in the chinese gec task. additionally, our error type analysis showed that sentencelevel errors remain to be addressed.