TMU - NLP System Using BERT -based Pre-trained Model to the NLP - TEA CGED Shared Task 2020 (2020)

Abstract: In this paper, we introduce our system for NLPTEA 2020 shared task of Chinese Grammatical Error Diagnosis (CGED). In recent years, pre-trained models have been extensively studied, and several downstream tasks have benefited from their utilization. In this study, we treat the grammar error diagnosis (GED) task as a grammatical error correction (GEC) problem and propose a method that incorporates a pre-trained model into an encoder-decoder model to solve this problem.

## introduction in the cged task, given a source sentence contains grammatical errors, systems are needed to output four kinds of results: detection level, identification level, position level, and correction level. we treat the cged task as a gec problem, so our system will output the corrected sentence directly. then we use a post-processing method to generate detection, identification, and position level results. gec can be regarded as a sequence-to-sequence task. gec systems receive an erroneous sentence written by a language learner and output the corrected sentence. in previous studies that adopted neural models for chinese gec @xcite , the performance was improved by initializing the models with a distributed word representation, such as word2vec @xcite . however, in these methods, only the embedding layer of a pre-trained model was used to initialize the models. in addition, chinese gec remains challenging because chinese is a complex language. for example, there are more than 10,000 chinese characters, and the glyphs or pronunciation of several chinese characters are similar. therefore, character errors introduced by chinese learners can be variable; further, for the chinese gec tasks, it is necessary to utilize a pre-trained model, which captures additional information about a language. in recent years, pre-trained models based on bidirectional encoder representations from transformers (bert) have been studied extensively @xcite , and the performance of many downstream natural language processing (nlp) tasks has been dramatically improved by utilizing these pre-trained models. in order to learn existing knowledge of a language, a bert-based pre-trained model is trained on a large-scale corpus using the encoder of transformer @xcite . subsequently, for a downstream task, a neural network model is initialized with the weights learned by a pre-trained model that has the same structure and is fine-tuned on training data of the downstream task. the performance is expected to improve by using this two-stage method because downstream tasks are informed by the knowledge learned by the pre-trained model. in this study, as shown in figure 1 , we develop a chinese gec model based on transformer by initializing the encoder of transformer with a chinese bert-based pre-trained model and then finetuning the correction model on chinese gec and ged corpus. our system achieves f 1 scores of 82.00, 52.66, 22.24, 14.17 on detection level, iden-tification level, position level and correction level evaluation. ## related work in building educational applications (bea) 2019 @xcite , several teams attempted to incorporate bert into their correction models. @xcite first fine-tuned bert on a learner corpus and then incorporated the word probability provided by bert into re-ranking features. using bert for re-ranking features, they obtained an approximately 0.7 point improvement of the f 0.5 score. @xcite used bert to solve the gec task by iteratively querying bert as a black box language model. they added a [mask] token into source sentences and predicted the word represented by the [mask] token. if the word probability predicted by bert exceeded the threshold, the word was output as a correction candidate. using bert, they obtained a 0.27 point improvement of the f 0.5 score. these studies show that bert helps improve the performance of a correction model; however, this improvement was marginal, and they did not explore the use of pre-trained models for weight initialization. ## method we adopt the method from @xcite to construct our correction model. additional details are introduced in the following sections. ## conclusion in this study, we initialized the encoder of the transformer with a chinese pre-trained model and used this system to challenge the cged task.