C hinese Grammatical Errors Diagnosis System Based on BERT at NLPTEA -2020 CGED Shared Task (2020)

Abstract: In the process of learning Chinese, second language learners may have various grammatical errors due to the negative transfer of native language. This paper describes our submission to the NLPTEA 2020 shared task on CGED. We present a hybrid system that utilizes both detection and correction stages. The detection stage is a sequential labelling model based on BiLSTM-CRF and BERT contextual word representation. The correction stage is a hybrid model based on the n-gram and Seq2Seq. Without adding additional features and external data, the BERT contextual word representation can effectively improve the performance metrics of Chinese grammatical error detection and correction.

## introduction with the improvement of china's international status, more and more foreigners begin to learn chinese. unlike english, chinese grammar lacks morphology and singular and plural changes, and its sentence patterns are flexible and changeable. in learning chinese, foreigners are prone to introduce grammatical errors due to the complexity of chinese itself, the negative transfer of mother tongue and target language, and the cultural differences of different countries. in order to promote the development of automatic detection of syntactic errors in chinese writing, the natural language processing techniques for educational applications(nlp-tea) have taken cged as one of the shared tasks since 2014. thanks to the cged task, some research achievements have been made in chinese grammatical error detection. based on those previous research results, this paper puts forward a new thinking direction for the cged task. some typical examples are shown in this paper is organized as follows: section 2 describes some related works in english and chinese grammar error diagnosis. section 3 introduces the hybrid system that we proposed. section 4 shows the evaluation and discussion of our system. section 5 concludes the paper and discusses future work. ## related work the automatic diagnosis of grammatical errors is a topic of natural language processing. more research on the task of automatic grammatical error recognition focuses on english. in the 1960s, the study of automatic proofreading of english texts was carried out abroad. the hoo (helping our own) (2011) task related to grammatical errors in the task are all about english, which attracts many english grammatical errors researchers. researchers have proposed a variety of technologies suitable for automatic detection and correction of english grammatical errors, such as rule-based methods @xcite , phrase-based statistical methods @xcite , machine learning-based methods @xcite . however, there are few studies on grammatical errors in modern chinese. starting in 2014, the natural language processing techniques for educational applications (nlptea) has added modern chinese grammatical error recognition tasks. these evaluations the task provides a good platform for researchers to showcase their work, and it also speeds up the progress of modern chinese grammatical errors in automatic recognition methods. at different stages of the development of science and technology, the research methods of modern chinese grammatical error recognition are different, from rule-based to statistics-based, and then to deep learning-based methods. @xcite proposed a model based on stacked lstm and crf in 2016, which improved the accuracy and recall rate of automatic grammatical error recognition. in the 2017 ijcnlp-2017 cged evaluation, @xcite proposed a sequence labelling model based on bilstm-crf, which combines the establishment of parts of speech, n-gram grammar, and dependency features, and uses multiple model results to merge and delete after the last 20% of the results are merged, and the results are voted three different integration mechanisms, the effect of automatic grammatical error recognition has been dramatically improved in the f1 value of the three levels, @xcite in the 2018 nlptea-2018 cged evaluation task. based on the bilstm-crf model, it combines new features such as gaussian point-by-point mutual information, and adopts multiple model results for probabilistic integration and mixed multiple results ranking. two different integration mechanisms are introduced in the post-processing. gec is typically formulated as a sentence correction task. a gec system takes a potentially erroneous sentence as input and is expected to transform it into its corrected version. the conll-2014 shared task test set is the most widely used dataset to benchmark gec systems. the test set contains 1,312 english sentences with error annotations by two expert annotators. models are evaluated with the maxmatch scorer, which computes a span-based fβ-score. in the nlpcc2018-task2-cgec @xcite , the you dao team @xcite regards the error correction task as a translation task. errors are divided into surface errors and grammatical errors. the similar phonetic table and 5-gram language model are used to solve low-level errors, and the transformer model based on character granularity and word granularity are used to solve high-level errors. combine the low-level model and the high-level model and finally use the 5-gram language model to analyze the corrected sentence's perplexity and select the sentence with the lowest perplexity. the ali team @xcite adopts a multi-model parallel structure, using three types of models: rule-based, statistics-based, and neural network. first, the low-level combination, which includes one rule based model, two smt based models, and four nmt based models, obtains the category candidates, and then the high-level combination merges the candidates generated by the low-level combination. ## system description the system proposed in this paper contains two parts: the error detection stage and the error correction stage. the hybrid model presented in this paper is shown in figure 1 . first, in the error detection stage, we integrate the bert method and attention mechanism on the traditional bilstm-crf model. the input is a sequence of characters . the output is the dynamic word vector sequence , after encoding layer and decoding layer, we can get the label sequence .then, in the error correction stage, we perform 3-gram extraction based on the corrected sentence sequence of cged2016-2018, and construct a quadruple with frequency information. according to the results obtained by the detection stage, we will extract the label sequences containing m or s, merge the error-checking results with the rewrite results of seq2seq, and obtain the final result information . ## experiment results cged evaluation indicators include false positive rate, accuracy, precision, recall rate, f1 value, in order to evaluate the performance of the system at the four levels of grammatical errors. we also find out that our result didn't perform well in fpr. because the cged task belongs to the cost unequal experiment, we should try to increase the cost of marking the sentences with non-error type as error in the experiment instead of treating the cost as the same. the performance of our hybrid system is shown in the following tables comparing to the average of all 43 formal runs in 2020. ## conclusion aiming at the problems of the traditional models for automatic recognition of grammatical errors in chinese, such as the complex features and the large number of model integrations that are difficult to train, this paper proposes a bert-bilstm-attention-crf+correction model that combines the bert word vector and attention mechanism. compare it with the multi-feature bilstm-crf and crf models. the experimental results on the 2020 nlptea evaluation data set show that the bert-bilstm-attention-crf model performs better than other models we submitted, proving the superiority of bert word vectors in feature representation. on the basis of the model proposed in this article, comparing the effects of embedding different pre-trained word vectors on the recognition effect, and how to add a large amount of external knowledge to the recognition model to improve performance are issues worth exploring in our future work.