Combining R es N et and Transformer for C hinese Grammatical Error Diagnosis (2020)

Abstract: Grammatical error diagnosis is an important task in natural language processing. This paper introduces our system at NLPTEA-2020 Task: Chinese Grammatical Error Diagnosis (CGED). CGED aims to diagnose four types of grammatical errors which are missing words (M), redundant words (R), bad word selection (S) and disordered words (W). Our system is built on the model of multi-layer bidirectional transformer encoder and ResNet is integrated into the encoder to improve the performance. We also explore two ensemble strategies including weighted averaging and stepwise ensemble selection from libraries of models to improve the performance of single model. In official evaluation, our system obtains the highest F1 scores at identification level and position level. We also recommend error corrections for specific error types and achieve the second highest F1 score at correction level.

## introduction chinese language is commonly regarded as one of the most complicated languages. compared to english, chinese has neither singular/plural change, nor the tense changes of the verb. in addition, word segmentation usually has to be processed before deeper analysis, since word boundaries are not explicitly given in chinese. all these problems make chinese learning challenging to new learners. in recent years, more and more people with different language and knowledge background have become interested in learning chinese as a second language. it is necessary to develop an automatic chinese grammatical error diagnosis (cged) tool to help to identify and correct grammatical errors written by these people. in order to promote the development of automatic grammatical error diagnosis in chinese learning, the natural language processing techniques for educational applications (nlp-tea) have taken cged as one of the shared tasks since 2014. many methods have been proposed to solve cged task. in this work, we introduce our system at nlptea-2020 cged task. for error detection, our system is built on the model of multi-layer bidirectional transformer encoder and resnet is integrated into the encoder to improve the performance. we also explore stepwise ensemble selection from libraries of models to improve the performance of the single model. for error correction, we design two models to recommend corrections for s-type and m-type errors separately. more specifically, we use the roberta @xcite and the n-gram language model for the s-type correction, and utilize a combination of pretrained masked language model and a statistical language model to generate possible correction results for m-type correction. in official evaluation, our system obtains the highest f1 scores at identification level and position level for error detection, and the second-highest f1 score at correction level. the paper is organized as follows: section 2 briefly introduces the cged shared task. section 3 talks about our methodology. section 4 shows the experiment result. section 5 shows the related work. finally, the conclusion and future work are drawn in section 6. ## chinese grammatical error diagnosis the goal of nlptea cged task is to indicate errors in the sentences written by chinese foreign language learners. the sentences contain input embedding layer input embedding layer input embedding layer transformer block transformer block transformer block output layer output layer output layer … … … … … … … … encoder layer y (label) (a)bert input embedding layer input embedding layer input embedding layer transformer block transformer block transformer block output layer output layer output layer … … … … … … … … encoder layer (b)resbert transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block transformer block y (label) y (label) y (label) y (label) y (label) figure 1: architectures of bert and resbert for grammatical error detection, where "bert" means the multilayer bidirectional transformer encoder. four types of grammatical errors, including missing words (m), redundant words (r), word selection errors (s) and word ordering errors (w). the input sentence may contain one or more such errors. given a sentence, the system needs to indicate: (1) if the sentence is correct or not; (2) what kind of errors the sentence contains; (3) the exact error position; (4) possible corrections for s-type and m-type errors. some typical examples are shown in ## related work the researchers used many different methods to study the english grammatical error correction task and achieved good results @xcite . compared with english, the research time of chinese grammatical error diagnosis system is short, the data sets and effective methods are lacking. @xcite still used n-gram as the main method, and added web resources to improve detection performance. lin and chu (2015) established a scoring system using n-gram, and get better correction options. in recent years, chinese grammatical error diagnosis has been cited as a shared task of nlptea cged. many methods are proposed to solve this task @xcite @xcite . @xcite proposed a bilstm-crf model based on character embedding on bigram embedding. @xcite combined machine learning with traditional n-gram methods, using bi-lstm to detect the location of errors and adding additional linguistic information, pos, ngram. @xcite used bi-lstm to generate the probability of each characters, and used two strategies to decide whether a character is correct or not. @xcite used the lstm-crf model to detect dependencies between outputs to better detect error messages. @xcite added more linguistic information on lstm-crf model such as pos, n-gram, pmi score and dependency features. their system achieved the best f1-scores in identification level and position level on cged2017 task. @xcite added richer features on bilstm-crf model such as word segmentation, gaussian epmi, combination of pos and pmi. they also adopted a probabilistic ensemble approach to improve system performance. their system achieved the best f1-score in identification level and position level on cged2018 task. ## conclusion and future work the paper describes our system on nlptea-2020 cged task, which combines resnet and bert for chinese grammatical error diagnosis. we also design two different ensemble strategies to maximize the model's capability. at all six evaluating levels, we have the best f1 scores in identification level and position level, the second-highest f1 score in correction top1 level, the third-highest f1 score in detection level. in the future, we are planning to build a more powerful grammatical error diagnosis system with more training data and try to improve the system's ability by using the different cross-domain corpus.