C hinese Grammatical Error Diagnosis Based on R o BERT a- B i LSTM - CRF Model (2020)

Abstract: Chinese Grammatical Error Diagnosis (CGED) is a natural language processing task for the NLPTEA6 workshop. The goal of this task is to automatically diagnose grammatical errors in Chinese sentences written by L2 learners. This paper proposes a RoBERTa-BiLSTM-CRF model to detect grammatical errors in sentences. Firstly, RoBERTa model is used to obtain word vectors. Secondly, word vectors are input into BiLSTM layer to learn context features. Last, CRF layer without hand-craft features work for processing the output by BiLSTM. The optimal global sequences are obtained according to state transition matrix of CRF and adjacent labels of training data. In experiments, the result of RoBERTa-CRF model and ERNIE-BiLSTM-CRF model are compared, and the impacts of parameters of the models and the testing datasets are analyzed. In terms of evaluation results, our recall score of RoBERTa-BiLSTM-CRF ranks fourth at the detection level.

## introduction the number of foreigners learning chinese is constantly increasing. some foreign countries even regard chinese as their second language. learners of chinese as a foreign language (cfl) may make grammatical errors in writing chinese. and the goal of chinese grammatical error diagnosis (cged) shared task is to develop nlp techniques to automatically diagnose grammatical errors in chinese sentences written by l2 learners. such errors fall into four categories: redundant words (denoted as a capital "r"), missing words ("m"), word selection errors ("s"), and word ordering errors ("w"). the criteria for judging correctness are determined at three levels as follows. (1) detection-level: to distinguish whether a sentence contains grammatical errors; (2) identificationlevel: to identify the types of those errors type; (3) position-level: to detect positions where errors occur. the quality of diagnosis is measured by fpr (false positive rate), pre (precision), rec (recall), and f1. cged shared task has been held since 2014 @xcite . in cged of nlp-tea 2018 @xcite , deep learning models are widely used, lstm-crf has been a standard implementation @xcite . while, in recent years, pre-training models, such as bert, xlnet, ernie @xcite and @xcite achieve good performance in various nlp tasks @xcite because of their fast convergence speed and less cost. this paper proposes a roberta-bilstm-crf model to detect grammatical errors. the model is described as follows: ## models we regard the cged task as a sequence labeling task. the illustrative graph of roberta-bilstm-crf is shown in figure 1 . chinese characters are input into roberta，and roberta converts each character into a one-dimensional vector. vector t1, t2, …tn fused with semantic features are output. the bilstm layer makes full use of the context information of the input sequence in the sequence labeling task so that it can predict label more accurately. the crf layer fully considers the context correlation when predicting the label. more importantly, the viterbi algorithm of crf uses the dynamic programming method to find the path with the highest probability. therefore, it fits better with the task of cged and avoids illegal sequences, such as 'b-r' tag followed by 'i-r' tag. ## dataset we collect training datasets of cged2016 (hsk) (lee et.al, 2016), cged2017 @xcite , cged2018, and cged2020 as training dataset and validation dataset, with a total of 21938 data units. the ratio of training dataset size to validation dataset size is about 8:2. we adopt the cged2018 testing dataset as our experimental testing dataset, with a total of 3549 data units. cged2020 testing dataset has a total of 1457 data units. table1 shows the number of data units, the number of errors grouped by error types in the training dataset, validation dataset, test2018, and test2020. we segment sentences into separate characters, and tag label for every character. label 'c' indicates correct character; 'b-x' indicates the beginning position for an error of type 'x' and 'i-x' shows the middle or ending position for an error of type 'x'. eight kinds of labels in our data: 'b-r', 'i-r', 'b-m', 'b-s', 'i-s', 'b-w', 'i-w', and 'c'. the sample of processed data is shown in ## conclusion and future work this paper proposes a roberta-bilstm-crf model to detect grammatical errors in the cged shared task. the results of experiments show roberta-bilstm-crf is a good model for detecting grammatical errors in general since roberta model obtains word vector according to data feature, and bilstm-crf captures sentencelevel features to predict and adjust labels. in the three runs submitted, our recall ranks fourth at detection-level in the cged shared task. in addition, we find that the performance of ernie-bilstm-crf is unreasonable on test2020 in our experiments, we will try to pursue reasons from model structure and characters of datasets in the future work.