Integrating BERT and Score-based Feature Gates for C hinese Grammatical Error Diagnosis (2020)

Abstract: This paper describes our proposed model for the Chinese Grammatical Error Diagnosis (CGED) task in NLPTEA2020. The goal of CGED is to use natural language processing techniques to automatically diagnose Chinese grammatical errors in sentences. To this end, we design and implement a CGED model named BERT with Score-feature Gates Error Diagnoser (BSGED), which is based on the BERT model, Bidirectional Long Short-Term Memory (BiLSTM) and conditional random field (CRF). In order to address the problem of losing partial-order relationships when embedding continuous feature items as with previous works, we propose a gating mechanism for integrating continuous feature items, which effectively retains the partial-order relationships between feature items. We perform LSTM processing on the encoding result of the BERT model, and further extract the sequence features. In the final test-set evaluation, we obtained the highest F1 score at the detection level and are among the top 3 F1 scores at the identification level.

## introduction recently, with the continuous development of china, more and more people have begun to learn chinese as their second language. due to the many complexities of chinese, such as the differences in how tenses are formed in chinese and english, many learners mistakenly write many chinese sentences with grammatical errors when they first learn chinese. therefore, it is necessary to develop a cged system, which can not only improve the learning efficiency of chinese learners, but also serve many downstream tasks based on chinese corpora. compared with english grammatical error diagnosis, chinese grammatical error correction has received limited interest in the research community. english grammar error detection models began being developed as early as the 1980s, such as the early writer's workbench system @xcite for detecting punctuation errors and style errors. later, a series of tasks for english grammatical error detection and correction were proposed, such as @xcite and @xcite . with the release of the cged task in the nlptea workshop in recent years, grammar diagnosis models for chinese have also begun to be developed. the goal of the cged task is to use natural language processing techniques to diagnose grammatical errors in chinese sentences written by learners who use chinese as a second language. the cged task allows researchers to exchange experiences and ultimately promote the development of this shared task. it defines four types of chinese grammatical errors, which are: redundant words (denoted as a capital "r"), missing words ("m"), word selection errors ("s"), and word ordering errors ("w"). the system developed for this task needs to identify the type and location of the errors in the input sentence. most recent solutions to the cged shared task convert the problem into a sequence labeling problem and use a bilstm-crf-based architecture as a basic framework to train the model. however, in previous work, feature engineering for the input sequence has become more and more complex. in addition, for some score-based features which exhibit partial-order relationships, such as the commonly used pmi score features, previous works usually learn their embedding matrix after discretizing the scores. through this process, the partial-order relationships between items will be lost, and the dimensionality of the feature embedding matrix will gradually increase as the granularity of the score discretization becomes finer, increasing the number of parameters needed to be trained. in response to the above problems, we design and implement bert with score-feature gates error diagnoser (bsged), and integrate score-based features through the use of a gating mechanism, which not only greatly reduces the workload of feature engineering, but also retains the original partial-order relationships for score-based features. experiments verify that the bsged model achieves excellent results with less feature engineering. in summary, our contributions are as follows: ## related work grammatical error diagnosis models appeared as early as the 1980s. early grammatical error diagnosis models used rule-based methods to check and correct grammatical errors @xcite . however, because the design of matching rules requires rich linguistic knowledge, it has become more and more difficult as well as time-consuming to design rules for such models. in order to deal with more complex error types, a series of grammatical error detection and correction models based on machine translation technology have been proposed. @xcite proposed a model that uses statistical machine tranalation (smt) techniques to detect and correct grammatical errors, which deal with mass/count noun confusions by translating the incorrect phrases as a whole. @xcite proposed a model for grammatical error diagnosis which combines rule-based and smt systems in a pipeline. the model first uses rules to detect errors and generate candidates. after the candidates are roughly screened by the n-gram language model, they are sent to the smt model for further screening. in the end, candidates will be further selected through language models and filtering rules. in order to solve the cged2018 shared task, @xcite proposed a sequence-to-sequence network to model the problem, and used a semi-supervised method to generate pseudo-grammatical error data for training the model. models based on machine translation require a large-scale training corpus to train the model. inspired by the powerful capabilities of neural machine translation (nmt) in grammatical error diagnosis, @xcite regarded cged as a sequence labeling problem, and used the powerful feature learning ability of an lstm network to model the input sequence, and achieved better results. @xcite incorporated more grammatical features into the model based on the bilstm-crf framework. based on the lstm-crf error detection model, @xcite combined three error correction models: a rule-based model, an nmt gec model, and an smt gec model. the three gec models aid the bilstm-crf model in marking possible error locations during the detection phase. @xcite designed a model that incorporates richer features and added a template matcher and probability fusion mechanism. ## conclusion and future work this paper describes our novel bsged model for the cged2020 shared task, which uses only a few and simple features, greatly reducing the workload of feature engineering for cged; a gating mechanism is also proposed to retain the original partialorder relationships between score-based features and at the same time reduce the amount of model training parameters. in addition, we connect the sequence encoding result of the bert model to the bilstm layer, which improves the bsged model's ability to capture long-term dependencies of the input sequence. bsged achieves the best f1 score at the detection level and the third highest f1 score at the identification level. in the future, we intend to use the mlm model to build a model that includes grammatical error correction, and apply the natural language generation capabilities of the pre-trained language model to the task of correcting chinese grammatical errors. in addition, we will also integrate more explicit grammatical rules, which will also greatly help the improvement of model performance.