BERT Enhanced Neural Machine Translation and Sequence Tagging Model for C hinese Grammatical Error Diagnosis (2020)

Abstract: This paper presents the UNIPUS-Flaubert team’s hybrid system for the NLPTEA 2020 shared task of Chinese Grammatical Error Diagnosis (CGED). As a challenging NLP task, CGED has attracted increasing attention recently and has not yet fully benefited from the powerful pre-trained BERT-based models. We explore this by experimenting with three types of models. The position-tagging models and correction-tagging models are sequence tagging models fine-tuned on pre-trained BERT-based models, where the former focuses on detecting, positioning and classifying errors, and the latter aims at correcting errors. We also utilize rich representations from BERT-based models by transferring the BERT-fused models to the correction task, and further improve the performance by pre-training on a vast size of unsupervised synthetic data. To the best of our knowledge, we are the first to introduce and transfer the BERT-fused NMT model and sequence tagging model into the Chinese Grammatical Error Correction field. Our work achieved the second highest F1 score at the detecting errors, the best F1 score at correction top1 subtask and the second highest F1 score at correction top3 subtask.

## introduction recently, the pre-trained language models such as bert @xcite obtain state-of-the-art results on a wide range of natural language processing (nlp) tasks, such as text classification, reading comprehension, machine translation @xcite , etc. the english grammatical error correction (gec) task also benefits from the pretrained language models. for example, in the work of @xcite , they not only follow @xcite to incorporate bert into an encoder-decoder model for gec, but also maximize the benefit by additionally training bert on gec corpora (bert-fuse mask) or fine-tuning bert as a ged model (bert-fuse ged). another route to improve the performance of gec is using bert as an encoder and incorporating it into a sequence tagging model @xcite @xcite . in the chinese nlp community, a variety of pre-trained chinese language models have been proposed and publicly available @xcite @xcite . those models are proved to have a significant improvement in a variety of down-stream tasks, including reading comprehension, natural language inference, sentiment classification, etc. in this paper, we apply the state-of-the-art english gec models to the cged task. our cged system consists of three types of models. we propose the position-tagging model, which is a sequence tagging model with a bert encoder, to concentrate on the error localization task. the output label consists of 8 types of tags and indicates the start and end of each error for the input sentence, but it will not tell us how to correct it in the case of s (word selection) and m (missing word) errors. the correction-tagging model @xcite @xcite concentrates on the error correction task, and the output label contains 8772 types of tags. the tags reveal the editing operations for each chinese character, e.g. keep, delete, append, and replace. the append tags (3788 in total) and replace tags (4982 in total) cover most chinese characters. the bert-fused model @xcite ) is proposed for neural machine translation (nmt) task and adaptively controls the interaction between representations from bert and each layer of the transformer @xcite by using the attention mechanism. @xcite transfers the bert-fused model to the english gec task and further advances it. due to time limitations, we only follow the training settings in @xcite . besides, we perform unsupervised data augmentation by introducing synthetic errors on a large amount of error-free corpora, then pair synthetic and original sentences to pre-train transformers @xcite . this paper is organized as follows: section 2 summarizes the recent developments in the field of cged. section 3 introduces the dataset we used to train the models, including human-annotated data and synthetic data. section 4 is the overview of each component of our system, including bertfused nmt, position-tagging model, correctiontagging model, and error annotation tool. section 5 describes our training and ensemble process. section 6 discusses the result of our models and section 7 concludes the paper. ## datasets training data the datasets of the nlptea 2014∼2018 & 2020 shared task of cged are corpora composed of parallel sentences written by chinese as a foreign language (cfl) learners and their corrections. the source sentences are selected from the essay section of the computer-based tocfl (test of chinese as a foreign language) and written-based hsk (pinyin of hanyu shuiping kaoshi, test of chinese level). before 2016, there are only tocfl data written in traditional chinese. in the dataset of 2016, we have both tocfl and hsk data. we use the opencc foot_0 package to convert the traditional chinese to simplified chinese for the tocfl corpus. since 2017, only hsk data are provided that are all written in simplified chinese. the grammatical errors were manually annotated by native chinese speakers. there are four kinds of errors: r (redundant word), m (missing word), s (word selection error), and w (word ordering error). each error type has a different proportion in the corpus and each sentence may contain several errors. for example, in the cged 2020 training set, w/s/r/m accounted for 7%, 42%, 23%, 28% of the total errors respectively. there are 2909 manually annotated errors in 1641 sentences, and only 2 sentences are error-free. we also collect several external datasets from nlpcc 2018 gec foot_1 and other resources foot_2 . the nlpcc 2018 gec data contains more than 700,000 sentences and each sentence may be correct or have one or more candidate corrections. ## system overview our system consists of a sequence labeling model concentrated on the error detection subtask, and two types of error correction models aimed at generating candidate corrections. data set # sent pairs source pt 61.0 m wiki2019zh, news2016zh ma 1.17 m cged 2016∼2018 train × 5 , hsk, nlpcc cged (2014 train + test, 2015 train +test)×3, ama 5.35 m (2016 train + 2017 train)×10, 2018 train×20, hsk × 2, nlpcc × 3, webtext2019zh, sogouca ## discussion for the bert-fused nmt models, the bert-fused stage improves the f1 scores for both non-pretraining and pre-training mode (see in the ensemble modeling, we found that fpr (false positive rate) decreased as the threshold in the voting stage increased. our submissions did not rank high in the fpr subtask, since we focused on the detection and correction rather than the fpr subtask. compared to the methods proposed in the nlptea 2018 shared task of cged, our system greatly improves the f1 score on correction top1 and correction top3 subtask on the cged 2018 test set. this advance mainly comes from: (1) we not only fully exploit the transformer model for the correction subtask, but also comprehensively incorporate the power of pre-trained bert-based models into every subtask of the cged task; (2) the low-resource problem in the gec task restricts the performance of nmt models @xcite , and we address this by utilizing the power of pre-trained bert models and synthesizing extensive artificial data. ## conclusion in this work, we present our solutions to the nlptea 2020 shared task of cged. three kinds of models are used in our system: position-tagging models, bert-fused nmt models and correctiontagging models. our hybrid system achieved the second-highest f1 score in the detection subtask, the highest f1 score in the correction top1 subtask and the second-highest f1 score in the correction top3 subtask, which shows that the cged task can benefit from the recent advances of pre-trained language models.