Tail-to-Tail Non-Autoregressive Sequence Prediction for C hinese Grammatical Error Correction (2021)

Abstract: We investigate the problem of Chinese Grammatical Error Correction (CGEC) and present a new framework named Tail-to-Tail ( TtT ) non-autoregressive sequence prediction to address the deep issues hidden in CGEC. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected based on the bidirectional context information, thus we employ a BERT-initialized Transformer Encoder as the backbone model to conduct information modeling and conveying. Considering that only relying on the same position substitution cannot handle the variable-length correction cases, various operations such substitution, deletion, insertion, and local paraphrasing are required jointly. Therefore, a Conditional Random Fields (CRF) layer is stacked on the up tail to conduct non-autoregressive sequence prediction by modeling the token dependencies. Since most tokens are correct and easily to be predicted/conveyed to the target, then the models may suffer from a severe class imbalance issue. To alleviate this problem, focal loss penalty strategies are integrated into the loss functions. Moreover, besides the typical fix-length error correction datasets, we also construct a variable-length corpus to conduct experiments. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure on tasks of error Detection and Correction.

## introduction grammatical error correction (gec) aims to automatically detect and correct the grammatical errors that can be found in a sentence @xcite . it is a crucial and essential application task in many natural language processing scenarios such as writing assistant @xcite @xcite , search engine @xcite @xcite , speech recognition systems @xcite @xcite , etc. grammatical errors may appear in all languages @xcite @xcite @xcite , in this paper, we only focus to tackle the problem of chinese grammatical error correction (cgec) @xcite . we investigate the problem of cgec and the related corpora from @xcite and nlpcc @xcite carefully, and we conclude that the grammatical error types as well as the corresponding correction operations can be categorised into three folds, as shown in figure 1: (1) substitution. in reality, pinyin is the most popular input method used for chinese writings. thus, the homophonous character confusion (for example, in the case of type i, the pronunciation of the wrong and correct words are both "feichang") is the fundamental reason which causes grammatical errors (or spelling errors) and can be corrected by substitution operations without changing the whole sequence structure (e.g., length). thus, substitution is a fixed-length (fixlen) operation. (2) deletion and insertion. these two operations are used to handle the cases of word redundancies and omissions respectively. (3) local paraphrasing. sometimes, light operations such as substitution, deletion, and insertion cannot correct the errors directly, therefore, a slightly subsequence paraphrasing is required to reorder partial words of the sentence, the case is shown in type iii of figure 1 . deletion, insertion, and local paraphrasing can be regarded as variable-length (varlen) operations because they may change the sentence length. however, over the past few years, although a number of methods have been developed to deal with the problem of cgec, some crucial and essential aspects are still uncovered. generally, sequence translation and sequence tagging are the two most typical technical paradigms to tackle the problem of cgec. benefiting from the development of neural machine translation @xcite , attention-based seq2seq encoder-decoder frameworks have been introduced to address the cgec problem in a sequence translation manner @xcite @xcite @xcite . seq2seq based translation models are easily to be trained and can handle all types of correcting operations above mentioned. however, considering the exposure bias issue @xcite , the generated results usually suffer from the phenomenon of hallucination @xcite and cannot be faithful to the source text, even though copy mechanisms @xcite are incorporated @xcite . therefore, @xcite and @xcite propose to purely employ tagging to conduct the problem of gec instead of generation. all correcting operations such as deletion, insertion, and substitution can be guided by the predicted tags. nevertheless, the pure tagging strategy requires to extend the vocabulary v to about three times by adding "insertion-" and "substitution-" prefixes to the original tokens (e.g., "insertion-good", "substitutionpaper") which decrease the computing efficiency dramatically. moreover, the pure tagging framework needs to conduct multi-pass prediction until no more operations are predicted, which is inefficient and less elegant. recently, many researchers fine-tune the pre-trained language models such as bert on the task of cgec and obtain reasonable results @xcite @xcite . however, limited by the bert framework, most of them can only address the fixed-length correcting scenarios and cannot conduct deletion, insertion, and local paraphrasing operations flexibly. moreover, during the investigations, we also observe an obvious but crucial phenomenon for cgec that most words in a sentence are correct and need not to be changed. this phenomenon is depicted in figure 2 , where the operation flow is from the bottom tail to the up tail. grey dash lines represent the "keep" operations and the red solid lines indicate those three types of correcting operations mentioned above. on one side, intuitively, the target cgec model should have the ability of directly moving the correct tokens from bottom tail to up tail, then transformer @xcite based encoder (say bert) seems to be a preference. on the other side, considering that almost all typical cgec models are built based on the paradigms of sequence tagging or sequence translation, maximum likelihood estimation (mle) @xcite is usually used as the parameter learning approach, which in the scenario of cgec, will suffer from a severe class/tag imbalance issue. however, no previous works investigate this problem thoroughly on the task of cgec. to conquer all above-mentioned challenges, we propose a new framework named tail-to-tail non-<@xmath0, which abbreviated as ttt, for the problem of cgec. specifically, to directly move the token information from the bottom tail to the up tail, a bert based sequence encoder is introduced to conduct bidirectional representation learning. in order to conduct substitution, deletion, insertion, and local paraphrasing simultaneously, inspired by @xcite , a conditional random fields (crf) @xcite layer is stacked on the up tail to conduct nonautoregressive sequence prediction by modeling the dependencies among neighbour tokens. focal loss penalty strategy @xcite ) is adopted to alleviate the class imbalance problem considering that most of the tokens in a sentence are not changed. in summary, our contributions are as follows: