Improving Autoregressive Grammatical Error Correction with Non-autoregressive Models (2023)

Abstract: Grammatical Error Correction (GEC) aims to correct grammatical errors in sentences. We find that autoregressive models tend to assign low probabilities to tokens that need corrections. Here we introduce additional signals to the training of GEC models so that these systems can learn to better predict at ambiguous positions. To do this, we use a non-autoregressive model as an auxiliary model, and develop a new regularization term of training by considering the difference in predictions between the autoregressive and non-autoregressive models. We experiment with this method on both English and Chinese GEC tasks. Experimental results show that our GEC system outperforms the baselines on all the data sets significantly.

## introduction grammatical error correction (gec) has attracted much attention in recent years, which aims to correct grammatical errors in a given text automatically. it is widely applied to natural language processing scenarios such as automatic speech recognition (asr) @xcite , writing assistant and language learning platforms, etc. the gec task is characterized by a significant overlap between input and output sentences with only a few errors requiring modification. since the transformer-based autoregressive @xcite (ar) model with sequence-to-sequence (seq2seq) architecture has been successful in many generation tasks, a few works @xcite have applied it to the gec task by taking the incorrect text as the source language and the text without errors as the target language, which has become a mainstream paradigm. however, in the gec task, the overlap of source and target sentences makes the ar model simply copy most of the tokens over from the input to the output. we further find that the ar has high confidence for the tokens that are unchanged between the source and target sentence, while it usually has low confidence for correcting operations such as insertion, deletion, and substitution. figure 1 is an example to illustrate this phenomenon. intuitively, we believe that the reasonable cause of this phenomenon is the class imbalance issue @xcite . with the influence of this problem, the ar model cannot confidently predict these incorrect tokens according to only the local context. therefore, a natural idea is to improve the model performance by exploiting the global information, which can be captured by the non-autoregressive (nar) @xcite model. although prior works have explored combining the two approaches through joint training, a combination for the gec task is still missing. besides, due to the inconsistency between ar and nar output, a simple combination of them will lead to poor performance. in this paper, we propose a simple yet novel approach to focus on incorrect tokens and integrate global information with the non-autoregressive model. specifically, by masking the tokens in the golden target sentence corresponding to the low confidence positions of the ar output, we construct the input for nar method. we combine the ar and nar generation mechanisms to effectively utilize global information by constraining the consistency of their output distribution. we conduct experiments on standard english gec datasets and evaluate the system against strong baselines. experimental results show that our approach can consistently achieve better results without relying on any resources other than the training data. furthermore, we compare with a combination method of ar and nar to verify whether the proposed model is more favorable for the gec task. here we use the chinese gec dataset as a benchmark to validate the generalization ability of the model. meanwhile, we also conduct comparative ablation studies to illustrate the effectiveness of our proposed method. ## methodology in this section, we elaborate on our proposed framework for gec. as shown in figure 2 , we introduce the cmlm-based nar model to integrate more context information into our single model. ## conclusion in this work, we propose a joint ar and nar learning objective for the gec, using a multi-task learning framework. to better predict tokens at low-confidence positions, we introduce additional signals to the training of gec models by using the nar model as an auxiliary model. meanwhile, we develop a new regularization term of training to constrain the inconsistency between the two manners. through our experiments in the english and chinese gec task, the proposed approach can significantly improve the gec model performance without additional inference costs. in the future, we are also interested in introducing syntax and lexical knowledge to focus on incorrect tokens to further improve performance.