Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction (2020)

Abstract: This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoder-decoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the fine-tuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-of-the-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/kanekomasahiro/bert-gec .

## introduction grammatical error correction (gec) is a sequenceto-sequence task where a model corrects an ungrammatical sentence to a grammatical sentence. numerous studies on gec have successfully used encoder-decoder (encdec) based models, and in fact, most current state-of-the-art neural gec models employ this architecture @xcite @xcite . in light of this trend, one natural, intriguing question is whether neural enddec gec models can benefit from the recent advances of masked language models (mlms) since mlms such as bert @xcite have been shown to yield substantial improvements in a variety of nlp tasks @xcite ). bert, for example, builds on the transformer architecture @xcite and is trained on large raw corpora to learn general representations of linguistic components (e.g., words and sentences) in context, which have been shown useful for various tasks. in recent years, mlms have been used not only for classification and sequence labeling tasks but also for language generation, where combining mlms with encdec models of a downstream task makes a noticeable improvement @xcite . common methods of incorporating a mlm to an encdec model are initialization (init) and fusion (fuse). in the init method, the downstream task model is initialized with the parameters of a pre-trained mlm and then is trained over a taskspecific training set @xcite . this approach, however, does not work well for tasks like sequence-to-sequence language generation tasks because such tasks tend to require a huge amount of task-specific training data and fine-tuning a mlm with such a large dataset tends to destruct its pre-trained representations leading to catastrophic forgetting @xcite . in the fuse method, pre-trained representations of a mlm are used as additional features during the training of a task-specific model @xcite . when applying this method for gec, what the mlm has learned in pre-training will be preserved; however, the mlm will not be adapted to either the gec task or the task-specific distribution of inputs (i.e., erroneous sentences in a learner corpus), which may hinder the gec model from effectively exploiting the potential of the mlm. given these drawbacks in the two common methods, it is not as straightforward to gain the advantages of mlms in gec as one might expect. this background motivates us to investigate how a mlm should be incorporated into an encdec gec model to maximize its bene-fit. to the best of our knowledge, no research has addressed this research question. in our investigation, we employ bert, which is a widely used mlm @xcite , and evaluate the following three methods: (a) initialize an encdec gec model using pre-trained bert as in @xcite (bert-init), (b) pass the output of pre-trained bert into the encdec gec model as additional features (bertfuse) @xcite , and (c) combine the best parts of (a) and (b). in this new method (c), we first fine-tune bert with the gec corpus and then use the output of the fine-tuned bert model as additional features in the gec model. to implement this, we further consider two options: (c1) additionally train pre-trained bert with gec corpora (bert-fuse mask), and (c2) fine-tune pre-trained bert by way of the grammatical error detection (ged) task (bert-fuse ged). in (c2), we expect that the gec model will be trained so that it can leverage both the representations learned from large general corpora (pre-trained bert) and the task-specific information useful for gec induced from the gec training data. our experiments show that using the output of the fine-tuned bert model as additional features in the gec model (method (c)) is the most effective way of using bert in most of the gec corpora that we used in the experiments. we also show that the performance of gec improves further by combining the bert-fuse mask and bertfuse ged methods. the best-performing model achieves state-of-the-art results on the bea-2019 and conll-2014 benchmarks. ## related work studies have reported that a mlm can improve the performance of gec when it is employed either as a re-ranker @xcite or as a filtering tool @xcite . encdec-based gec models combined with mlms can also be used in combination with these pipeline methods. @xcite proposed sequence labeling models based on correction methods. our method can utilize the existing encdec gec knowledge, but these methods cannot be utilized due to the different architecture of the model. besides, to the best of our knowledge, no research has yet been conducted that incorporates information of mlms for effectively training the encdec gec model. mlms are generally used in downstream tasks by fine-tuning @xcite , however, @xcite demonstrated that it is more effective to provide the output of the final layer of a mlm to the encdec model as contextual embeddings. recently, @xcite addressed the mismatch problem between contextual knowledge from pre-trained models and the target bilingual machine translation. here, we also claim that addressing the gap between grammatically correct raw corpora and gec corpora can lead to the improvement of gec systems. ## methods for using pre-trained mlm in gec model in this section, we describe our approaches for incorporating a pre-trained mlm into our gec model. specifically, we chose the following approaches: (1) initializing a gec model using bert; ## results also in this setting, bert-fuse ged outperformed other models in all cases except for the fce-test set, thus, achieving state-of-the-art results with a single model on the bea2019 and conll14 datasets. in the last row, the ensemble model yielded high scores on all corpora, improving state-of-the-art results by 0.2 points in conll14. ## analysis 6.1 hidden representation visualization we investigate the characteristics of the hidden representations of vanilla (i.e., without any finetuning) bert and bert fine-tuned with ged. we visualize the hidden representations of the same words from the last layer of bert h l . they were chosen depending on correctness in a different context, using the above models. these target eight words foot_6 that have been mistaken more than 50 times, were chosen from w&i-dev. we sampled the same number of correctly used cases for the same word from the corrected side of w&i-dev. figure 1 visualizes hidden representations of bert and fine-tuned bert. it can be seen that the vanilla bert does not distinguish between correct and incorrect clusters. the plotted eight words are gathered together, and it can be seen that hidden representations of the same word gather in the same place regardless of correctness. on the other hand, fine-tuned bert produces a vector space that demonstrates correct and incorrect words on different sides, showing that hidden representations take grammatical errors into account when fine-tuned on gec corpora. moreover, it can be seen that the correct cases divided into 8 clusters, implying that bert's information is also retained. ## conclusion in this paper, we investigated how to effectively use mlms for training gec models. our results show that bert-fuse ged was one of the most effective techniques when it was fine-tuned with gec corpora. in future work, we will investigate whether bert-init can be used effectively by using methods to deal with catastrophic forgetting.