Comparison of Grammatical Error Correction Using Back-Translation Models (2021)

Abstract: Grammatical error correction (GEC) suffers from a lack of sufficient parallel data. Studies on GEC have proposed several methods to generate pseudo data, which comprise pairs of grammatical and artificially produced ungrammatical sentences. Currently, a mainstream approach to generate pseudo data is back-translation (BT). Most previous studies using BT have employed the same architecture for both the GEC and BT models. However, GEC models have different correction tendencies depending on the architecture of their models. Thus, in this study, we compare the correction tendencies of GEC models trained on pseudo data generated by three BT models with different architectures, namely, Transformer, CNN, and LSTM. The results confirm that the correction tendencies for each error type are different for every BT model. In addition, we investigate the correction tendencies when using a combination of pseudo data generated by different BT models. As a result, we find that the combination of different BT models improves or interpolates the performance of each error type compared with using a single BT model with different seeds.

## introduction grammatical error correction (gec) aims to automatically correct errors in text written by language learners. it is generally considered as a translation from ungrammatical sentences to grammatical sentences, and gec studies use machine translation (mt) models as gec models. after @xcite applied an encoder-decoder (encdec) model @xcite to gec, various encdec-based gec models have been proposed @xcite @xcite @xcite . gec models have different correction tendencies in each architecture. for example, a gec model based on cnn @xcite tends to correct errors effectively using the local context @xcite . furthermore, some studies have combined multiple gec models to exploit the difference in correction tendencies, thereby improving performance @xcite . despite their success, encdec-based models require considerable amounts of parallel data for training @xcite . however, gec suffers from a lack of sufficient parallel data. accordingly, gec studies have developed various pseudo data generation methods @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . moreover, @xcite showed that the correction tendencies of the gec model are different when using (1) a pseudo data generation method by adding noise to latent representations and (2) a rule-based pseudo data generation method. furthermore, they improved the gec model by combining pseudo data generated by these methods. therefore, the combination of pseudo data generated by multiple methods with different tendencies allows us to improve the gec model further. one of the most common methods to generate pseudo data is back-translation (bt) @xcite . in bt, we train a bt model (i.e., the reverse model of the gec model), which generates an ungrammatical sentence from a given grammatical sentence. subsequently, a grammatical sentence is provided as an input to the bt model, generating a sentence containing pseudo errors. finally, pairs of erroneous sentences and their input sentences are used as pseudo data to train a gec model. @xcite reported that a gec model using bt achieved the best performance among other pseudo data generation methods. however, most previous gec studies using bt have used the bt model with the same architecture as the gec model @xcite @xcite @xcite . thus, it is unclear whether the correction tendencies differ when using bt models with different architectures. we investigated correction tendencies of the gec model using pseudo data generated by different bt models. specifically, we used three bt models: transformer @xcite , cnn @xcite , and lstm @xcite . the results showed that correction tendencies of each error type are different for each bt model. in addition, we examined correction tendencies of the gec model when using a combination of pseudo data generated by different bt models. as a result, we found that the combination of different bt models improves or interpolates the f 0.5 scores of each error type compared with that of single bt models with different seeds. the main contributions of this study are as follows: htut and tetreault (2019) compared four gec models-transformer, cnn, prpn @xcite , and on-lstm @xcite -using pseudo data generated by different bt models. specifically, they used transformer and cnn as bt models. it was reported that the transformer using pseudo data generated by cnn achieved the best f 0.5 score. however, the correction tendencies for each bt model were not reported. moreover, although using pseudo data for pre-training is common in gec @xcite @xcite @xcite , they used a less common method of utilizing pseudo data for re-training after training with genuine parallel data. therefore, we used transformer as the gec model and investigated correction tendencies when using transformer, cnn, and lstm as bt models. further, we used pseudo data to pre-train the gec model. ## discussion we examined the number of edit pairs in pseudo data generated by each bt model. we annotated pseudo data using errant and extracted edit pairs from the pseudo source sentences and target sentences. ## conclusions in this study, we investigated correction tendencies based on each bt model. the results showed that the correction tendencies of each error type varied depending on the bt models. in addition, we found that the combination of different bt models improves or interpolates the f 0.5 score compared with that of single bt models with different seeds.