Grammatical Error Correction Using Pseudo Learner Corpus Considering Learner’s Error Tendency (2020)

Abstract: Recently, several studies have focused on improving the performance of grammatical error correction (GEC) tasks using pseudo data. However, a large amount of pseudo data are required to train an accurate GEC model. To address the limitations of language and computational resources, we assume that introducing pseudo errors into sentences similar to those written by the language learners is more efficient, rather than incorporating random pseudo errors into monolingual data. In this regard, we study the effect of pseudo data on GEC task performance using two approaches. First, we extract sentences that are similar to the learners’ sentences from monolingual data. Second, we generate realistic pseudo errors by considering error types that learners often make. Based on our comparative results, we observe that F0.5 scores for the Russian GEC task are significantly improved.

## introduction recently, several studies have proposed models to solve grammatical error correction (gec) task as an application of writing support for language learners of various languages, such as english or russian. a standard approach to improve gec models is to incorporate pseudo errors into large monolingual datasets for pretraining. in particular, previous works achieved state-of-the-art performance by pre-training the model using pseudo data with a subsequent finetuning of the pre-trained model using a learner corpus @xcite @xcite @xcite . considering the aforementioned approach, several methods have been proposed for the generation of pseudo data for pre-training a gec model. in theory, it is possible to include all types of errors in a dataset via random error generation. however, considering the limitations of computational resources required to train a gec model using large pseudo datasets, there is a need to generate pseudo datasets with only realistic errors. thus, in this study, we generate pseudo data to train gec models considering the types of errors made by language learners and study the effect of this realistic pseudo training data. first, we extract sentences similar to the training data from monolingual datasets to generate pseudo data for pretraining. second, we analyze the error tendency of learners and add pseudo errors considering the errors learners tend to make in english and russian languages. through experiments, we show that the proposed pseudo data generation method improves the f 0.5 scores of the gec model. in summary, the primary contributions of this study are as follows: ## related works pseudo data have been generated for gec tasks in several previous works. @xcite generated pseudo data by adding randomly generated pseudo errors, in an error-free sentence. in particular, in this approach, randomly selected words were replaced or deleted from a large monolingual dataset. in addition, a random word was inserted into sentences, and words in a sentence were swapped around. a similar approach was proposed by @xcite , where an original word is masked or retained to generate pseudo data for pre-training. however, both of these methods generate errors that are not similar to the real errors made by language learners. the data in moreover, grundkiewicz and junczys-dowmunt (2019) developed a confusion set that retained out-of-vocabulary words and preserved consistent letter casing. however, using this approach, unrealistic errors might be included in the pseudo data because it primarily considers the surface of words. further, @xcite conducted a gec experiment in multiple lan- the to cost total at … a government … original sent. erroneous sent. the for cost total at … the governments … preposition error noun number error about, by, for, in, from, of, with, on, at article error the, [no article] guages, such as english, russian, german, and czech, and proposed a pseudo error generation model for czech, considering errors in diacritics. in the present study, we incorporate the most common error types in monolingual data based on language-specific prior knowledge to obtain development data. ## method for pseudo data generation first, we describe the method for pseudo data generation that considers learner error types. subsequently, we use the generated pseudo data for pretraining a gec model. in this study, we combine the proposed method of pseudo data generation with previous methods. in particular, we incorporate the basic random approach (deletion, insertion, swapping) in our approach, as well as the more recent sophisticated approach proposed by @xcite (character level perturb, confusion set based on an unsupervised spellchecker). ## conclusions in this study, we studied the effect of pseudo data obtained using two approaches. in particular, we confirmed that combining data selection and realistic error injection approaches to obtain pseudo data improved the f 0.5 scores. moreover, we analyzed the recall for each error type. based on our experimental results, we observed that the recall for error types considered in our study improved or were comparable.