Cross-lingual Transfer Learning for Grammatical Error Correction (2020)

Abstract: In this study, we explore cross-lingual transfer learning in grammatical error correction (GEC) tasks. Many languages lack the resources required to train GEC models. Cross-lingual transfer learning from high-resource languages (the source models) is effective for training models of low-resource languages (the target models) for various tasks. However, in GEC tasks, the possibility of transferring grammatical knowledge (e.g., grammatical functions) across languages is not evident. Therefore, we investigate cross-lingual transfer learning methods for GEC. Our results demonstrate that transfer learning from other languages can improve the accuracy of GEC. We also demonstrate that proximity to source languages has a significant impact on the accuracy of correcting certain types of errors.

## introduction grammatical error correction (gec) is the task of correcting grammatically incorrect sentences. the demand for gec has grown significantly in recent decades because of the increasing opportunities for cross-cultural collaboration. previous studies in the literature primarily focused on improving automated gec for the english language. thus, because of the large amount of data available for training, several machine-learning-based methods have achieved high scores in english gec @xcite @xcite . in recent years, researchers have started working on other languages, including russian and czech @xcite . however, for these languages, the language resources required to train the gec models accurately are not sufficiently available. it is known that using high-resource languages as the source languages can improve the accuracy of deep neural models for low-resource target languages in various settings @xcite @xcite . one such setting involves cross-lingual transfer learning @xcite , which aims to improve the accuracy of low-resource target models using knowledge from high-resource source models. the similarities between these languages is a key factor for successfully transferring grammatical knowledge @xcite . for example, languages within the same language family share several rules of grammar and nuances of vocabulary, which aid the learning process of the target models. however, thus far, no study has investigated the use of cross-lingual transfer learning for gec from other languages; therefore, it is unclear if useful grammatical knowledge (e.g., case inflection or conjugation) can be transferred. in this study, we investigate the following three research questions with respect to cross-lingual transfer learning for gec: (a) does cross-lingual transfer learning improve gec? to help answer this question, we compare the results of gec models trained with and without transfer learning. in addition, we compare several crosslingual transfer learning methods. our results indicate that using transfer learning from similar languages can improve the accuracy of gec for certain grammatical errors. in particular, we show that the error correction performance of similar lexical items is improved and that the transfer of grammatical knowledge is possible. additionally, we demonstrate that transfer learning is more effective for some types of errors than others, depending on the size of the target data. ## related work most recent gec methods use the encoder-decoder (encdec) model, which requires large-scale training data @xcite . therefore, several studies created additional pseudodata in low-resource scenarios @xcite . for example, because it is easy to generate a grammatically incorrect sentence from a grammatically correct sentence, extensive research has been conducted on generating pseudo-data from large-scale monolingual corpora @xcite . in addition, the use of encdec models pretrained with large-scale unlabeled data is known to be effective for gec @xcite . these studies aimed to improve the performance of gec using large-scale training data. furthermore, research has also been conducted on the use of linguistic knowledge from other languages in neural machine translation (nmt). @xcite proposed a method to fine-tune nmt models trained from high-resource language pairs on low-resource language pairs. @xcite demonstrated that a language can be translated with no training data by jointly training one model by concatenating the training data from multiple languages. @xcite presented a method that uses a bidirectional nmt encoder for cross-lingual contextual word representations, to generate dialog responses. for question answering, @xcite proposed a cross-lingual transfer learning method that uses generative adversarial networks. these studies focused on the tasks for which semantic information is more important, as opposed to gec, for which grammatical information is the key factor. various studies have analyzed the transfer of syntactic knowledge between languages. @xcite proposed a part-of-speech tagging method for learning language-independent and languagedependent expressions between languages by combining two models corresponding to the expressions. @xcite utilized adversarial training to train contextual encoders that produce invariant representations across languages, thereby facilitating cross-lingual transfers for dependency parsing. wu and dredze (2019) used multilingual bert for five tasks, such as pos tagging and dependency parsing, and demonstrated that its performance can be improved by using multilingual knowledge. as in our study, these studies perform transfer learning between languages in tasks for which syntactic information is important. however, it is not clear whether linguistic knowledge about grammatical errors can be transferred across languages. several gec studies using l1 information have been conducted. rozovskaya and roth (2011) adopted information from five l1s with different priorities to preposition correction using the naïve bayes classifier. @xcite extended this method to eleven l1s and three error types. @xcite demonstrated that using the same l1 for training and test data in an smt-based gec system improved the system performance. @xcite extended this method by incorporating three different l1 neural language models into an smt-based gec model as features to adapt to each l1. in these studies, gec was performed considering the l1 information; however, unlike our study, the objective of these studies did not include the transfer of grammatical knowledge between languages. ## conclusion in this study, we show that certain grammatical knowledge can be transferred across languages for gec. in particular, the correction performance of a model transferred from a similar language is greatly improved. additionally, we show that the performance improves more when correcting errors for similar grammatical items are corrected than when those for dissimilar grammatical items are corrected. we also show that cross-lingual transfer learning is effective for gec in both low-resource and high-resource languages to some extent. in the future, we plan to compare and visualize word embedding in models with and without transfer learning and investigate why mlm and tlm are effective for transfer learning in gec. we also plan to investigate the results of training and transferring multiple source languages instead of only one.