T rans GEC : Improving Grammatical Error Correction with Translationese (2023)

Abstract: Data augmentation is an effective way to improve model performance of grammatical error correction (GEC). This paper identifies a critical side-effect of GEC data augmentation, which is due to the style discrepancy between the data used in GEC tasks (i.e., texts produced by non-native speakers) and data augmentation (i.e., native texts). To alleviate this issue, we propose to use an alternative data source, translationese (i.e., human-translated texts), as input for GEC data augmentation, which 1) is easier to obtain and usually has better quality than non-native texts, and 2) has a more similar style to non-native texts. Experimental results on the CoNLL14 and BEA19 English, NLPCC18 Chinese, Falko-MERLIN German, and RULEC-GEC Russian GEC benchmarks show that our approach consistently improves correction accuracy over strong baselines. Further analyses reveal that our approach is helpful for overcoming mainstream correction difficulties such as the corrections of frequent words, missing words, and substitution errors. Data, code, models and scripts are freely available at https://github.com/NLP2CT/TransGEC .

## introduction grammatical error correction (gec) is a task of automatically correcting an ungrammatical sentence into a corrected version. training gec models highly relies on labeled data (i.e., ungrammatical sentences to their grammatical ones), but such resources are scarce and expensive to construct. data augmentation, which exploits a large amount of unlabeled data for performance improvement, is a popular research line of gec @xcite @xcite . however, there is a stylistic discrepancy between the data used for gec tasks and data augmentation. for most gec tasks @xcite , their training and * co-corresponding author testing instances are produced by non-native speakers, whereas the data used for augmentation are mainly native language resources @xcite @xcite . @xcite have shown a large difference between non-native and native texts, which means that style mismatch might be a side-effect limiting the further enhancement of gec data augmentation. a more ideal way is to directly use non-native texts as input for data augmentation. however, such resources are very few, and their quality is hard to be guaranteed. in this paper, we propose the transgec method which uses human-translated texts (aka translationese) as input for augmentation. improving gec with translationese has the following advantages: 1) easy-to-obtain, the training corpus of machine translation tasks consists of abundant translationese, and its identification has been well studied @xcite ; 2) similar style, non-native texts and translationese are closer to each other than native texts @xcite ; and 3) high quality, most translationese is produced by bilingual experts, whose quality can be better guaranteed than the majority of non-native texts. preliminary experiments on the comparison of different kinds of texts confirm our assumption that translationese indeed has a similar style to gec data. this enables us to further explore translationese for gec in two steps: 1) obtaining translationese, we propose to fine-tune bert-based classifiers to identify translationese from the parallel corpora (e.g., wmt corpus) of machine translation tasks; and 2) improving gec with translationese, we propose to add artificial noise to the identified translationese, and treat the noisy/corrected version as the input/output for training gec models. experimental results on the widely-used conll14 and bea19 english, nlpcc18 chinese, falko-merlin german, and rulec-gec russian gec benchmarks show that transgec outperforms strong (m)t5-large pre-trained model @xcite , lrgec baselines @xcite , and existing data augmentation methods @xcite . further analyses show that transgec improves the correction accuracy of major difficulties (e.g., correction of frequent words, missing words, and substitution errors), but still has room for improvement in minor issues (e.g., correction of rare words, word order, and deletion errors). our main contributions are summarized as: another research line uses pre-trained language models to improve the model performance of gec. @xcite extract external knowledge from language models for gec training, @xcite further treat the language models as a part of the network for gec training. all the above work has a potential limitation: while the training and test data of gec tasks are produced by nonnative speakers, the data used for augmentation or pre-training are mainly native texts. this style discrepancy is a threat to gec data augmentation. @xcite ; @xcite propose to use machine-translated text for gec data augmentation, but their intuition is not using the text with a similar style but producing noisy text through machine translation. our approach focuses on the style mismatch problem by introducing translationese (human-translated texts) as input for data augmentation, providing a reasonable explanation for their model improvements. translationese refers to the presence of unusual properties of human-translated texts and thus becomes an alternate name for such texts. a reason might be that translators are affected by the style of the source language and ignore the rules of the target language during translation @xcite . translationese tends to show less lexical diversity compared to native texts @xcite . @xcite point out that there are many common idioms unconsciously used in native texts. @xcite and @xcite report that translationese has some unique characteristics, e.g., simplification, explicitation and normalization. @xcite provide a systematic study and find that the non-native texts and translationese are closer to each other than to native texts. a research line discusses the effect of translationese in machine translation tasks since translationese widely exists in parallel corpora. @xcite reveal the side-effect of using translationese in machine translation evaluation and recommend only evaluating native texts. @xcite demonstrate that translationese hinders the model from generating more adequate and fluent translations. another line focuses on identifying translationese from parallel sentences to control the training of downstream tasks. @xcite propose a support vector machine-based classifier to identify translationese while @xcite use a convolution neural network-based classifier. @xcite train a classifier that distinguishes between native and translationese based on significant differences in their text content. to the best of our knowledge, the discussion and application of translationese has not yet been introduced to gec tasks. this paper takes the first step into using translationese for improving gec. represent our reproduced results based on the released english data by @xcite and the chinese data by @xcite and @xcite . native (ours) refers to the results based on our collected native text (i.e., the wmt news crawl data for english and the people's daily data for chinese), and gec refers to the results of the original gec data (i.e., conll14 english and nlpcc18 chinese benchmarks). the vertical axis represents the normalized statistical results for each linguistic property, where a higher value indicates a greater proportion of linguistic properties. the style of translationese is similar to that of original gec data. we first explain why gec models need other kinds of alternatives as input for data augmentation, and then give preliminary experiments and results to show that translationese can be a decent alternative. motivation the performance of gec systems highly depends on the quality and quantity of annotated training data (i.e., ungrammatical sentences and their grammatical version). due to the high cost of collecting such data, the research of data augmentation techniques (i.e., utilizing unlabeled data) for gec has become a popular topic. by looking at the most widely-used gec benchmark -conll14 @xcite and bea19 @xcite shared tasks, the training corpora includes nus corpus of learner english (nu-cle) @xcite , lang-8 corpus @xcite , fce v2.1 @xcite and w&i @xcite , all of which are produced by non-native language learners. however, existing methods directly use native texts as input for data augmentation for gec tasks. for example, @xcite and @xcite use wikipedia data, while @xcite and @xcite utilize one billion word benchmark @xcite data. previous studies have validated that there exists a style gap between native and non-native texts @xcite . we argue that such gap brings a side-effect to model performance, limiting the further improvements of gec data augmentation. utilizing non-native texts might be a better choice, however, there exist few non-native text resources and it is not easy to collect the text from scratch and guarantee their quality. this motivates us to find some other alternatives, which are easyto-obtain, high-quality, and with a closer style to the non-native text of gec tasks. preliminary experiments @xcite have shown that non-native texts and translationese are closer to each other than each of them to native texts. motivated by them, in this experiment, we explore the similarities between gec data and translationese on the english and chinese gec tasks. we compare our collected native texts and gec data on the properties of lexical richness, cohesive markers, collocations, pronouns, content words, and function words. to make a fair comparison, we directly use the same data provided by @xcite and @xcite to reproduce the results of native texts and translationese. the settings are shown in appendix a.1. as shown in figure 1 , the trend of our collected native texts and gec data is consistent with that of the native texts and translationese provided by existing work. for example, both the translationese and gec data are of lower lexical richness and contain more cohesive markers and function words than the native texts. one outlier is the result of english pronouns, and the reason is the overuse of personal pronouns such as 'i' and 'you' in the gec data. however, by looking at the result of chinese pronouns, it still has the same trend. the above results confirm our assumption that translationese and gec data have a similar style than native texts. specifically, the native source monolingual text is translated to machine translated text through a trained machine translation system. the translationese is identified via the bert classifier, which is fine-tuned with the same amount of machine translated text and native target monolingual text. the obtained translationese is injected with specific noise to produce a synthetic gec corpus which is merged with the original gec corpus to train a gec system. ## transgec the observations made above enable us to further improve gec with translationese. figure 2 shows the overall framework of transgec, which contains two parts: obtaining translationese and improving gec with translationese. obtaining translationese existing parallel corpora of machine translation (mt) tasks @xcite have a huge amount of translationese on both sides. however, most parallel corpora do not annotate whether an instance is native or translated. therefore, previous studies @xcite have had to train a classifier to identify and obtain translationese from parallel corpora. in this paper, to obtain translationese from existing parallel training corpora of mt, we propose to fine-tune bert-based classifiers using a small number of machine translated texts @xcite , which can alleviate the limitation of @xcite relying on a large amount of machine translated texts to train a convolutional neural network-based classifier from scratch. specifically, given a parallel corpus d @xmath0, y n )} n @xmath1, we first need to train a machine translation model f @xmath2: then, the machine translated texts y mt can be obtained by translating the native source sentences: given the generated y mt and collected y native , we fine-tune the bert-based pre-trained language model as a classifier to distinguish whether a sentence is native or not. after that, we use the finetuned bert-based classifier to label the target side of the parallel corpus d mt , and identify the sentences which have lower classification probabilities to be native texts as translationese y trans . improving gec with translationese this part exploits the obtained translationese y trans as input for gec data augmentation. motivated by @xcite , artificial noise is added to y trans and the synthetic gec corpus d syn can be viewed as: after that, we can train a gec model with the original corpus d gec and synthetic corpus d syn : ## analysis in this section, we analyze our results from two perspectives: error types and linguistic properties. error types we investigate the performance of different error types for english and chinese gec tasks. we use the errant toolkit @xcite for english. for chinese, we use the adapted errant released by @xcite . as shown in ## conclusion this paper introduces a transgec method that uses translationese as input for data augmentation of gec. preliminary experiments on native texts, translationese, and gec data confirm that the translationese and gec data share a similar style compared to native texts. based on the evidence, we propose a simple and effective method to mine translationese from parallel corpora by classifiers and construct a synthetic gec corpus by adding artificial noise to the translationese. experimental results on the conll14 and bea19 english, nlpcc18 chinese, falko-merlin german, and rulec-gec russian benchmarks show that the models augmented with translationese can outperform strong baselines. further analyses show that our approach performs well in solving major difficulties (e.g., correction of frequent words, missing words, and substitution errors), but still has some room for improvement in minor issues (e.g., correc-tion of rare words, word order, and deletion errors).