N a SGEC : a Multi-Domain C hinese Grammatical Error Correction Dataset from Native Speaker Texts (2023)

Abstract: We introduce NaSGEC, a new dataset to facilitate research on Chinese grammatical error correction (CGEC) for native speaker texts from multiple domains. Previous CGEC research primarily focuses on correcting texts from a single domain, especially learner essays. To broaden the target domain, we annotate multiple references for 12,500 sentences from three native domains, i.e., social media, scientific writing, and examination. We provide solid benchmark results for NaSGEC by employing cutting-edge CGEC models and different training data. We further perform detailed analyses of the connections and gaps between our domains from both empirical and statistical views. We hope this work can inspire future studies on an important but under-explored direction–cross-domain GEC.

## introduction grammatical error correction (gec) aims to remove all underlying textual errors in a given sentence without changing its meaning @xcite . during the past decade, gec has attracted a lot of research interest and has been integrated into many real-life applications like writing assistants. a significant effort has been undertaken to build high-quality datasets for research on gec. most gec datasets are for english @xcite @xcite , which mainly collect sentences from learner essays. for chinese gec (cgec), datasets are relatively scarce. similar to english gec, most of them are built from essays written by learners, including nlpcc18 @xcite , cged @xcite , yaclc @xcite , and mucgec @xcite . besides learner gec, there is also great demand for correcting errors made by native speakers. for english gec, researchers have already constructed to alleviate these problems, this work proposes nasgec (pronounced as /"neisgek/), a multidomain dataset from native speaker texts for chinese gec. nasgec comprises 12,500 sentences from 3 native text domains: social media platform (media), undergraduate theses (thesis), and chinese examinations (exam). these domains are closely related to real-life gec application scenarios, i.e., writing aid, paper proofreading, and chinese teaching. based on detailed data analysis (see section 3), we demonstrate that they have diverse writing styles and error distributions, thus posing great challenges for existing models and will be an ideal testbed for domain adaptation techniques. furthermore, there are usually different correction methods for an error, as shown in using nasgec, we conduct extensive experiments. we evaluate the performance of the stateof-the-art (sota) cgec model on nasgec with different kinds of training data. we first train the model on commonly-used human-annotated training sets. since these training sets are collected from learner texts while nasgec is a native dataset, we also generate synthetic training data from native texts. the multi-domain property of nasgec enables us to shed light on the domain problem in cgec. we conduct domain transfer experiments and design three indicators for evaluating domain differences. in summary, our main contributions can be concluded as follows: to further improve model performance in a specific domain, we propose a simple domainaware data augmentation method. all codes and models have been released at https://github.com/hillzhang1999/ nasgec . we will also release the dataset after improving it according to reviewers' comments. ## construction of nasgec this section describes the construction process of nasgec in detail. as shown in figure 1 , we first collect raw sentences from three domains. then, each sentence is assigned to two annotators for independent annotation. to guarantee data quality, an expert will carefully review the annotation results. ## analysis of nasgec overall statistics. we list detailed statistics of nasgec and other existing datasets for comparison in we also present the statistics of two mainstream learner datasets, i.e., nlpcc18 @xcite and mucgec @xcite . compared with those learner datasets, sentences in nasgec are significantly longer but contain much fewer edits, as natives make mistakes far less frequently than learners and seldom make obvious mistakes. besides, sentences in nasgec also have more name entities and a higher level of lexical variety, showing that natives have a larger vocabulary. error type distributions. we use the tool provided by mucgec to classify extracted edits into 4 error types according to their correction operations. figure 2 shows the distributions of these error types in nasgec and other datasets for comparison. within nasgec, the most frequent error type in media and theis is substituted errors. after further decomposition, we find that the majority of substituted errors in these 2 domains are caused by spelling or misuse of punctuation, as native speakers usually make such minor mistakes due to carelessness when typing essays or papers. the media domain also has a considerable proportion of missing errors, mainly caused by missing punctuation. such errors often occur in informal texts, as the absence of punctuation generally does not affect the understanding of the sentence. compared with the other domains, exam has a more even type distribution, where the proportion of substituted, missing, and redundant errors is quite close. like media and thesis domains of nasgec, the learner dataset mucgec also has a high proportion of substituted and missing errors. after a deeper look into samples, we find that learners are more prone to misuse verbs or nouns due to lexical or grammatical unfamiliarity, and they also tend to miss more specific words instead of punctuation. among all datasets, cctc has the most unbalanced distribution: the substituted errors account for nearly 70%, and we find most of them are caused by spelling. although both come from chinese examinations, fcgec and nasgec-exam still have some discrepancies, such as fcgec contains more redundant errors, which may be due to different annotation guidelines and data sources. ## benchmark experiments on nasgec this section provides benchmark results for nas-gec with a current sota cgec model. following previous work, we train the model on humanannotated training data from learner texts. however, there exists a gap between learner training data and our native dataset. so we also use synthetic native training data to mitigate the gap. ## domain analysis within nasgec in this section, we conduct domain transfer experiments on nasgec by splitting data and performing fine-tuning. we devise indicators of gec domain shifts to gain more insights into the connections and differences between our domains. to further improve model performance in specific domains, we also propose a simple domain-aware data augmentation method. ## comparison with existing datasets in this section, we compare nasgec with existing cgec datasets, including both native and learner datasets, by analysis of domain shift indicators ( nasgec vs. existing learner datasets. most existing cgec datasets are for learners. we select mucgec @xcite from them for comparison, because it actually covers several previous learner datasets, e.g., nlpcc18 @xcite and cged @xcite . from domain shift indicators in from domain transfer experiments in we encourage future work to explore better ways to transfer between native and learner domains, which will allow us to apply the rich experience of learner gec to under-explored native gec. ## conclusion this paper presents nasgec, a new multi-domain native cgec dataset, which consists of 12,500 sentences from three representative native domains. we clearly describe the construction process and perform detailed data analysis. we conduct benchmark experiments with the sota bart-based cgec model and two kinds of training data. we also launch domain transfer experiments and devise domain shift indicators, in order to have a clearer understanding of our domains. we hope nasgec can spur future work on cross-domain gec evaluation, domain adaptation for gec, and more.