CCTC : A Cross-Sentence C hinese Text Correction Dataset for Native Speakers (2022)

Abstract: The Chinese text correction (CTC) focuses on detecting and correcting Chinese spelling errors and grammatical errors. Most existing datasets of Chinese spelling check (CSC) and Chinese grammatical error correction (GEC) are focused on a single sentence written by Chinese-as-a-second-language (CSL) learners. We find that errors caused by native speakers differ significantly from those produced by non-native speakers. These differences make it inappropriate to use the existing test sets directly to evaluate text correction systems for native speakers. Some errors also require the cross-sentence information to be identified and corrected. In this paper, we propose a cross-sentence Chinese text correction dataset for native speakers. Concretely, we manually annotated 1,500 texts written by native speakers. The dataset consists of 30,811 sentences and more than 1,000,000 Chinese characters. It contains four types of errors: spelling errors, redundant words, missing words, and word ordering errors. We also test some state-of-the-art models on the dataset. The experimental results show that even the model with the best performance is 20 points lower than humans, which indicates that there is still much room for improvement. We hope that the new dataset can fill the gap in cross-sentence text correction for native Chinese speakers.

## introduction chinese text correction (ctc) aims at detecting and correcting errors in chinese text. text correction has important applications in the domain of education, journalism, and publishing. for many native chinese speakers, such as journalists, writers, and bloggers, a text correction system for native chinese speakers will greatly improve the efficiency of their proofreading. in the field of nlp, chinese text corrections usually includes two tasks: chinese spelling check (csc) @xcite @xcite and chi- the existing csc and chinese gec test sets @xcite @xcite are mainly generated from essays written by chinese-as-a-second-language (csl) learners. the essays written by csl learners are significantly different from those written by native chinese speakers. specifically, essays written by csl learners usually contain more errors and are more likely to make mistakes in the misuse of words. in contrast, texts produced by native speakers contain sparser errors and typically make mistakes that are caused by oversight. these significant differences prevent researchers from using the existing test sets directly to evaluate text correction systems for native speakers. figure 1 shows the errors made by csl learners and native speakers, respectively. we can see the csl learners make some mistakes that are obvious to native speakers. the word "爱情" usually refers to the love between a couple, while "关爱" indicates the love of an elder for a younger child. in chinese, these two words are not interchangeable. however, for csl learners, it is easy to mistakenly write "关爱" as "爱情" because they can both be translated into "love" in english. similarly, the words "利益" and "好处" can both be translated into "benefit" in english, but the word "利益" cannot be used with "健康" (health) in chinese. native speakers will not make these mistakes. for native chinese speakers, the most common errors are caused by oversight, which the writers themselves are capable of correcting. for example, the misspelling of "信息"(information) as "信心" (confidence) is due to the similarity of the pinyin for xinxi and xinxin, respectively. besides, the test sets for non-native speakers, such as cged @xcite tend to write simpler sentences with limited topics. in contrast, the texts written by native speakers tend to have complicated sentences with various topics. moreover, the existing datasets of csc and gec are mainly for sentence-level correction. however, some errors usually need to be corrected via the cross-sentence information @xcite . for example, in figure 2, it is difficult to see what is wrong with each sentence individually. according to the previous sentences, we know that the word "蜘蛛" (spider) should be corrected as "红蜘蛛" (red spider). to better evaluate the text correction system's performance on document-level texts produced by native speakers, we propose a new dataset cctc (cross-sentence chinese text correction). since every chinese character may be erroneous, the scale of annotation is large. without any auxiliary hints, the annotators will be prone to miss the errors. therefore, we give the annotators some hints about the position and type of errors produced by several csc and gec systems. we first annotate all the sentences from 200 documents and find only 11.4% sentences with errors. errors in sentences with candidate errors account for more than 90% of all errors. in order to maximize the diversity of topics and increase the number of errors in the dataset, we only annotate the sentences with error candidates for another 1,300 documents. concretely, we wrong: 红蜘蛛俗称火蜘蛛、火龙。红蜘蛛……。危害特点： 蜘蛛是一种危害作物种类较多的害虫，以成虫、幼虫 或若虫群聚在叶背吸取汁液。 correct: 红蜘蛛俗称火蜘蛛、火龙。红蜘蛛……。危害特点： 红蜘蛛是一种危害作物种类较多的害虫，以成虫、幼 虫或若虫群聚在叶背吸取汁液。 translation: red spider is commonly known as fire spider and fire dragon. red spider … . damage characteristics: spider red spider is a pest that affects more crop species, with adults, larvae, or worm clusters in the back of the leaves to suck sap. annotate 1,500 texts from the internet, and the annotated text includes a total of 30,811 sentences and more than 1 million chinese characters. we utilize several types of state-of-the-art models for experiments and analyses on our dataset. we also evaluate the performance of native speakers on cctc. the experimental results show that even the model with the best performance is still 20 points worse than the human, which indicates that there is still much room for improvement. to summarize, our contributions are as follows: we have done some experiments and analyses for cross-sentence errors, which we hope will be helpful for subsequent studies of crosssentence text correction. we hope that cctc will contribute towards the development of cross-sentence chinese text correction for native speakers. our datasets are publicly available at https://github.com/ destwang/ctcresources . ## existing datasets the chinese text correction related datasets mainly include chinese spelling check (csc) and grammatical error correction (gec). statistics information is shown in ## cctc dataset we construct a new cross-sentence chinese text correction dataset for native speakers. we extract the raw text from wudaocorpora @xcite , which mainly includes news, blogs, and some popular science articles. we pre-process the collected documents, remove personal information, advertisements, and noisy articles, then sample 1,500 documents for annotation. we take 100 of these documents for verification. we can determine by the author's information that all the 100 documents candidates generation to facilitate manual annotation and reduce error omission, we utilize several different models to generate error candidates. specifically, we select three different kinds of models as follows. the detailed information of the training set will be described in the next section. to cover as many errors as possible, we lower the thresholds of the three models. in this way, these models will generate more candidates to find out the erroneous parts of the documents. annotation following @xcite , errors are divided into four types: spelling errors (word selection errors), redundant words, missing words, and word ordering errors. the data are annotated by five annotators, with an average of about 120 hours and 2k sentences each. our annotators annotate the dataset on an annotation tool prepared in advance. we pay our annotators appropriately according to the number of annotated sentences. we firstly annotate 3,784 sentences from 200 documents, including sentences with error candidates and sentences without candidates. after annotating, we find that there are only 431 sentences with errors. errors in candidate sentences account for more than 90% of all errors. in order to maximize the diversity of topics and increase the number of errors in the dataset, we only annotate the sentences with error candidates for another 1,300 documents. we name the dataset with 200 annotated documents as cctc-h, which means a highquality dataset. the remaining 1,300 documents are divided into two parts, 650 of which are used as the training set and the other 650 documents as the cctc-w, which means this test dataset contains a wider range of documents. to conclude, we annotate 1,500 documents from the internet, and the annotated texts include a total of 30,811 sentences and more than 1 million chinese characters. the detailed statistics of different candidate generation methods are shown in in order to ensure the quality of the annotated data, we take 500 sentences from the training set, validation set, and test set, respectively, and annotate these sentences without candidate errors. similar to @xcite , annotator agreement is calculated at the sentence level using cohen's kappa. kappa is 0.76, 0.72, and 0.78 for the cctc-train, cctc-w, and cctc-h, respectively, showing that our dataset has a higher agreement than the previous dataset. ## cross-sentence errors we randomly analyze 100 errors and find that crosssentence information is necessary for only 11% of the errors. however, cross-sentence information can be helpful for 38% of errors, such as when the corrected word appears in context. to test the help of cross-sentence information for chinese text correction, we try a simple crosssentence correction method, which increases the length of the input sequence. we vary from singlesentence correction to multi-sentence correction, and figure 6 shows the experimental results. from figure 7 : examples of cctc. the above sentence is an example of failure to correct during human testing, and the below one is an example for mis-correction by spellgcn. the experimental results, we can see that for a trained model, the performance of the model increases as the input sequence length grows. this also shows that the cross-sentence information is helpful for chinese text correction. the models often mis-correct some lowfrequency words due to the lack of context of a document. in figure 7 , the model mistakenly modify "天生桥自然" (tiansheng bridge) as "天生与 自然" (natural). in fact, the word "天生桥" has appeared many times in the context of the document. if we could better use the cross-sentence contextual information, it would help better with the correction. based on this, we do not simply split the document into individual sentences but keep the complete cross-sentence information. we hope it will be helpful for subsequent studies of cross-sentence text correction. ## conclusion in this paper, we propose a novel cross-sentence chinese text correction dataset for native speakers. concretely, we manually annotated 1,500 chinese texts written by native speakers collected from the internet. the new dataset consists of 30,811 sentences and more than 1,000,000 chinese characters. it contains spelling errors, redundant words, missing words, and word ordering errors. csc and gec systems developed for native speakers can be better evaluated on cctc than the previous datasets. we also test some state-of-the-art models on the dataset. the experimental results show that even the model with the best performance is still 20 points worse than the human, which indicates that there is still much room for improvement.