CSCD - NS : a C hinese Spelling Check Dataset for Native Speakers (2024)

Abstract: In this paper, we present CSCD-NS, the first Chinese spelling check (CSC) dataset designed for native speakers, containing 40,000 samples from a Chinese social platform. Compared with existing CSC datasets aimed at Chinese learners, CSCD-NS is ten times larger in scale and exhibits a distinct error distribution, with a significantly higher proportion of word-level errors. To further enhance the data resource, we propose a novel method that simulates the input process through an input method, generating large-scale and high-quality pseudo data that closely resembles the actual error distribution and outperforms existing methods. Moreover, we investigate the performance of various models in this scenario, including large language models (LLMs), such as ChatGPT. The result indicates that generative models underperform BERT-like classification models due to strict length and pronunciation constraints. The high prevalence of word-level errors also makes CSC for native speakers challenging enough, leaving substantial room for improvement.

## introduction chinese spelling check (csc) is a task to detect and correct spelling errors in chinese texts. there are two primary user groups for csc: (1) chinese learners, including teenage students and individuals who use chinese as a second language, and (2) chinese native speakers. it is obvious that the latter user group has a larger population and more diverse applications, therefore, this paper concentrates on csc for native speakers. however, there is still no csc dataset specifically designed for native speakers. existing csc datasets, such as sighan13, 14, and 15 @xcite @xcite , are all sourced from chinese learners. spelling errors made by chinese learners differ greatly from those made by native speakers. this is because chinese input relies on chinese input methods (ime), and modern chinese imes always have powerful language models, making it difficult to recommend candidates that clearly do not fit the context. as shown in figure 1 , native speakers using chinese imes are unlikely to make such an unusual error. furthermore, the size of existing datasets is limited. as shown in to address the aforementioned issues, we introduce cscd-ns, a chinese spelling check dataset designed for native speakers. the dataset is sourced from real weibo (a chinese social media platform) posts, which contain genuine spelling errors made by native speakers during their input process. moreover, the dataset comprises 40,000 samples, which is ten times larger than previous datasets and this is also the largest dataset for the csc task. to conduct an in-depth investigation into the distribution of spelling errors, we develop a tagging system that operates at phonetic and semantic levels. the analysis indicates that native speakers make a higher proportion of homophonic and word-level errors compared to chinese learners, with the proportion of word-level errors doubling. due to the lack of labeled data, previous studies always build additional pseudo data to improve the performance of models. however, these methods, which rely on confusion sets @xcite or asr transcriptions @xcite , do not align with the real-world input scenario. therefore, we propose a novel method that directly simulates the input process through the chinese ime and adds sampled noises to construct high-quality pseudo data. experimental results show that our method can better fit the real error distribution and bring greater improvements. we conduct comprehensive experiments on cscd-ns, with different model sizes (from 0.1b to 13b parameters), architectures (encoder-only, encoder-decoder, and decoder-only), and learning approaches (fine-tuning and in-context learning). we also evaluate the performance of chatgpt and gpt4. the results demonstrate that bert-like classification models outperform generative models, as the latter struggle with the simultaneous constraints of text length and pronunciation. concurrently, the csc task for native speakers is challenging due to the high proportion of word-level errors, leaving substantial room for improvement. in summary, our contributions are as follows: ## cscd-ns in this section, we will show how to build cscd-ns and discover the error distribution. ## data augmentation the manual annotation of csc dataset is very expensive, therefore, how to construct pseudo data has always been a valuable topic. in this section, we introduce a novel method that can generate highquality pseudo data on a large scale. ## experiments in this section, we evaluate the performance of different models on cscd-ns and compare different pseudo-data construction methods. ## conclusion in this paper, we focus on csc for native speakers. for this scenario, we propose a new dataset, cscd-ns, which is also the largest dataset for csc. we further unveil the specific error distribution, with a significantly higher proportion of word-level errors. moreover, we introduce an ime-based pseudo-data construction approach, enabling large-scale generation of high-quality pseudo-data. we explore the performance of various models and first evaluate chatgpt and gpt4 on the csc task. our experiments demonstrate that bert-like models exhibit better performance than generative models, but there is still a considerable room for improvement. we hope these data resources and our findings could stimulate further research in this area.