Self-Supervised Curriculum Learning for Spelling Error Correction (2021)

Abstract: Spelling Error Correction (SEC) that requires high-level language understanding is a challenging but useful task. Current SEC approaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) utilizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model’s performance has been shown sensitive to the difficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are naturally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speakers are hard), and we expect that designing a curriculum correspondingly for model learning may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and propose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model training, which in return improves the CL measurement. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and establish a new state of the art (74.38% F1).

## introduction spelling error correction (sec) aims to automatically correct the spelling errors in written text either at word-level or character-level @xcite @xcite @xcite . ## related work spelling error correction. sec is helpful for many applications, such as essay scoring @xcite , search @xcite , optical character recognition (ocr) @xcite , machine translation and tagging @xcite , and many studies have been conducted on the sec task. unsupervised approaches using language models and rules @xcite are widely adopted. sec is treated as a sequential labeling problem in machine learning approaches, and conditional random fields or hidden markov models @xcite curriculum learning. cl @xcite aims to facilitate the model training in an easy-tohard order, which leads to improved model performance @xcite @xcite . many studies adopt cl to reinforce learning to optimize the model parameters @xcite . cl also has shown to be useful for data processing to improve the quality of the training data @xcite . recently, cl has been widely employed in the machine learning for nlp. it improves the performance and the training efficiency of the nmt models based on linguistic features @xcite @xcite , enhances the multi-domain correlation, and addresses the domain imbalance issue @xcite . it also has been explored in other tasks, such as response generation @xcite and reading comprehension @xcite . self-supervised learning. the basic idea of self-supervised learning (ssl) is to automatically generate or find supervision signals to solve tasks. for instance, it is used to learn representations from unlabeled data @xcite . @xcite use ssl to mine useful attention supervision information from the training corpus to refine attention mechanisms. @xcite combine the ssl with pseudo-labels and meta-learning during inference to improve generalization. @xcite use an emergent nmt system to simultaneously select training data and learn internal nmt representations in a ssl way without parallel data. ssl is also adopted to solve many other problems, such as document-level context or sentence summarization @xcite , dialogue learning @xcite , improving data scarcity or labeling costs @xcite and generating meta-learning tasks from unlabeled text @xcite . comparison to previous work. compared to previous cl studies, we apply ssl to cl and propose sscl that uses the model to measure data difficulty for training instance selection in an easy-to-hard order. compared to previous sec approaches, we employ sscl for the training of sec, which establishes a new sota (74.38% f1) on the sighan 2015 chinese spelling check task. ## conclusion in this paper, we applied curriculum learning to spelling error correction and present a novel self-supervised curriculum learning method. we verify the effectiveness of the sscl ap-proach on the sighan 2015 chinese spelling check task. experiment results show that sscl is able to significantly improve the performance of the state-of-the-art soft-masked bert model and establishes a new state-of-the-art performance (74.38% f1). the fact that sscl brings about more improvements than the previous norm-based and uncertainty-aware cl approaches also supports its effectiveness as a cl approach.