From Spelling to Grammar: A New Framework for C hinese Grammatical Error Correction (2022)

Abstract: Chinese Grammatical Error Correction (CGEC) aims to generate a correct sentence from an erroneous sequence, where different kinds of errors are mixed. This paper divides the CGEC task into two steps, namely spelling error correction and grammatical error correction. We firstly propose a novel zero-shot approach for spelling error correction, which is simple but effective, obtaining a high precision to avoid error accumulation of the pipeline structure. To handle grammatical error correction, we design part-of-speech (POS) features and semantic class features to enhance the neural network model, and propose an auxiliary task to predict the POS sequence of the target sentence. Our proposed framework achieves a 42.11 F-0.5 score on CGEC dataset without using any synthetic data or data augmentation methods, which outperforms the previous state-of-the-art by a wide margin of 1.30 points. Moreover, our model produces meaningful POS representations that capture different POS words and convey reasonable POS transition rules.

## introduction grammatical error correction (gec) takes erroneous sequences as input and generates correct sentences. in recent years, english gec task has attracted wide attention from researchers. by employing pre-trained models @xcite or incorporating synthetic data @xcite , the sequence-to-sequence models achieve remarkable performance on english gec task. besides, several sequence labeling approaches are proposed to cast text generation as token-level edit prediction @xcite @xcite . chinese grammatical error correction (cgec) is less addressed. previous works adopt ensemble methods by combining seq2seq networks with heuristic rules @xcite or sequence editing approaches @xcite . different from english, chinese language utilizes function words instead of affixes to represent forms and tenses, making it hard to design detailed chinese-specific edit labels. the simple label strategy (for example, keep, delete, append_x) has been proved not competitive with the seq2seq model on cgec task @xcite . generally, the errors occurring in chinese texts can be divided to spelling errors and grammar errors. for example in figure 1 , "主" is a spelling error which should be subtitued to "注". while "和 (and)" relates to grammatical error which should be deteled. according to our statistics on hsk data foot_0 , which is collected from the writing section of chinese proficiency exam, the proportion of spelling errors is about 18.58%. unfortunately, most of previous works mix two kinds of errors and adopt the same model to handle them. moreover, as the basic elements for sentence understanding and processing, spelling errors will influence the usage of high-level features in the cgec task. although amounts of linguistic features have been investigated to improve many natural language processing tasks, deep syntactic and semantic knowledge is rarely explored in cgec. therefore, we propose a new framework for cgec with two steps: spelling error correction and semantic-enriched grammatical error correction @xcite . we propose a novel zero-shot method for chinese spelling error correction, by taking advantage of the pre-trained bert and chinese phonetic information, which is straightforward but achieves a satisfying precision. further, we introduce semantic knowledge into the seq2seq model to correct grammatical errors. we carefully analyze the reliability and utility of part-of-speech (pos) in erroneous-correct paired sentences, and design an effective method to integrate pos and semantic representations into the neural network model. moreover, we introduce an auxiliary task of pos sequence prediction, where a conditional random field (crf) layer is added to ensure the valid of generated pos sequences and stimulate the model to learn grammar-level corrections. we conduct extensive experiments on cgec nlpcc dataset @xcite . experimental results show that our proposed zero-shot spelling error correction module achieves a 60.25 precision, which lays a good foundation for further leveraging word-level features. with the pre-trained bart for initialization, our model achieves a new stateof-the-art result of 42.11 f 0.5 score, which outperforms all previous approaches including pre-trained models and ensemble methods. we also evaluate model performance on cged-2020 test dataset @xcite and obtain satisfying results. to sum, our contributions are as follows: ## observation and intuition various types of linguistic features have been exploited in nlp, which bring improvement on different tasks. however, it remains an open issue to introduce linguistic features to gec. different from other nlp tasks, the gec task takes erroneous sentences as input, based on which the extra features might bring noise to the gec model that harms the performance. ## zero-shot spelling error correction we present a new framework for cgec as shown in figure 2 , which consists of spelling error correction (sec) and grammatical error correction (gec). firstly, we propose a smart zero-shot method for sec. formally, the input erroneous sentence is represented as @xmath0, x 2 , ...x n ), and the target correct sentence is denoted as @xmath1, y 2 , ...y m ), where n, m mean the length of sentence. we use @xmath2, x2 , ..., xn ) to represent the output of the zero-shot spelling error correction module: specially, if a token x i in x has a relatively high probability of being written incorrectly, it will be substituted with a [mask] token. then, the pre-trained bert model @xcite is employed to generate top-3 candidate tokens @xmath3, v 2 , v 3 ) with high probability. among the candidates, we select the token that most likely appears in the [mask] position: if there exists more than one token belonging to simset(x i ), we choose the token with the highest score generated by bert. according to the previous study, over 80% spelling errors in chinese are related to phonological similarity @xcite . so, we set simset(x i ) to be the collection of homophones of x i . figure 2 gives an example in the left part. in the given sentence, 金(golden) is suspected to be written incorrectly and substituted with @xcite . the top three tokens with the highest score generated by bert are: 去(last), 今(this) and 每(every). among them, 今/jin shares the same pinyin with 金/jin. so we replace 金 with 今 in the original sentence. a problem in sec is how to decide whether a token x i is likely to be written incorrectly. intuitively, the punctuation and commonly used chinese characters, whose occurrences in the training dataset are over k c , are less likely to be written incorrectly. we directly keep these tokens unchanged to improve the precision of sec module and reduce the computational expense. to find out the appropriate threshold value k c , we conduct experiments on the test set of @xcite , which is designed specially for chinese spelling error correction and contains 1100 examples collected from chinese language learners. as shown in figure 3 , the precision score on sighan test dataset has peaked at 66.1 when k @xmath4, 000 and gradually declines after k @xmath5, 000. in order to restrain error accumulation of the pipeline structure, we hope our sec module to be high in precision. accordingly, we set k @xmath6, 000 in our experiment. ## integrating semantics for grammatical error correction we adopt the transformer encoder-decoder architecture for grammatical error correction. to project semantic knowledge to cgec, at the encoder we incorporate the semantic knowledge representations, and at the decoding we design the pos sequence generation as an auxiliary task. we add the semantic knowledge embedding e semantics to the original word embedding to serve as the input of encoder: in the decoding phase, we take the hidden state of timestep t to predict the t th token in the target sentence: ## conclusion in this paper, we divide cgec into two consecutive tasks: spelling error correction and grammatical error correction. we propose a zero-shot spelling error correction method, utilizing the pretrained bert model and taking advantage of chinese phonological knowledge. it achieves a high precision score to avoid error accumulation in the pipeline structure. based on the careful analysis on real data, we inject proper semantic features into the encoder. and at the same time, we generate correct pos sequence as a sub-task to help generate correct sentences, where crf is applied to guarantee the validness of the generated pos sequence. initialized by the pre-trained bart model, our proposed framework achieves a new state-of-the-art result on cgec task, outperforming the previous best result by a large margin.