S pell GCN : Incorporating Phonological and Visual Similarities into Language Models for C hinese Spelling Check (2020)

Abstract: Chinese Spelling Check (CSC) is a task to detect and correct spelling errors in Chinese natural language. Existing methods have made attempts to incorporate the similarity knowledge between Chinese characters. However, they take the similarity knowledge as either an external input resource or just heuristic rules. This paper proposes to incorporate phonological and visual similarity knowledge into language models for CSC via a specialized graph convolutional network (SpellGCN). The model builds a graph over the characters, and SpellGCN is learned to map this graph into a set of inter-dependent character classifiers. These classifiers are applied to the representations extracted by another network, such as BERT, enabling the whole network to be end-to-end trainable. Experiments are conducted on three human-annotated datasets. Our method achieves superior performance against previous models by a large margin.

## introduction spelling errors are common in our daily life, caused typically by human writing, automatic speech recognition, and optical character recognition systems. among these errors, misspelling a character frequently occurs due to the similarity between characters. in chinese, many characters are phonologically and visually similar, but semantically very different. according to @xcite , about 83% of errors are related to phonological similarity and 48% are related to visual similarity. the chinese spelling check (csc) task aims to detect and correct such misuse of the chinese language. despite recent development, csc remains a challenging task. notably, the spelling checking on chinese is very different from english, due to its language input 餐厅的换经费产适合约会 (phonics) cān t īng dē huàn j īng fèi chán shì hé yuē huì bert 餐厅的月消费最适合约会 (phonics) cān t īng dē yuè xiāo fèi zuì shì hé yuē huì gold label 餐厅的环境非常适合约会 (phonics) cān t īng dē huán jìng fēi cháng shì hé yuē huì and the meaning of each character changes dramatically when the context changes. therefore, a csc system needs to recognize the semantics and aggregate the surrounding information for necessary modifications. previous methods followed the line of generative models. they used either language models @xcite @xcite or sequenceto-sequence models @xcite . to fuse the external knowledge of the similarity between characters, some of them leveraged a confusion set, which contains a set of similar character pairs. for instance, @xcite proposed to produce several candidates by retrieving the confusion set and then filter them via language models. @xcite used a pointer network to copy a similar character from the confusion set. these methods attempted to utilize the similarity information to confine the candidates, rather than modeling the relationship between characters explicitly. in this paper, we propose a novel spelling check convolutional graph network (spellgcn) that cap-tures the pronunciation/shape similarity and explore the prior dependencies between characters. specifically, two similarity graphs are constructed for the pronunciation and shape relationship correspondingly. spellgcn takes the graphs as input and generates for each character a vector representation after the interaction between similar characters. these representations are then constructed into a character classifier for the semantic representation extracted from another backbone module. we use bert @xcite due to its powerful semantic capacity. combining the graph representations with bert, spellgcn can leverage the similarity knowledge and generate the right corrections accordingly. regarding the example as in experiments were conducted on three open benchmarks. the results demonstrate that spell-gcn improves bert evidently, outperforming all competitor models by a large margin. in summary, our contributions are as follows: ## related work the csc task is a long-standing problem and has attracted much attention from the community. the research emerges in recent years @xcite @xcite @xcite @xcite , together with other topics, e.g., grammar error correction (gec) @xcite @xcite . csc focuses on detecting and correcting character errors, while gec also includes errors that need deletion and insertion. previous work handles csc using unsupervised language models @xcite . the errors are detected/corrected by evaluating the perplexity of sentences/phrases. however, these models were unable to condition the correction on the input sentence. to circumvent this problem, several discriminative sequence tagging methods were adopted for csc @xcite . for more flexibility and better performance, several sequence-to-sequence models were also employed @xcite @xcite , as well as bert @xcite . recent attention was paid to utilizing the external knowledge of character similarity. the similarity knowledge can be gathered into a dictionary, i.e., confusion set, where similar pairs are stored. @xcite first used the dictionary to retrieve similar candidates for potential errors. @xcite incorporated a copy mechanism into a recurrent neural model. when given similar characters as input, their model uses the copy mechanism to directly copy the character to the target sentence. in a sense, these models face difficulty in modeling the relationship between similar characters as the similarity information is solely used for candidate selection. to capture the pronunciation/shape similarity and explore the prior dependencies between characters, we propose to use graph convolution network (gcn) @xcite to model character inter-dependence, which is combined with the pre-training of bert @xcite for the csc task. gcn has been applied to model the relationship on several tasks. @xcite equipped it into the relation extraction task where relations construct a hierarchical tree. @xcite ; @xcite use it to model spatial-temporal to predict traffic flow. gcn was also used to model the relationship between labels in a multi-label task @xcite . in this paper, it is the first time that gcn is applied successfully into the csc task. the relationship in csc is much different from those tasks where objects in the graph are semantically related. by contrast, the similar characters are semantically distinct in csc. therefore, we deeply investigate the effect of our spellgcn and propose several essential techniques. ## approach in this section, we elaborate on our method for csc. firstly, the problem formulation is presented. then, we introduce the motivations for spellgcn, followed by its detailed description. at last, we present its application in the csc task. ## experiments in this section, we describe our experiment in detail. we first present the training data and test data, as well as the evaluation metrics. then we introduce our main results for spellgcn. after that, the ablation studies are made to analyze the effect of the proposed components, followed by a case study. finally, quantitative results are provided. ## conclusions we proposed spellgcn for csc to incorporate both phonological and visual similarities into language models. the empirical comparison and the results of analytical experiments verify its effectiveness. beyond csc, spellgcn can be generalized to other situations where specific prior knowledge is available, and to other languages by leveraging specific similarity graphs analogously. our method can also be adapted to grammar error correction, which needs insertion and deletion, by utilizing more flexible extractors such as levenshtein trans-former @xcite . we leave this direction to future work.