Context-aware Stand-alone Neural Spelling Correction (2020)

Abstract: Existing natural language processing systems are vulnerable to noisy inputs resulting from misspellings. On the contrary, humans can easily infer the corresponding correct words from their misspellings and surrounding context. Inspired by this, we address the stand-alone spelling correction problem, which only corrects the spelling of each token without additional token insertion or deletion, by utilizing both spelling information and global context representations. We present a simple yet powerful solution that jointly detects and corrects misspellings as a sequence labeling task by fine-turning a pre-trained language model. Our solution outperform the previous state-of-the-art result by 12.8% absolute F0.5 score.

## introduction a spelling corrector is an important and ubiquitous pre-processing tool in a wide range of applications, such as word processors, search engines and machine translation systems. having a surprisingly robust language processing system to denoise the scrambled spellings, humans can relatively easily solve spelling correction @xcite . however, spelling correction is a challenging task for a machine, because words can be misspelled in various ways, and a machine has difficulties in fully utilizing the contextual information. misspellings can be categorized into non-word, which is out-of-vocabulary, and the opposite, realword misspellings @xcite . the dictionary look-up method can detect non-word misspellings, while real-word spelling errors are harder to detect, since these misspellings are in the vocabulary @xcite . in this work, we address the stand-alone @xcite spelling correction problem. it only * work performed during internship with baidu usa word+char encoder model i love you affine @xmath0: combined word-level and character-level encoder model. right: subword-level model using bio2 tagging scheme @xcite . we formulate the stand-alone spelling correction as a sequence labeling task and jointly detect and correct misspellings. inspired by the human language processing system, we propose a novel solution on the following aspects: (1) we encode both spelling information and global context information in the neural network. (2) we enhance the real-word correction performance by initializing the model from a pre-trained language model (lm). ## approach we use the transformer-encoder @xcite to encode the input sequences and denote it as encoder. as illustrated in figure 1 , we present both word+char encoder and subword encoder, because we believe the former is better in encoding spelling information, while the latter has the benefit of utilizing a large pre-trained lm. word+char encoder. we use a word encoder to extract global context information and a character encoder to encode spelling information. as shown in equation 1, in order to denoise the noisy word sequence s * to the clean sequence s, we first separately encode s * using a wordlevel transformer-encoder encoder word and each noisy spelling sequence c * k of token k via a character-level transformer-encoder encoder char . for encoder word , we replace non-word misspellings, i.e. oov words, with a unk token. for encoder char , we treat each character as a token and each word as a "sentence", so each word's character sequence embedding h k char is independent of each other. since the transformer-encoder @xcite computes contextualized token representations, we take h char , the [cls] token representation of each character sequence as the local character-level representation of s * . finally, we jointly predict s by concatenating the local and global context representations. subword encoder. alternatively, we use subword tokenization to simultaneously address the spelling and context information. formally, as shown in equation 2, given a noisy subword token sequence s * sub , we encode it using a transformerencoder encoder sub and simply use an affine layer to predict the sequence of each subword token's corresponding correct word token s sub in bio2 tagging scheme @xcite . we use cross-entropy loss as our training objective. finally, in addition to the natural misspelling noise, we apply a synthetic character-level noise to the training set to enhance the model's robustness to unseen misspelling patterns. the details will be introduced in section 3.1. ## related work and background many approaches are proposed for spelling correction @xcite @xcite @xcite @xcite , such as edit-distance based approaches @xcite @xcite @xcite @xcite , approaches based on statistical machine translation @xcite @xcite , and spelling correction for user search queries @xcite . most of them do not use contextual information, and some use simple contextual features @xcite @xcite . in recent years, there are some attempts to develop better spelling correction algorithms based on neural nets @xcite . similar to our baselines scrnn @xcite and mude @xcite , @xcite proposed a nested rnn to hierarchically encode characters to word representations, then correct each word using a nested gru @xcite . however, these previous works either only train models on natural misspellings @xcite or synthetic misspellings @xcite , and only focus on denoising the input texts from orthographic perspective without leveraging the retained semantics of the noisy input. on the other hand, tal weiss proposed deep spelling @xcite , which uses the sequence-tosequence architecture @xcite to generate corrected sentences. note that deep spelling is essentially not a spelling corrector since spelling correction must focus only on the misspelled words, not on transforming the whole sentences. for similar reasons, spelling correction is also different from gec (grammar error correction) @xcite . as a background, recently pre-trained neural lms @xcite @xcite @xcite trained on large corpus on various pre-training tasks have made an enormous success on various benchmarks. these lms captures the probability of a word or a sentence given their context, which plays a crucial role in correcting real-word misspellings. however, all of the lms mentioned are based on subword embeddings, such as wordpiece @xcite or byte pair encoding @xcite to avoid oov words. ## conclusion we leverage novel approaches to combine spelling and context information for stand-alone spelling correction, and achieved state-of-the-art performance. our experiments gives insights on how to build a strong stand-alone spelling corrector: (1) combine both spelling and context information, (2) leverage a pre-trained lm and (3) use the synthetic character-level noise.