N eu S pell: A Neural Spelling Correction Toolkit (2020)

Abstract: We introduce NeuSpell, an open-source toolkit for spelling correction in English. Our toolkit comprises ten different models, and benchmarks them on naturally occurring misspellings from multiple sources. We find that many systems do not adequately leverage the context around the misspelt token. To remedy this, (i) we train neural models using spelling errors in context, synthetically constructed by reverse engineering isolated misspellings; and (ii) use richer representations of the context. By training on our synthetic examples, correction rates improve by 9% (absolute) compared to the case when models are trained on randomly sampled character perturbations. Using richer contextual representations boosts the correction rate by another 3%. Our toolkit enables practitioners to use our proposed and existing spelling correction systems, both via a simple unified command line, as well as a web interface. Among many potential applications, we demonstrate the utility of our spell-checkers in combating adversarial misspellings. The toolkit can be accessed at neuspell.github.io.

## introduction spelling mistakes constitute the largest share of errors in written text @xcite . therefore, spell checkers are ubiquitous, forming an integral part of many applications including search engines, productivity and collaboration tools, messaging platforms, etc. however, many well performing spelling correction systems are developed by corporations, trained on massive proprietary user data. in contrast, many freely available off-the-shelf correctors such as enchant @xcite , gnu aspell @xcite , and jamspell (ozinov, 2019), do not effectively use the context of the misspelled word. for instance, they fail to disambiguate :::::: in this paper, we describe our spelling correction toolkit, which comprises of several neural models that accurately capture context around the misspellings. to train our neural spell correctors, we first curate synthetic training data for spelling correction in context, using several text noising strategies. these strategies use a lookup table for wordlevel noising, and a context-based character-level confusion dictionary for character-level noising. to populate this lookup table and confusion matrix, we harvest isolated misspelling-correction pairs from various publicly available sources. further, we investigate effective ways to incorporate contextual information: we experiment with contextual representations from pretrained models such as elmo @xcite and bert @xcite and compare their efficacies with existing neural architectural choices ( § 5.1). lastly, several recent studies have shown that many state-of-the-art neural models developed for a variety of natural language processing (nlp) tasks easily break in the presence of natural or synthetic spelling errors @xcite @xcite . we determine the usefulness of our toolkit as a countermeasure against character-level adversarial attacks ( § 5.2). we find that our models are better defenses to adversarial attacks than previously proposed spell checkers. we believe that our toolkit would encourage practitioners to incorporate spelling correction systems in other nlp applications. ## models in neuspell our toolkit offers ten different spelling correction models, which include: (i) two off-the-shelf nonneural models, (ii) four published neural models for spelling correction, (iii) four of our extensions. the details of first six systems are following: to better capture the context around a misspelt token, we extend the sc-lstm model by augmenting it with deep contextual representations from pre-trained elmo and bert. since the best point to integrate such embeddings might vary by task @xcite , we append them either to semi-character embeddings before feeding them to the bilstm or to the bilstm's output. currently, our toolkit provides four such trained models: elmo/bert tied at input/output with a semicharacter based bi-lstm model. implementation details neural models in neuspell are trained by posing spelling correction as a sequence labeling task, where a correct word is marked as itself and a misspelt token is labeled as its correction. out-of-vocabulary labels are marked as unk. for each word in the input text sequence, models are trained to output a probability distribution over a finite vocabulary using a softmax layer. we set the hidden size of the bi-lstm network in all models to 512 and use {50,100,100,100} sized convolution filters with lengths {2,3,4,5} respectively in cnns. we use a dropout of 0.4 on the bi-lstm's outputs and train the models using cross-entropy loss. we use the bertadam 5 optimizer for models with a bert component and the adam (kingma and ba, 2014) optimizer for the remainder. these optimizers are used with default parameter settings. we use a batch size of 32 examples, and train with a patience of 3 epochs. during inference, we first replace unk predictions with their corresponding input words and then evaluate the results. we evaluate models for accuracy (percentage of correct words among all words) and word correction rate (percentage of misspelt tokens corrected). we use allennlp foot_5 and huggingface foot_6 libraries to use elmo and bert respectively. all neural models in our toolkit are implemented using the pytorch library @xcite , and are compatible to run on both cpu and gpu environments. performance of different models are presented in ## synthetic training datasets due to scarcity of available parallel data for spelling correction, we noise sentences to generate misspelt-correct sentence pairs. we use 1.6m sentences from the one billion word benchmark @xcite dataset as our clean corpus. using different noising strategies from existing literature, we noise ∼20% of the tokens in the clean corpus by injecting spelling mistakes in each sentence. below, we briefly describe these strategies. random: following @xcite , this noising strategy involves four character-level operations: permute, delete, insert and replace. we manipulate only the internal characters of a word. the permute operation jumbles a pair of consecutive characters, delete operation randomly deletes one of the characters, insert operation randomly inserts an alphabet and replace operation swaps a character with a randomly selected alphabet. for every word in the clean corpus, we select one of the four operations with 0.1 probability each. we do not modify words of length three or smaller. word: inspired from belinkov and bisk (2017), we swap a word with its noised counterpart from a pre-built lookup table. we collect 109k misspeltcorrect word pairs for 17k popular english words from a variety of public sources. 8 for every word in the clean corpus, we replace it by a random misspelling (with a probability of 0.3) sampled from all the misspellings associated with that word in the lookup table. words not present in the lookup table are left as is. prob: recently, @xcite released a corpus of 20m correct-misspelt word pairs, generated from logs of a search engine. 9 we use this corpus to construct a character-level confusion dictionary where the keys are character, context pairs and the values are a list of potential character replacements with their frequencies. this dictionary is subsequently used to sample character-level errors in a given context. we use a context of 3 characters, and backoff to 2, 1, and 0 characters. notably, due to the large number of unedited characters in the corpus, the most probable replacement will often be the same as the source character. prob+word: for this strategy, we simply concatenate the training data obtained from both word and prob strategies. ## evaluation benchmarks natural misspellings in context many publicly available spell-checkers correctors evaluate on isolated misspellings @xcite @xcite . whereas, we evaluate our systems using misspellings in context, by using publicly available datasets for the task of grammatical error correction (gec). since the gec datasets are annotated for various types of grammatical mistakes, we only sample errors of spell type. among the gec datasets in bea-2019 shared task foot_9 , the write & improve (w&i) dataset along with the locness dataset are a collection of texts in english (mainly essays) written by language learners with varying proficiency levels @xcite . the first certificate in english (fce) dataset is another collection of essays in english written by non-native learners taking a language assessment exam @xcite and the lang-8 dataset is a collection of english texts from lang-8 online language learning website @xcite . we combine data from these four sources to create the bea-60k test set with nearly 70k spelling mistakes (6.8% of all tokens) in 63044 sentences.