Hierarchical Character Tagger for Short Text Spelling Error Correction (2021)

Abstract: State-of-the-art approaches to spelling error correction problem include Transformer-based Seq2Seq models, which require large training sets and suffer from slow inference time; and sequence labeling models based on Transformer encoders like BERT, which involve token-level label space and therefore a large pre-defined vocabulary dictionary. In this paper we present a Hierarchical Character Tagger model, or HCTagger, for short text spelling error correction. We use a pre-trained language model at the character level as a text encoder, and then predict character-level edits to transform the original text into its error-free form with a much smaller label space. For decoding, we propose a hierarchical multi-task approach to alleviate the issue of long-tail label distribution without introducing extra model parameters. Experiments on two public misspelling correction datasets demonstrate that HCTagger is an accurate and much faster approach than many existing models.

## introduction a spelling corrector is an important and universal tool for a wide range of text-related applications, such as search engines, machine translation, optical character recognition, medical records, text processors and essay scoring. although spelling error correction is a long-studied problem, it remains a challenging task because words can be misspelled in a variety of forms, including in-word errors, cross-word errors, non-word errors and realword errors, depending on the subtle contextual information. in this paper, we focus on solving the spelling correction problem in user-generated short text, such as queries in search engines or tweets on social media, which has three unique properties compared to long essays. first, search queries or tweets are often short and lack context. second, most short text contains pure spelling errors and almost no grammatical errors. third, instant spell checkers used in search engines or social medias have strict latency requirements. in general, popular approaches to spelling correction make use of parallel corpora in which the source sentence contains spelling errors and the target sentence is error-free. recently, the transformer-based sequence-tosequence (seq2seq) model @xcite has gradually proven to be effective on spelling correction problems. unlike neural machine translation, spelling errors tend to occur locally for a few characters while the rest of the text is correct. to cope with this situation, zhao et al. propose a scheme to incorporate a copy mechanism in seq2seq. the success of this type of seq2seq model depends on large scale annotated datasets, which are often generated by constructing text noise from clean text in previous studies. moreover, this approach suffers from slower inference time and lack of interpretability. another class of approaches is based on sequence labeling. instead of generating the output sequence in an autoregressive fashion, pie @xcite and gector @xcite predicts token-level edit operations in one of {keep, delete, replace, append} by leveraging pre-trained transformer encoders, such as bert @xcite . such models can generate the outputs for all tokens in parallel, and therefore significantly reduces the latency of sequential decoding as in seq2seq models while achieving comparable accuracy. however, the approaches in both papers predict edit operations at token level. it can be expected that the replace and append operations are associated with a huge pre-defined vocabulary dictionary. for real-life usages it is infeasible to enumerate all correctly spelled words in the label space. to address the aforementioned shortcomings, considering the unique properties of short text misspelling correction, in this paper, we propose a new model called hierarchical character tagger, or hctagger for short, which uses a pre-trained language model at the character level as a text encoder, and then predicts character-level edits. it is motivated by the straightforward observation that spelling errors usually occur at character level. for the misspelling-correction pair of @xmath0, its character-level edit labels would be [s: keep, h: keep, i: replace with o, e: keep, s: keep], which is represented in a much smaller label space compared to [shies: replace with shoes] at token level. while most spelling errors occur within 1edit distance for each token, for broader coverage we also include character sequence edit operations like replace with oa. furthermore, the distribution of edit labels is long-tailed. as shown in figure 1 , keep and delete are more frequent than labels of replace/append with certain character(s). if these labels are treated as equivalent, the overall accuracy of the model will be constrained by the unevenness of the label distribution. therefore, for decoding, we aggregate original fine-grained edit labels into four coarsegrained labels @xcite . we propose a hierarchical multi-task approach to learn both the fine-grained and coarse-grained edit labels at the same time without introducing any extra model parameters. through extensive experiments on two public datasets, we demonstrate that our proposed hc-tagger effectively improves the performance and latency of short text spelling correction. ## approach we describe our model hctagger in this section. ## experiments in this section, we describe the experiments performed on two public datasets for hctagger. meanwhile, we compare it with several state-ofthe-art baselines. ## related works hasan et al. use character-based statistical machine translation to correct user queries in the ecommerce domain. they extract training data from query refinement logs, and evaluate the results on an internal dataset. grammar error correction (gec) is an extensively researched nlp task. this task contains grammar errors, including spelling, punctuation, grammatical, and word choice errors. pie @xcite and gector @xcite are the state-of-the-art models that predict token-level edit operations {keep, delete, replace, insert} by leveraging pre-trained transformer encoders like bert. however, their models are not specifically designed for correcting spelling errors, which most often occur at character level. they rely on a small (∼1k) pre-defined token-level replace and insert dictionary. including all correctlyspelled tokens in the dictionary will make the label space too large. transformer based seq2seq models @xcite prove to be successful on grammar error corrections, but heavily depends on synthetically generated error datasets. character based seq2seq models @xcite are also explored. such model architectures involve a separate autoregressive decoder and attention module, which makes the inference time much slower. in particular for spelling error correction task, where the misspelling and correction only differ by one or more characters, seq2seq models seem too heavy. neuspell @xcite is a spelling correction toolkit, which implements a wide range of models like sc-elmo-lstm and bert. they regard spelling correction as a token-level sequence labeling task, where the output for each token is its error-free form. for each word in the input text sequence, models are trained to output a probability distribution over a finite vocabulary. besides the aforementioned excessive label space problem at token-level, another shortcoming of this toolkit is that it assumes the misspelled and correction sentences have exactly the same number of tokens. therefore, cross-word errors such as power @xmath1. ribeiro et al. treat spelling correction as a character-level local sequence transduction task by first predicting insertion slots in the input using learned insertion patterns, and then using a sequence labeling task to output tokens or a special token delete. they maintain a dictionary to keep track of the insertion context. for example, letter a is inserted frequently after letter s. while our pretrained language model layer implicitly encodes such insertion context without the need of keeping a dictionary. ## conclusions we presented the hierarchical character tagger to correct user-generated short text misspellings. hctagger predicts character-level edits, which has smaller label space than token-level edits. pretrained character-level language model embedding that we use is lightweight and much faster than bert-like text encoders in many other state-of-theart models, while achieving similar or even higher accuracy for short text spelling error correction task. moreover, our novel hierarchical multi-task decoding approach can be extended to any scenario that contains a hierarchical long-tail distributed label space.