Arabisc: Context-Sensitive Neural Spelling Checker (2020)

Abstract: Traditional statistical approaches to spelling correction usually consist of two consecutive processes — error detection and correction — and they are generally computationally intensive. Current state-of-the-art neural spelling correction models usually attempt to correct spelling errors directly over an entire sentence, which, as a consequence, lacks control of the process, e.g. they are prone to overcorrection. In recent years, recurrent neural networks (RNNs), in particular long short-term memory (LSTM) hidden units, have proven increasingly popular and powerful models for many natural language processing (NLP) problems. Accordingly, we made use of a bidirectional LSTM language model (LM) for our context-sensitive spelling detection and correction model which is shown to have much control over the correction process. While the use of LMs for spelling checking and correction is not new to this line of NLP research, our proposed approach makes better use of the rich neighbouring context, not only from before the word to be corrected, but also after it, via a dual-input deep LSTM network. Although in theory our proposed approach can be applied to any language, we carried out our experiments on Arabic, which we believe adds additional value given the fact that there are limited linguistic resources readily available in Arabic in comparison to many languages. Our experimental results demonstrate that the proposed methods are effective in both improving the quality of correction suggestions and minimising overcorrection.

## introduction misspelling detection or/and correction modules are seen as critical components of many real-world nlp applications. this has also been regarded as an important research area of nlp for years. the spelling errors are broadly classified into two categories: non-word errors (nwe), and real-word errors (rwe). if the misspelled string is a valid word of a language, it is called an rwe, otherwise it is an nwe @xcite . in this context, @xcite found that the rwe rate ranges from 2% for a small lexicon to 10% for a 50,000-word lexicon and almost 16% for a 350,000-word lexicon. in this work, we investigate both error types (i.e. rwe and nwe) with our context-aware spelling error detection and correction models. we demonstrate that our approach is capable of detecting and correcting both nwes and rwes in a text. as an illustration, we present two sentences that contain misspelled words below, with a justification of why context-sensitive error detection and correction could be an ideal solution for this problem. english: in the first english sentence, we can see that the word principle is a correct word that we can find in a dictionary; however, its use in this context is incorrect and the right word in this context is to be principal. hence, we can call this an rwe, and we can clearly see that this requires help from the neighbouring lexical contexts for error detection and correction. similarly, in the arabic example, we can see that the adjective (almthly) was incorrectly used instead of (almthla) to describe (altariq). like the error in the english example, this error requires the same treatment. traditional rule-based and statistical approaches to spelling correction rely on error detection first before offering correction suggestions. this minimises the chances of making unrequired corrections at least for common words. however, creating a good spelling checker using such traditional approaches involves building a large lexical database and thousands of human-generated rules for nwes, or large phrase tables for rwes @xcite . this, in effect, requires a lot of linguistic resources and tools as well as massive computing resources. many neural approaches @xcite to spelling checking normally correct errors directly over an entire input sentence. presenting an entire sentence to the network or decoder for correction involves the risk of modifying words that are correct in the context and should not be changed. for instance, the experiments carried out by @xcite demonstrate how neural spelling checking models can make overcorrection mistakes with examples. they categorise such errors as follows (q: input; a: ground truth; s: system output): in this work, we propose a context-sensitive neural model, arabisc, foot_0 which adds more control to the spelling correction process using language modelling, i.e. a many-to-one lstm network, and it consists of two processes: (i) identifying spelling errors, and (ii) offering correction suggestions. the idea is that we have to only correct the mistakes not the whole sentence. in other words, we combine the best of two worlds (statistical foot_1 and neural) i.e. we detect potential spelling mistakes and then offer diverse correction suggestions for the user to choose from in one go. 3 although we tested our method on standard arabic (fosha), it can theoretically be applied to any other language. the rest of this paper is organised as follows. section 2 elaborates on our methodology including the architecture of our proposed model. section 3 describes the experimental results and findings with some discussions, while section 4 concludes and suggests some avenues for future work. ## methodology the backbone of our approach involves building and using language modelling, i.e. a many-to-one lm for text generation. the task is to check a given input sentence word by word, predict the next word, and to find out whether the current word cw of the input sentence is in the list of high-scoring candidates b generated by the lm given the context of cw (previous and following words of cw). if cw is not in the list, correction suggestions are offered based on the edit distance @xcite between cw and the candidates in b. in our work, we compare two different models, namely: (i) a single-input model that uses only the preceding words of cw as context, and (ii) a dual-input model that uses the preceding and following words of cw as context. we describe our models in detail in the following section. ## conclusion in this paper, we presented a deep many-to-one neural network-based context-sensitive spelling checking and correction model. in short, we modelled words that come both before and after the word to be corrected as the conditional context in language model predictions. the experimental results suggest that our approach has achieved considerable success in terms of both offering better correction suggestions and minimising overcorrection. our project, arabisc, code, spelling correction models and data sets are now available as an open-source project via an open repository. 12 in the future, we plan to increase the training data size to see how our models will perform on a large-scale data set and more languages other than arabic. the state-of-the-art bidirectional encoder representation from transformers (bert) architecture @xcite ) makes use of transformer @xcite , an attention mechanism that learns contextual relations between words in a text and can offer powerful masked language modelling. as an alternative to our lstm lms, we plan to investigate using bert masked language models in arabisc. we evaluated our models on a test set that contains a small number of examples. in the future, we plan to increase the size of test set exam-12 https://github.com/ymoslem/arabisc ples. currently, our models operate at word level for spell-checking and correction. this could be an issue while encountering the out-of-vocabulary items. in the future, we aim to investigate applying byte-pair encoding @xcite or similar word-segmentation technique in our model.