BS pell: A CNN -Blended BERT Based B angla Spell Checker (2023)

Abstract: Bangla typing is mostly performed using English keyboard and can be highly erroneous due to the presence of compound and similarly pronounced letters. Spelling correction of a misspelled word requires understanding of word typing pattern as well as the context of the word usage. A specialized BERT model named BSpell has been proposed in this paper targeted towards word for word correction in sentence level. BSpell contains an end-to-end trainable CNN sub-model named SemanticNet along with specialized auxiliary loss. This allows BSpell to specialize in highly inflected Bangla vocabulary in the presence of spelling errors. Furthermore, a hybrid pretraining scheme has been proposed for BSpell that combines word level and character level masking. Comparison on two Bangla and one Hindi spelling correction dataset shows the superiority of our proposed approach.

## introduction bangla is the native language of 228 million people which makes it the sixth most spoken language in the world foot_0 . this sanskrit originated language has 11 vowels, 39 consonants, 11 modified vowels and 170 compound characters @xcite . there is vast difference between bangla grapheme representation and phonetic utterance for many commonly used words. as a result, fast typing of bangla yields frequent spelling mistakes. almost all bangla native speakers type using english qwerty layout keyboard @xcite ) which makes it difficult to type bangla compound characters, phonetically similar single characters and similar pronounced modified vowels correctly. thus bangla typing speed, if error-free typing is desired, is slow. an accurate spell checker (sc) can be a solution to this problem. existing bangla scs include phonetic rule (uz- @xcite and clustering based methods @xcite . these methods do not take misspelled word context into consideration. another n-gram based bangla sc @xcite takes only short range previous context into consideration. recent state-of-the-art (sota) spell checkers have been developed for chinese language, where a character level confusion set (similar characters) guided sequence to sequence (seq2seq) model has been proposed by @xcite . another research used similarity mapping graph convolutional network in order to guide bert based character by character parallel correction @xcite . both these methods require external knowledge and assumption about confusing character pairs existing in the language. the most recent chinese sc offers an assumption free bert architecture where error detection network based soft-masking is included @xcite . this model takes all n characters of a sentence as input and produces the correct version of these n characters as output in a parallel manner. figure 1 : heterogeneous character number between error word and corresponding correctly spelled word one of the limitations in developing bangla sc using sota bert based implementation @xcite is that number of input and output characters in bert has to be exactly the same. such scheme is only capable of correcting substitution type errors. as compound characters are common in bangla words, an error made due to the substitution of such characters also changes word length (see the table in figure 1 ). so, we introduce word level prediction in our proposed bert based model. the table shown in figure 2 illustrates the importance of context in bangla sc. although the red marked words of this figure are the misspelled versions of the corresponding green marked correct words, these red words are valid bangla words. but if we check these red words based on sentence semantic context, we can realize that these words have been produced accidentally because of spelling error. an effective sc has to consider word pattern, its prior context and its post context. spelling errors often span up to multiple words in a sentence. figure 3 provides an example where all four words have been misspelled. the correction of each word has context dependency on a few other words of the same sentence. the problem is that these words that form the correction context are also misspelled. the table in the figure shows the words to look at in order to correct each misspelled word. in the original sentence (colored in red), all these words that need to be looked at for context are misspelled. if a sc cannot understand the approximate underlying meaning of these misspelled words, then we lose all context for correcting each misspelled word which is undesirable. we propose a word level bert @xcite based model bspell. this model is capable of learning prior and post context dependency through the use of multi-head attention mechanism of stacked transformer encoders @xcite . the model uses cnn based learnable se-manticnet sub-model to capture semantic meaning of both correct and misspelled words. bspell also uses specialized auxiliary loss to facilitate word level pattern learning and vanishing gradient problem removal. we introduce hybrid pretrainingfor bspell to capture both context and word error pattern. we perform detailed evaluation on three error datasets that include a real life bangla error dataset. our evaluation includes detailed analysis on possible lstm based scs, sc variants of bert and existing classic bangla scs. ## related works several studies on bangla sc development have been conducted in spite of bangla being a low resource language. a phonetic encoding oriented bangla word level sc based on soundex algorithm was proposed by @xcite . this encoding scheme was later modified to develop a double metaphone encoding based bangla sc (uzzaman and @xcite . they took into account major context-sensitive rules and consonant clusters while performing their encoding scheme. another word level bangla sc able to handle both typographical and phonetic errors was proposed by @xcite . an n gram model was proposed by @xcite for checking sentence level bangla word correctness. an encoderdecoder based seq2seq model was proposed by @xcite for bangla sentence correction task which involved bad arrangement of words and missing words, though this work did not include incorrect spelling. a recent study has included hindi and telugu sc development, where mistakes are assumed to be made at character level @xcite . they have used attention based encoderdecoder modeling as their approach. sota research in this domain involves chinese scs as it is an error prone language due to its confusing word segmentation, phonetically and visually similar but semantically different characters. a seq2seq model assisted by a pointer network was employed for character level spell checking where the network is guided by externally generated character confusion set @xcite . another research incorporated phonological and visual similarity knowledge of chinese characters into bert based sc model by utilizing graph convolutional network @xcite . a recent bert based sc has taken advantage of gru (gated recurrent unit) based soft masking mechanism and has achieved sota performance in chinese character level sc in spite of not providing any external knowledge to the network @xcite . another external knowledge free approach namely faspell used bert based seq2seq model @xcite . hanspeller++ is notable among initially implemented chinese scs @xcite . it was an unified framework utilizing a hidden markov model. ## conclusion in this paper, we have proposed a sc named bspell for bangla and hindi language. bspell uses seman-ticvec representation of input misspelled words and a specialized auxiliary loss for the enhancement of spelling correction performance. the model exploits the concept of hybrid masking based pretraining. we have also investigated into the limitations of existing bangla scs as well as other sota scs proposed for high resource languages. bspell has two main limitations -(a) it cannot handle accidental merge or split of words and (b) it cannot correct misspelled rare words. a potential research direction can be to eradicate these limitations by designing models that can perform prediction at sub-word level which includes white space characters and punctuation marks.