Robust Text Classification using Sub-Word Information in Input Word Representations. (2019)

Abstract: Word based deep learning approaches have been used with increasing success recently to solve Natural Language Processing problems like Machine Translation, Language Modelling and Text Classification. However, performance of these word based models is limited by the vocabulary of the training corpus. Alternate approaches using character based models have been proposed to overcome the unseen word problems arising for a variety of reasons. However, character based models fail to capture the sequential relationship of words inherently present in texts. Hence, there is scope for improvement by addressing the unseen word problem while also maintaining the sequential context through word based models. In this work, we propose a method where the input embedding vector incorporates sub-word information but is also suitable for use with models which successfully capture the sequential nature of text. We further attempt to establish that using such a word representation as input makes the model robust to unseen words, particularly arising due to tokenization and spelling errors, which is a common problem in systems where a typing interface is one of the input modalities.

## introduction recent research has demonstrated the success of word based models for nlp problems like machine translation @xcite and text classification @xcite . it is well established in literature that the dictionary of words contained in the training corpus have significant bearing on the performance of these models. for example, in the case of language modelling, an unseen word can never be predicted and models also tend to have lower accuracies when predicting words in the vicinity of an unseen word. models for text classification also suffer from a similar problem wherein one or more unseen words in the input may significantly increase classification error. alternate approaches using character based models @xcite have been proposed to overcome the unseen word problem which ails word-based deep learning networks. however, character based models fail to capture the sequential relationship of words inherently present in texts. the main contribution of this paper is to establish the suitability and robustness of an input embedded vector which incorporates sub-word information @xcite with a recurrent neural network model for sentence classification and also establish the capability of such a configuration to deal effectively with the unseen word problem, especially arising due to word segmentations and spelling errors. ## related work the input to text based deep learning models is usually a numeric vector representation of text, commonly called embedded vectors. the embedded vector of each word is designed to be indicative of its semantic relationship with other words or characters as available in the corpus in embedded space. this usually constitutes the very first layer of the network. this layer may be initialized randomly or with pre-trained vectors the pre-trained vectors may be static or may also be learned with the network. these pre-trained vectors are typically generated from a large training corpus which is usually not directly related to the problem at hand, but is representative of language as a whole. one of the most commonly used pre-trained embedding is proposed by @xcite where a neural network approach is used to generate word vectors based on a 1.6 billion words data set. an alternate approach discussed in @xcite focusses on whether distributional word representations are best learned from count-based methods rather than prediction-based methods. since these vectors are pre-trained using large corpora, they contain meaningful semantic representations of even words not seen in the training corpus for the specific problem, which helps deal with the unseen word problem to a certain extent. in addition, models tend to converge faster when pre-trained vectors are used. however, pre-trained word embedding approaches continue to have difficulty with words not in the dictionary of the input embedding. also, rare words are often not represented as well as more frequently occurring words. words not seen in the training corpus are usually either marked as unknown (unk) or excluded altogether from the input. to address these drawbacks, several alternate approaches have been proposed. @xcite proposed a 9layer character based cnn model which addresses the unseen word problem in word based models. however, this cnn based approach fails to capture sequence context features in the text. @xcite proposed an architecture in which the character embedding is input to a cnn, the output of which acts as input to an rnn. in such models, the cnn component captures the n-gram features of text and rnn takes care of sequence context of such features in the text. for character based cnn models, the context or relationship between multiple characters and words are captured by convolution filters or kernels. a set of fixed filter sizes (n-grams) may not completely capture wordlevel information. also, capturing longer context is difficult in cnn models. another alternate approach is proposed by @xcite where each word is represented as a bag of character ngrams and a vector representation is associated to each character n-gram. words are represented as a sum of these representations. this is found to be especially effective when dealing with morphologically rich languages. this has been used with shallow models for sentiment analysis and tag identification problem in @xcite . however, for more complex problems over a larger number of classes, the higher representational power of deep networks such as rnns and cnns may be desired. in this work, we apply the method for generating vector representations proposed in @xcite to deep learning networks such as the architecture proposed in @xcite and explore the extent to which unseen word problem, especially arising due to misspellings and tokenization errors, is addressed. ## proposed approach unseen words are a common occurrence in nlp problems and arise from a variety of situations. the most common reasons for unseen words is simply a lack of exhaustive training data for a specific problem. this problem is largely dealt with by using pre-trained distributions trained on a large corpus. another common source of unseen words is morphological variance. this is a scenario where the unseen word is close to a seen word both superficially and semantically. research described in @xcite and @xcite show that input vectors incorporating sub-word information have proved effective in tackling this problem. another source of unseen words are misspellings or incorrect word segmentation. this is a common problem faced in multi-modal applications such as voice assistants wherein one of the input modalities is a typing interface. these types of errors seem to be similar to the unks arising from morphological variance wherein the unseen word shares a close superficial as well as semantic similarity with a seen word. we propose to use pre-trained embedded vectors to deal with the unseen word problem, especially due to misspellings, using word vectors which incorporate sub-word information. an rnn based sentence classifier with an architecture similar to the one proposed by @xcite is used and compared with the performance of the distributions described by @xcite and @xcite on standard data sets for text classification. in addition, we intend to simulate the unk problem due to misspellings and incorrect tokenization by applying rules to the standard data sets. these rules consist of common misspellings such as "ei" instead of "ie", incorrect double consonants ('aggressive' vs 'aggresive') and so on. ## datasets and experimental setup the sentence classification task is chosen for the work described in this paper. in our first set of experiments, we apply the reference word-based sequence learning architecture of @xcite to the sentence classification problem on three standard datasets. the pre-trained embedding learnt through the method proposed in @xcite is used as a static input embedding and is not updated as part of the training for the specific problem. we compare the performance of these sub-word based input embedded vectors with the popular glove and word2vec pre-trained word representations using this architecture. this is used to establish the suitability of the word representations of @xcite to deep learning architectures. in our second set of experiments, we apply the reference word-based sequence learning architecture of @xcite to the sentence classification problem on three standard datasets which are modified to incorporate misspellings. two standard misspelling dictionaries are used to generate the misspelled versions of the standard datasets. this is done in order to simulate realworld situations, such as multi-modal smart assistants, where the input to a sentence classification system may be via a textual input interface, and therefore prone to misspellings. the performance of sub-word based word vectors @xcite is compared with reference distributions. this is used to establish that word representations which are constructed using subword information are robust and more suitable for use in multi-modal commercial applications than the more popular glove and word2vec word representations. the datasets used for these experiments are described in detail in the following sections. ## results the reference model of @xcite described in the sections above with glove and word2vec embedding vectors used as the input word vectors has been compared with the proposed word embedding on the three standard sentence classification datasets as described in the the main purpose of this experiment is to prove the performance of the proposed word embedding using sub-words with a word-based rnn sequence learning architecture. the performance of the proposed input word embedding applied to the reference architecture is comparable to the performance with the pennington (2014) and @xcite input embedded vectors. this illustrates the suitability of the sub-word embedding for use with deep neural networks. the accuracies shown in the wikipedia misspelling dictionary mainly focusses on typographic misspellings. the different datasets are also differently prone to spelling errors. in the case of subj dataset, 964 out of 1000 test sentences are modified, but a majority of these misspellings are words like 'the' and 'and', which are typically less likely to affect the classification result. however, 40% of the mpqa test set is modified and about 29% of the ag news data set is modified by the wikipedia misspelling dictionary. this serves to illustrate the need to deal with misspellings as part of any commercial application. the results in the results in in most of the cases, only a minor drop in accuracy is observed ranging from 0.7% to 2.4%. the only outlier is the wikipedia dictionary modified subj dataset where a significant drop of over 4.7% is seen. detailed analysis shows that the most common spelling modifications in this dataset are the words 'the' and 'and' which the subword based representation doesn't deal with well as the number of sub-words for very short words are too less to have a significant impact in generating the word vectors. some more specific situations which are not handled well by the proposed approach are discussed in the following section along with the direction our future work will take to address these problems. overall, the performance of the proposed approach is close enough to the performance on the original datasets without misspellings to indicate that the proposed approach is not only comparable to the state-of-the-art when applied to deep learning architectures but also solves the unk problems arising due to typographic and phonetic misspellings to a significant extent. ## discussion it is seen that 7% of the attendees of the toefl 3 examination, a test of english, tend to make spelling errors, even in an environment where the sole focus is correctness of grammar and language. our study of internal data from a voice assistant applications indicates that in excess of 25% of all data input using a typing interface contains errors in spelling and word breaks. this illustrates the need for a method to handle spelling errors gracefully and reliably, especially for more natural ai applications. the major motivation to conduct the investigations presented in this work was to come up with a technique to deal with the misspelling problem which is inherently present in multimodal voice assistants where the primary input paradigm is speech, which is not prone to misspellings at all, and the secondary modality is a typing interface which is quite prone to spelling and word segmentation errors. the goal was to use a technique wherein the models trained on wellformed data are robust to errors in spelling rather than to implement a relatively clumsy rule-based preprocessing module which would attempt to datasets best accuracy (without misspelling) proposed approach (wikipedia misspelling) proposed approach (aspell misspelling) subj 86 81.3 84.58 mpqa 89.8 87.3 89 ag news 87.5 86.3 86.79 a study of the misspelling dictionaries and the substitutions made to the standard datasets using these dictionaries gives further clarity on the various types of spelling errors commonly seen. the first level classification of the types of spelling errors is typographical and phonetic. typographical errors consist mainly of omission, addition or swapping of characters. all these three cases seem to be handled reasonably well using the sub-words approach to construct word vectors. the second major category of phonetic misspellings consists mainly of replacement of characters by other similar characters such as 'destruction' vs 'distruction' and so on. other common errors of this kind are incorrect usage of double consonants, 'ei' instead of 'ie' and so on. the majority of these cases are also handled well by the sub-word based word representations. one of the observations while analyzing the drop in accuracy of the models on the misspelled datasets is that misspellings towards the middle of the word are not dealt with as well as misspellings near either end of the words since more number of sub-words are affected in this case. we are currently working on some improvements to the word representations to overcome this problem. ## conclusion the work in this paper demonstrates that input embedded vectors which incorporate sub-word information and are learnt through a shallow network are well-suited for use with sequence aware deep learning networks. it also showcases the effectiveness of such a configuration in dealing with various common types of spelling errors arising due to both typographic as well as due to phonetic reasons. the results showing that the accuracy of the proposed configuration on the standard datasets with misspellings is comparable to the best performance on the misspelling free datasets indicate that the proposed configuration almost entirely solves the problem of spelling errors. the proposed work is especially suited for use is multi-modal applications as it not only seamlessly handles spelling errors but performs as well as state-of-the-art systems on correctly spelled inputs. one example of such a real-world application is a multi-modal voice assistant which allows textual input in addition to speech input. modern multi-modal voice assistants attempt to support a very wide range of complex functionality for which deep learning networks are a natural choice and will greatly benefit from an input embedding which seamlessly handles misspellings. moreover, the approach used to construct these input embedded vectors also handles morphological variance and is applicable across languages. this work also establishes the similarity in nature between morphological variance and spelling or tokenization errors wherein the unseen word is both semantically and superficially similar to an actual seen word, and therefore improvements made in dealing with one are likely to be beneficial in dealing with the other. this opens up the possibility of a wide area of research as this work proves a significant overlap between two problem statements which were hitherto perceived to be different.