Does a Hybrid Neural Network based Feature Selection Model Improve Text Classification? (2020)

Abstract: Text classification is a fundamental problem in the field of natural language processing. Text classification mainly focuses on giving more importance to all the relevant features that help classify the textual data. Apart from these, the text can have redundant or highly correlated features. These features increase the complexity of the classification algorithm. Thus, many dimensionality reduction methods were proposed with the traditional machine learning classifiers. The use of dimensionality reduction methods with machine learning classifiers has achieved good results. In this paper, we propose a hybrid feature selection method for obtaining relevant features by combining various filter-based feature selection methods and fastText classifier. We then present three ways of implementing a feature selection and neural network pipeline. We observed a reduction in training time when feature selection methods are used along with neural networks. We also observed a slight increase in accuracy on some datasets.

## introduction text classification assigns one or more class labels from a predefined set to a document based on its content. text classification has broad applications in real-world scenarios such as document categorization, news filtering, spam detection, optical character recognition (ocr), and intent recognition. giving high weights to relevant features is the objective of text classification. the field of text classification has gained more interest during the machine learning (ml) era. many discriminative and generative machine learning classifiers have achieved excellent results in the field of text classification @xcite . feature selection and feature extraction methods are often used to reduce high dimensionality @xcite . feature extraction generates features from text @xcite . feature selection (fs) selects the most prominent features @xcite . these feature selection and extraction methods are used along with traditional classification algorithms. these methods reduced the curse of dimensionality and increased the classification accuracy @xcite . recently, deep learning models are used to learn better text representations and to classify the text @xcite . such models include convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite , transformer models @xcite , and graph convolutional networks (gcn) @xcite . these nn models capture semantic and syntactic information in local and global word sequences. even though the neural networks capture a complex and dense representation of data, the set of words introducing noise in the classifier is still present. such words add the burden of increased vocabulary, which results in increased textual representation and an increase in the training time of the classifiers @xcite . similar to the traditional approaches, we want to understand the effects of using statistical feature selection algorithms beforehand to calculate the features' relevance and then train a fasttext textclassification algorithm on those relevant features. using this feature selection and neural network pipeline, we assume that the complexity of dealing with larger vocabulary decreases. including feature selection with fasttext text-classification helps reduce the classifier's training time and helps the classifier reach better local optima, showing a significant increase in classification accuracy. in this work, we analyzed a feature selection and neural network pipeline for text classification. we used a hybrid feature selection method to get a score on relevant features. using this score, we formulated three methods. the first and second methods deal with modifying the original text by extracting the relevant features. the third method deals with using the feature selection scores and pass it along with the word embeddings. we then observed the effect of feature selection on various neural networks. the rest of the paper is organized as follows. section 2 gives a brief review of previous works in the field of feature selection and text classification. section 3 presents a detailed procedure of the proposed pipeline and presents the experiments and datasets used for our study. section 4 reports the performance of text classifiers with and without feature selection methods. section 5 concludes the paper. ## literature survey this section presents a brief description of the neural network (nn) classification algorithms and various feature selection methods. ## proposed pipeline in this section, we present our feature selection and neural network pipeline. the feature selection and neural network pipeline start with selecting a good tokenizer to tokenize the data and create a feature set. the tokenizer used for our feature selection is the sentencepiece tokenizer @xcite . sentencepiece tokenizer implements subword units by using byte-pair-encoding (bpe) @xcite and unigram language model @xcite . in the feature subset generation, we considered a hybrid feature selection method known as hy-brid @xcite . it has proved that a combination of the features selected by various methods is more effective and computationally faster than the features selected by individual filter and wrapper methods. similar to the hybrid model, we used three filters to obtain the relevancy score. the filters we considered were chi2, anova-f, and mi. these filters calculate the relevancy between the word and the class labels. before feature selection, we used the bag-of-words(bow) model to vectorize the data. in the bow model, each feature vector is represented by t f idf scores. then we used statistical measures such as χ 2 , anova-f, and mi for obtaining feature scores. analysis of variance (anova 2 ) is a statistical method used to check the means of two or more groups that are significantly different from each other. mutual information (mi 3 ) is frequently used to measure the mutual dependency between two variables. using different statistical methods, we measured the relevance of each feature. we then aggregate the relevance scores of all satistical methods for each feature. the relevance of a feature x i is given by, instead of an lr classifier given in the hybrid model, we used the fasttext classifier @xcite for the feature selection. we used the fast-text classifier as it is often on par with deep learning classifiers in terms of accuracy and performs faster computations. the fasttext classifier treats the average of word embeddings as document embeddings, then feeds document embeddings into a feed-forward nn or a multinomial lr classifier. we used pre-trained fasttext word embeddings @xcite while training a classifier. to get the final features list, we sorted the normalized, aggregated value in descending order and divided the entire feature space into k sets. in our model, we divided the sorted feature space into 20 sets. the value of k is fixed to 20 using a trial and error basis. we take the first set as the vocabulary of the classifier. we then trained the classifier and noted its accuracy. in the second iteration, we considered the vocabulary as the combination of first 1 a detailed explanation and a simple example of χ 2 is given at https://www.mathsisfun.com/data/chi-squaretest.html 2 a detailed explanation for anova is given in https://towardsdatascience.com/anova-for-feature-selectionin-machine-learning-d9305e228476. according to the proposed fs method, the final feature set is considered relevant, and they are necessary to perform the text classification. in contrast, the other features have little to no effect on the text classification or might degrade the classifier's performance. after feature subset generation, we propose three methods for including the feature selection information before training the neural network classifiers. we felt that removing the features given low rank by our fs method might disturb the original data's grammatical structure, thus disturbing the word to word dependencies. we masked the low ranked words with the help @xmath0. < m @xmath1, and pos preserves the word's part of speech. the visual representation of method 2 is shown in figure 1 3. method 3 (meta embeddings): as shown in figure 2 , we pass the relevancy and feature selection information along with embeddings in this method. each slot holds the filter scores, i.e., chi, anova, mi scores of each feature. the last slot holds a 1 or 0 value. 1 is used for the selected features, and 0 is used for low ranked features that were not selected by our hybrid feature selection approach. we analyzed and evaluated the above methods with various state-of-the-art nn classifiers on the benchmark datasets. ## conclusion in our work, "does a hybrid nn fs model improve text classification?", we used the nn based hybrid fs method to extract relevant features and used nn classifiers for text classification. we extracted the relevant or high ranked features using filter-based methods and a fasttext classifier. we then proposed three methods on how the feature selection can be included in the nn classification process. first, modifying the corpus by considering only relevant features. second, modifying the data by masking the low ranked features, and the third method introduces feature selection information along with word embeddings. we observed that method 1 had shown a significant reduction in training time when large datasets or slower models are used, accompanied by a minimal change in classification accuracy. by introducing m ask + p 0s(word), we inferred that the masked word was a burden to the classifier, and it always tried to adjust the word embeddings, which resulted in increased epoch time during training and a slightly negative effect on classification accuracy. whereas method 3 has shown no effect on decreasing the training time, it has shown a maximum of 4% increase in the classification accuracy compared to baseline. it proved that introducing feature scores along with pre-trained word embeddings while training the nn classifier is beneficial. instead of opting for random naive vocabulary reduction techniques such as using min df and max df (minimum and maximum document frequency) for selecting features, by using fs methods, we can calculate the relevance of the word beforehand and use that as metadata to the nn classifier. when the datasets are huge, these methods are of more significance. we can use the modified data while tuning the hyperparameters. then we can use the real data to train and evaluate the model. even in the critical domain datasets such as "medical", we cannot rely on removing a word based on min df and max df scores. each word in those datasets should be treated with utmost significance. fs methods help in such scenarios by calculating the word's relevance and helps maintain better vocabulary before training neural network classifiers.