Leveraging Semantic and Sentiment Knowledge for User-Generated Text Sentiment Classification (2022)

Abstract: Sentiment analysis is essential to process and understand unstructured user-generated content for better data analytics and decision-making. State-of-the-art techniques suffer from a high dimensional feature space because of noisy and irrelevant features from the noisy user-generated text. Our goal is to mitigate such problems using DNN-based text classification and popular word embeddings (Glove, fastText, and BERT) in conjunction with statistical filter feature selection (mRMR and PCA) to select relevant sentiment features and pick out unessential/irrelevant ones. We propose an effective way of integrating the traditional feature construction methods with the DNN-based methods to improve the performance of sentiment classification. We evaluate our model on three real-world benchmark datasets demonstrating that our proposed method improves the classification performance of several existing methods.

## introduction sentiment analysis is used to classify usergenerated review/comments into positive and negative classes, and widely applied to various domains such as businesses and organizations, politics, health, education, etc. existing proposals for text sentiment analysis can be mainly divided into lexicon-based and corpus-based approaches. sentiment lexicons may ignore important domainspecific sentiment words incurring concerns with word coverage. unlike lexicon-based approach, corpus-based approaches requires careful consideration of sentiment clues behind sentiment words, that is crucial for determining a text's sentiment orientation. we propose an effective method for improving sentence-level classification performance by integrating the traditional feature construction method with the dnn-based method, while considering semantics, context and sentiment clue. first, we parse the review sentences and employ linguistic rules to identify mixed opinionated sentences. then the pos tags are assigned to the sentiment bearing words: adjectives, adverbs, verbs, and nouns by stanford pos tagger. next we leverage the integrated wide coverage sentiment lexicon (wcsl) @xcite as the semantic and sentiment information to identify and extract sentiment bearing words. after that, we employ statistical features reduction algorithms namely mrmr (ding and peng) and pca @xcite for optimum features selection. further we process the optimum sentiment features and convert them to word vector by employing word embedding methods (e.g., glove, fasttext, and bert). finally, we apply a cnn classifier to process the word vector/vector embedding and predict the sentiment class of each sentence. our main contribution is summarized as follow: (1) we use semantic and sentiment knowledge, linguistic rules, and integrated wcsl to identify and extract the sentiment features in the sentence. (2) we reduce the dimensionality of feature space by employing the mrmr and pca statistical filter algorithms to filter out redundant features and select the optimum sentiment features. (3) the experimental results of our proposed method using three real word benchmark domain datasets show that the suggested sentiment analysis model improves the performance of several previous baseline methods significantly. ## related work many traditional feature-based machine learning methods have been largely used for textual sentiment classification @xcite @xcite . these approaches have employed bag of words, high order n-grams, part of speech (pos) patterns and linguistic patterns for sentiment features representation and sentiment classification. while traditional feature-based selection approaches might lower the text pre-processing pos tagger adjective adverb verb noun knowledge and embedding wcsl review text input data sentiment knowledge text processor context detection noise removal stemming lower case tokenization tfidf sentiment features extraction sentiment features selection optimum sentiment features word embedding glove fasttext embedding this movie has differe nt amazi ng charac ters output positive negative dimensionality reduction cnn convolutional layer (filters) max -pooling layer flatten layer fully connected layer class prediction phase 1 phase 2 phase 3 bert filter size: 3,4,5 sentence matrix 6x5 dimensionality of textual data to improve classification performance, classifiers still face sparsity issues due to a lack of adequate data representation strategies. word embeddings: word2vec, glove, fasttext, and bert (zhang and wallace, 2015; mikolov et al., 2017; kenton and toutanova) are alternative approaches recently used for the dense representation of words of the text of analysis. deep neural network (dnn) models: cnn, bilstm, and bigru with word embeddings have achieved tremendous results in textual sentiment analysis (kim; rezaeinia et al., 2017; lei et al.; huang et al.; khasanah, 2021). however, according to recent studies, dnn-based methods select some irrelevant and redundant features and also ignore the sentiment clue behind each sentiment word which affects its performance in terms of classification accuracy (rezaeinia et al., 2017; ayinde et al.; denil et al., 2013). although traditional featurebased methods have benefits in interpretability and time complexity, dnn-based methods outperform classic feature-based methods. ## methodology our proposed effective method for sentiment classification is composed of three main phases: (1) text pre-processing (2) knowledge and embedding (3) cnn architecture. the overall framework of our proposed method is shown in figure 1 . ## experiments experimental setup we tested our system using three real-world benchmark datasets: (1) movie reviews (mr) @xcite , ( 2 ) stanford sentiment treebank (sst-2) datasets (socher et al.), (3) customer review datasets (cr) (hu and liu). mr composed of 5331 positive and 5331 negative review samples. sst-2 contains positive and negative sentences, there are 9,613 single sentences in the dataset, which were obtained from movie reviews. cr consists of 14 products extracted from amazon (hu and liu). sst-2 have standard training-test splits. mr and cr do not have such a standard split, we apply 10-fold cross validation, which is consistent with previous research (huang et al.) on the dataset. we hold out 10 % of the training data for mr and cr for development purposes (e.g. for early stopping), we adopt classification accuracy as an evaluation measure. we generate 300dimensional word vectors for glove and fasttext embedding. the bert-base model case version (network layers @xmath0, hidden layer dimension @xmath1, @xmath2, total number of parameters surpass 110 m, learning rate for @xmath3. we employed wide coverage sentiment lexicon (wcsl) for sentiment information extraction from review texts. we used mrmr and pca filter-based feature selection algorithm for top k optimum feature selection. top 2000 features of mr, 1500 features of sst-2, and 1000 features of cr dataset are feed to each channel in cnn. the dropout rate for each network's layer is 0.5, and 78.1 78.1 79.9 80.1 80.1 81.7 82.8 85.0 85.1 74 76 78 80 82 84 86 pca+cnn mrmr+cnn pca +glove +cnn pca +fastext +cnn mrmr +glove +cnn mrmr +fasttext+cnn bert+cnn pca+bert+cnn mrmr+bert+cnn mr 80.3 80.4 82.2 82.5 82.9 83.1 82.4 84.9 85.2 77 78 79 80 81 82 83 84 85 86 pca+cnn mrmr+cnn pca+glove+cnn pca+fasttext+cnn mrmr+glove+cnn mrmr+fasttext+cnn bert+cnn pca+bert+cnn mrmr+bert+cnn cr 80.1 80.8 80.9 81.0 82.1 83.8 82.5 84.9 85.1 77 78 79 80 81 82 83 84 85 86 pca +cnn mrmr +cnn pca+glove +cnn pca+fasttext +cnn mrmr+glove +cnn mrmr+fasttext+cnn bert+cnn pca+bert+cnn mrmr+bert+cnn sst-2 83.1 accuracy (%) accuracy (%) accuracy (%) ## conclusion we propose an effective way of integrating the traditional feature construction method with the deep learning method to improve the overall performance of sentiment classification. to this end, we leverage semantic and sentiment knowledge using integrated wcsl to extract and select the relevant sentiment features for word embedding and sentiment classification. by employing mrmr and pca filter-based algorithms and pre-trained embedding models (glove, fasttext, and bert) to select optimum sentiment features and consider the semantics and context of words, we can filter out irrelevant and redundant features and reduce the dimensionality of feature space. in-depth experiments with three benchmark domain datasets demonstrate the effectiveness of the proposed model.