Variational Weakly Supervised Sentiment Analysis with Posterior Regularization (2021)

Abstract: Sentiment analysis is an important task in natural language processing (NLP). Most of existing state-of-the-art methods are under the supervised learning paradigm. However, human annotations can be scarce. Thus, we should leverage more weak supervision for sentiment analysis. In this paper, we propose a posterior regularization framework for the variational approach to the weakly supervised sentiment analysis to better control the posterior distribution of the label assignment. The intuition behind the posterior regularization is that if extracted opinion words from two documents are semantically similar, the posterior distributions of two documents should be similar. Our experimental results show that the posterior regularization can improve the original variational approach to the weakly supervised sentiment analysis and the performance is more stable with smaller prediction variance.

## introduction sentiment analysis is a task of identifying the sentiment polarity expressed in textual data @xcite . most state-of-the-art sentiment analysis methods in the literature are supervised methods which require many labeled training data. however, human annotations in the real world are scarce. while we assume there is abundant annotated data to train more and more complex models, there is still a need to consider weakly supervised methods that require less human annotation. one way to perform weakly supervised sentiment analysis is using a predefinited lexicon @xcite . a lexicon consists of many opinion words. for each opinion word, its polarity (positive or negative) and strength (the degree to which the opinion word is positive or negative) are annotated by domain experts. lexiconbased weakly supervised methods perform a dictionary lookup and assign a polarity according to all opinion words extracted from a document. a good lexicon requires high precision and high coverage, which needs a lot of human effort. another way to do weakly supervised sentiment analysis is using limited keywords @xcite . compared with lexiconbased methods, user-provided keywords require less human effort. among keyword-based methods, there are two directions. first, @xcite leveraged limited keywords to expand more keywords and generate pseudo-labeled data, and then performed self-training on real unlabeled data for model refinement. possible improvements of this direction include investigating more advanced keywords expansion techniques to generate better pseudo-labeled samples @xcite and developing more advanced self-training algorithms @xcite . second, the variational weakly supervised (vws) sentiment analysis @xcite used target-opinion word pairs as supervision signal. its objective function is to predict an opinion word given a target word. for example, in a sentence "the room is big," "room" is a target word and "big" is an opinion word. by introducing a latent variable (the sentiment polarity), they can learn a wellapproximated posterior distribution via optimizing the evidence lower bound. the posterior probability here is the probability of a possible polarity (e.g., positive or negative) given a document, which is a typical sentiment classifier. a potential issue with vws is that optimizing the objective function may not guide the role of the latent variable to be sentiment polarity. for example, when half of reviews mention "big room" and half of reviews mention "small room," the latent variable is possibly related to the size of rooms, but the expected role of the latent variable is the sentiment polarity of rooms. hence how to control and regularize the posterior distribution is very im-portant. one indirect way to control the posterior distribution is clever initialization @xcite . originally, vws aims to predict the sentiment polarity of each aspect for the multi-aspect sentiment analysis. so it uses the overall sentiment polarity to pretrain the model so that the posterior distribution has a good initialization. the overall polarity and polarity of each aspect are highly correlated. thus, the initialization is highly likely to be similar to the true posterior distribution. in this paper, we propose to use posterior regularization to regularize the vws approach for sentiment analysis. there are two types of side information we can leverage to regularize the latent variable. first, calculating the similarity between keywords and extracted opinion words from a document can guide the model to decide which polarity that the document belongs to. second, calculating the similarity between extracted opinion words from two documents can guide the model to decide whether two documents belong to the same polarity. the first type of side information is very easy to leverage, and it is reflected by our pretraining process. when a document is similar to keywords associated with a specific polarity, then we enforce that the posterior probability of a specific polarity should be large. in this case, in the pretraining process, we assign a pseudo label to the document. the second type of side information does not directly suggest which sentiment polarity that a document should be assigned to. it enforces pairwise constraints to the model. our proposed posterior regularization leverage the second side information to ensure that when two documents are similar (dissimilar), the regularization enforces the posterior distribution of two documents to be similar (dissimilar). our code is available at https://github.com/ hkust-knowcomp/vws-pr . ## methodology in this section, we first review the variational weakly supervised (vws) sentiment analysis method in section 2.1. then we introduce our posterior regularization in section 2.2. ## experiments in this section, we evaluate the empirical performance of our method on binary sentiment classification tasks. ## related work in this section, we review the related work on weakly supervised sentiment analysis. using a lexicon is a typical way to perform weakly supervised sentiment analysis. one line of works perform simple assignment, i.e., majority voting, based on sentiment orientation scores of extracted opinion words. some methods @xcite used sentiment orientation scores in existing lexicons directly, and aggregated them within a document to determine polarity. some methods developed their own semantic orientation estimation algorithm. for example, @xcite first identified phrases in the review and then estimated the semantic orientation of each extracted phrases. the semantic orientation of a given phrase is calculated by comparing its similarity to a positive reference word ("excellent") with its similarity to a negative reference word ("poor"). this method determined the sentiment polarity based on the average semantic orientation of the phrases extracted from the review. @xcite used the minimum path distance between a phrase and pivot words ("good" and "bad") in wordnet to estimate the semantic orientation of extracted phrases. another line of works involve learning process when using a lexicon. @xcite proposed a constrained non-negative matrix tri-factorization approach to sentiment anal-ysis, and used a sentiment lexicon as prior knowledge. in these models, a term-document matrix is approximated by three factors that specify soft membership of terms and documents in one of k classes. all three factors are non-negative matrices. the first factor is a matrix representing knowledge in the word space, i.e., each row represents the posterior probability of a word belonging to the k classes. the second factor is a matrix providing a condensed view of the term-document matrix. the third factor is a matrix representing knowledge in document space, i.e., each row represents the posterior probability of a document belonging to the k classes. @xcite applied a regularization to encourage that the first factor is close to prior knowledge. this regularization is different from ours because it requires prior knowledge such as a predefined lexicon. a predefined lexicon needs a lot of human effort. @xcite applied a regularization based on an intuition that if two documents are sufficiently close to each other, they tend to share the same sentiment polarity. this intuition of this regularization is similar to ours. but when they compare document similarity, they use textual similarity (e.g., cosine similarity of bag of words) rather than similarity on opinion words. the regularization is applied under matrix factorization framework, it is not straightforward to fit in neural network based models. using keywords is another way to perform weakly supervised sentiment analysis. @xcite leveraged keywords to generate pseudolabeled samples for model pretraining, and then performed self-training on unlabeled data for model refinement. the possible improvement of this direction would be investigating more advanced keywords expansion techniques to generate better pseudo-labeled samples and developing a more advanced self-training algorithm. lotclass @xcite , a parallel work to ours, fine-tuned a masked language model to generate relevant words that can replace label name such as "good" and "bad," and performed self-training on unlabeled data for model refinement. fine-tuning in lot-class can be viewed as an advanced keyword expansion process using language models. vws @xcite used target-opinion word pairs as supervision signal. its objective function is to predict an opinion word given a target word. by introducing a latent variable (the sentiment polarity), they can learn a well-approximated posterior dis-tribution via optimizing the evidence lower bound. the posterior probability here is the probability of a possible polarity (e.g., positive or negative) given text representation. ## conclusion we propose a posterior regularization framework for the vws sentiment analysis to better control the posterior distribution. the intuition behind the posterior regularization is that if extracted opinion words from two documents are semantically similar (dissimilar), the posterior distribution of two documents should be similar (dissimilar). our experiments show that our posterior regularization can improve vws and the performance is more stable.