An Uncertainty-Aware Encoder for Aspect Detection (2021)

Abstract: Aspect detection is a fundamental task in opinion mining. Previous works use seed words either as priors of topic models, as anchors to guide the learning of aspects, or as features of aspect classifiers. This paper presents a novel weakly-supervised method to exploit seed words for aspect detection based on an encoder architecture. The encoder maps segments and aspects into a low-dimensional embedding space. The goal is approximating similarity between segments and aspects in the embedding space and their ground-truth similarity generated from seed words. An objective function is proposed to capture the uncertainty of ground-truth similarity. Our method outperforms previous works on several benchmarks in various domains.

## introduction aspect detection is essential for downstream tasks in opinion mining such as aspect-based sentiment analysis and opinion summarization @xcite . given an input review segment, for instance, in the restaurant domain, "nevertheless the food itself is pretty good.", we need to detect its aspect category on which opinions have been expressed (e.g., location, drinks, food, ambience, and service). the supervised approach requires a large amount of examples @xcite . its unsupervised counterpart learns aspects using techniques such as topic models and autoencoders. the learned aspects are then manually mapped to golden aspects for prediction. weakly-supervised methods aim at using minimal supervision in terms of seed words to learn aspect predictors. the topic modeling approach assumes that review contents are generated from aspect probability distributions. topic models try to learn these distributions using estimators such as maximum likelihood estimation. seed words are injected into topic models as prior knowledge to guide the estimation of aspect distributions @xcite . the independence assumption in topic models, i.e., the words in a review segment are generated independently from each other, leads to generating incoherent aspects. this phenomenon is more severe as many review segments only have a few words. @xcite propose using a restricted boltzmann machine for joint aspect detection and sentiment classification. however, their model requires various linguistic tools and external resources including part-of-speech tagging, tf-idf weighting, sentiwordnet, and aspect and sentiment seed words. the aspect seed words are acquired by learning a latent dirichlet allocation on raw segments and manually mapping the learned topics to golden aspects, while the sentiment seed words are acquired based on sentiwordnet. to overcome this shortage, the neural approach leverages rich representation from contextual language models to capture semantic similarity between words frequently co-occurring in the same contexts @xcite . model parameters are learned in neural frameworks such as autoencoder, joint learning or knowledge distillation. @xcite construct word embeddings and explicit aspect embeddings by jointly learning a skip-gram style language model and maximizing the likelihood of aspects and sentiments given seed words. their model is further reinforced by knowledge distillation based on pseudo-labels from previously learned aspect embeddings, and later by self-training, both with a convolutional neural network (cnn) classifier. in a recent study, @xcite use a contrastive loss to learn aspect embeddings and manually map them to golden aspects. their model is further enhanced by knowledge distillation with a contextual language model encoder @xcite . @xcite co-train student-teacher classifiers in a knowledge distillation framework. the teacher is designed as a bag-of-seed-words classifier with the weights updated during iterative co-training. another direction is directly using pre-trained embeddings of aspect labels as aspect vectors and scoring against segment vectors with cosine similarity for prediction @xcite . in this paper, we propose a novel weaklysupervised method to exploit seed words for aspect detection. our motivation is from the success of using uncertainty in matrix factorization-based collaborative filtering for movie recommendation from implicit feedback @xcite . the authors show that by adding a confidence score for each user-item pair, their model could learn better from both positive and negative pairs. in our work, we define a confidence score so that ambivalent segments will have a low score. the proposed model is now aware of this ambivalence in learning via a specific designed objective function. our contributions are as follows: ## method the aspect detection problem is defined as assigning a review segment to one of the k pre-defined aspect categories. in the unsupervised settings, a corpus of review segments is given. in the weaklysupervised setting as in this work, the segment corpus and sets of seed words for aspect categories are given. as in the literature, we assume that the seed words have already been acquired manually or been extracted automatically from a small number of labeled examples. our model is depicted in figure 2 : firstly, the encoder maps an input segment and the aspects into the same embedding space. for general, the encoder takes all its seed word embeddings as an embedding matrix. otherwise, the segment and the typical aspects are encoded as mean of their (seed) word embeddings. a similarity function in the embedding space is defined as the dot product of a segment vector and an aspect vector. finally, the objective function approximates this similarity and the ground-truth similarity generated from the seed words. for the general aspect, the objective function performs a global max pooling over the similarities between the segment and the seed words of general to select the best seed word for updating. the encoder is an embedding-lookup table and is identical to the word embeddings matrix @xmath0. w is initialized by pre-trained in-domain word embeddings. for this task, we used the skip-gram model @xcite . we are going into the details of our model in the subsequent sections. section 3.4 is dedicated to the generation of ground-truth similarity from seed words. ## experiments in this part, we first describe the datasets used in our experiments in section 4.1, following by the ex-perimental settings (section 4.2). the methods selected for comparison are introduced in section 4.3. the evaluation results are finally discussed in section 4.4. in all the experiments, our model is referred to as uce, which stands for uncertaintyaware encoder. ## analysis in this section, we conduct an in-depth analysis of our model on the number of seed words and embedding space learning. firstly, the effect of the number of seed words l on typical aspects was investigated. the results are demonstrated on the dev set of the oposum laptop bags domain. l was chosen in the range of [1, 50] with an interval of 5. the number of seed words for general was fixed to 30. as shown in figure 4 , the performance reaches a peak at 5 seed words. it gradually decreases when more seed words are added.