Embarrassingly Simple Unsupervised Aspect Extraction (2020)

Abstract: We present a simple but effective method for aspect identification in sentiment analysis. Our unsupervised method only requires word embeddings and a POS tagger, and is therefore straightforward to apply to new domains and languages. We introduce Contrastive Attention (CAt), a novel single-head attention mechanism based on an RBF kernel, which gives a considerable boost in performance and makes the model interpretable. Previous work relied on syntactic features and complex neural models. We show that given the simplicity of current benchmark datasets for aspect extraction, such complex models are not needed. The code to reproduce the experiments reported in this paper is available at https://github.com/clips/cat .

## introduction we consider the task of unsupervised aspect extraction from text. in sentiment analysis, an aspect can intuitively be defined as a dimension on which an entity is evaluated (see figure 1 ). while aspects can be concrete (e.g., a laptop battery), they can also be subjective (e.g., the loudness of a motorcycle). aspect extraction is an important subtask of aspect-based sentiment analysis. however, most existing systems are supervised (for an overview, cf. @xcite . as aspects are domain-specific, supervised systems that rely on strictly lexical cues to differentiate between aspects are unlikely to transfer well between different domains @xcite . another reason to consider the unsupervised extraction of aspect terms is the scarcity of training data for many domains (e.g., books), and, more importantly, the complete lack of training data for many languages. unsupervised aspect extraction has previously been attempted with topic models @xcite , topic model hybrids @xcite , and re-the two things that really drew me to vinyl were the expense and the inconvenience . a: aspect vectors s: sentence (word vectors) att: attention vector d: sentence summary food staff ambience rbf figure 2: an overview of our aspect extraction model. figure 3 : examples of contrastive attention (γ=.03) tention (cat ), based on radial basis function (rbf) kernels. compared to conventional attention mechanisms @xcite , cat captures more relevant information from a sentence. our method outperforms more complex methods, e.g., attention-based neural networks @xcite . in addition, our method automatically assigns aspect labels, while in previous work, labels are manually assigned to aspect clusters. finally, we present an analysis of the limitations of our model, and propose some directions for future research. ## method like previous methods @xcite , our method (see figure 2 ) consists of two steps: extraction of candidate aspect terms and assigning aspect labels to instances. both steps assume a set of in-domain word embeddings, which we train using word2vec @xcite . we use a small set of in-domain documents, containing about 4 million tokens for the restaurant domain. ## datasets we use several english datasets of restaurant reviews for the aspect extraction task. all datasets have been annotated with one or more sentencelevel labels, indicating the aspect expressed in that sentence (e.g., the sentence "the sushi was great" would be assigned the label food). we evaluate our approach on the citysearch dataset @xcite , which uses the same labels as the semeval datasets. to avoid optimizing for a single corpus, we use the restaurant subsets of the semeval 2014 @xcite and semeval 2015 @xcite datasets as development data. note that, even though our method is completely unsupervised, we explicitly allocate test data to ensure proper methodological soundness, and do not optimize any models on the test set. following previous work @xcite , we restrict ourselves to sentences that only express exactly one aspect; sentences that express more than one aspect, or no aspect at all, are discarded. additionally, we restrict ourselves to three labels: food, service, and ambience. we adopt these restrictions in order to compare to other systems. additionally, previous work @xcite reported that the other labels, anecdotes and price, were not reliably annotated. ## evaluation we optimize all our models on semeval '14 and '15 training data; the scores on the citysearch dataset do not reflect any form of optimization with regards to performance. we optimize the hyperparameters of each model separately (i.e., the number of aspect terms and γ of the rbf kernel), leading to the following hyperparameters: for the regular attention, we select the top 980 nouns as aspect candidates. for the rbf attention, we use the top 200 nouns and a γ of .03. we compare our system to four other systems. w2vlda (garcía-pablos et al., 2018) is a topic modeling approach that biases word-aspect associations by computing the similarity from a word to a set of aspect terms. serbm @xcite a restricted boltzmann machine (rbm) that learns topic distributions, and assigns individual words to these distributions. in doing so, it learns to assign words to aspects. we also compare our system to two attention-based systems. first, abae @xcite , which is an auto-encoder that learns an attention distribution over words in the sentence by simultaneously considering the global context and aspect vectors. in doing so, abae learns an attention distribution, as well as appropriate aspect vectors. second, ae-csa @xcite , which is a hierarchical model which is similar to abae. in addition to word vectors and aspect vectors, this model also considers sense and sememe @xcite vectors in computing the attention distribution. note that all these systems, although being unsupervised, do require training data, and need to be fit to a specific domain. hence, all these systems rely on the existence of in-domain training data on which to learn reconstructions and/or topic distributions. furthermore, much like our approach, abae, ae-csa, and w2vlda rely on the availability of pre-trained word embeddings. additionally, ae-csa needs a dictionary of senses and sememes, which might not be available for all languages or domains. compared to other systems, our system does require a ud pos tagger to extract frequent nouns. however, this can be an off-the-shelf pos tagger, since it does not need to be trained on domain-specific data. we also compare our system to a baseline based on the mean of word embeddings, a version of our system using regular attention, and a version of our system using contrastive attention (cat ). the results are shown in ## analysis we perform an ablation study to see the influence of each component of our system; specifically, we look at the effect of pos tagging, in-domain word embeddings, and the amount of data on performance. only selecting the most frequent words as aspects, regardless of their pos tag, had a detrimental effect on performance, giving an f-score of 64.5 (∆-21.9), while selecting nouns based on adjectivenoun co-occurrence had a smaller detrimental effect, giving an f-score of 84.4 (∆-2.2), higher than abae and serbm. replacing the in-domain word embeddings trained on the training set with pretrained glove embeddings @xcite foot_0 had a large detrimental effect on performance, dropping the f-score to 54.4 (∆-32); this shows that in-domain data is important. to investigate how much in-domain data is required to achieve good performance, we perform a learning curve experiment (figure 4 ). we increase the training data in 10% increments, training five word2vec models at each increment. as the fig- ## conclusion we present a simple model of aspect extraction that uses a frequency threshold for candidate selection together with a novel attention mechanism based on rbf kernels, together with an automated aspect assignment method. we show that for the task of assigning aspects to sentences in the restaurant domain, the rbf kernel attention mechanism outperforms a regular attention mechanism, as well as more complex models based on auto-encoders and topic models.