Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification (2021)

Abstract: Recent work on aspect-level sentiment classification has employed Graph Convolutional Networks (GCN) over dependency trees to learn interactions between aspect terms and opinion words. In some cases, the corresponding opinion words for an aspect term cannot be reached within two hops on dependency trees, which requires more GCN layers to model. However, GCNs often achieve the best performance with two layers, and deeper GCNs do not bring any additional gain. Therefore, we design a novel selective attention based GCN model. On one hand, the proposed model enables the direct interaction between aspect terms and context words via the self-attention operation without the distance limitation on dependency trees. On the other hand, a top-k selection procedure is designed to locate opinion words by selecting k context words with the highest attention scores. We conduct experiments on several commonly used benchmark datasets and the results show that our proposed SA-GCN outperforms strong baseline models.

## introduction aspect-level sentiment classification is a finegrained sentiment analysis task, which aims to identify the sentiment polarity (e.g., positive, negative or neutral) of a specific aspect term (also called target) appearing in a sentence. for example, "despite a slightly limited menu, everything prepared is done to perfection, ultra fresh and a work of food art.", the sentiment polarity of aspect terms "menu" and "food" are negative and positive, respectively. the opinion words "limited" and "done to perfection" provide evidences for sentiment polarity predictions. this task has many applications, such as restaurant recommendation and purchase recommendation on e-commerce websites. to solve this problem, recent studies have shown that the interactions between an aspect term and its context (which include opinion words) are crucial in identifying the sentiment polarity towards the given term. most approaches consider the semantic information from the context words and utilize the attention mechanism to learn such interactions. however, it has been shown that syntactic information obtained from dependency parsing is very effective in capturing long-range syntactic relations that are obscure from the surface form @xcite . a recent popular approach to learn syntaxaware representations is employing graph convolutional networks (gcn) @xcite model over dependency trees @xcite @xcite @xcite , which introduces syntactic inductive biases into the message passing. in some cases, the most important context words, i.e. opinion words, are more than two-hops away from the aspect term words on the dependency tree. as indicated by figure 1 , there are four hops between the target "mac os" and the opinion words "easily picked up" on the dependency tree. this type of cases requires more than two layers of gcn to learn interactions between them. however, previous works show that gcn models with two layers often achieve the best performance @xcite , deeper gcns do not bring additional gain due to the over-smoothing problem @xcite , which makes different nodes have similar representations and lose the distinction among nodes. in order to solve the above problem, we propose a novel selective attention based gcn (sa-gcn) model, which combines the gcn model over dependency trees with a self-attention based sequence model over the sentence. on one hand, the selfattention sequence model enables the direct interaction between an aspect term and its context so that it can take care of the situation where the term is far away from the opinion words on the dependency tree. on the other hand, a top-k attention selection module is applied after the self-attention opera- specifically, the base model is the gcn model over dependency trees. the model uses the pretrained bert to obtain representations of the aspect term and its context words as the initial node features on the dependency tree. next, the gcn outputs are fed into a multi-head top-k attention selection module. for each head, the self-attention operation is applied over the sentence to get a dense attention score matrix, where ith row corresponds the attention scores of all words to the i-th word in the sentence. then for each word, context words with top-k attention scores are selected and others are ignored, which sparsifies the attention score matrix and forms a sparse graph. we design two strategies to get the sparse graph: i) applying top-k selection over the attention matrix obtained by summing attention score matrices of all heads, and thus different heads share the same sparse graph; ii) applying top-k selection on individual attention score matrix of each head, and thus different heads have its own sparse graph. finally, we apply a gcn layer again to integrate information from such sparse graph(s) for each head, and concatenate the gcn outputs w.r.t. different heads as the final word representation for sentiment analysis. the main contributions of this work are summarized as the following: capturing the interaction between the aspect term and opinion words is essential for predicting the sentiment polarity towards the aspect term. in recent work, various attention mechanisms, such as co-attention, self-attention and hierarchical attention, were utilized to learn this interaction @xcite @xcite @xcite @xcite @xcite . specifically, they first encoded the context and the aspect term by recurrent neural networks (rnns), and then stacked several attention layers to learn the aspect term representations from important context words. after the success of the pre-trained bert model @xcite , @xcite utilized the pre-trained bert as the encoder. in the study by @xcite , the task was considered as a review reading comprehension (rrc) problem. rrc datasets were post trained on bert and then fine-tuned to the aspect-level sentiment classification. @xcite utilized millions of extra data based on bert to help sentiment analysis. the above approaches mainly considered the semantic information. recent approaches attempted to incorporate the syntactic knowledge to learn the syntax-aware representation of the aspect term. @xcite proposed adarnn, which adaptively propagated the sentiments of words to target along the dependency tree in a bottom-up manner. nguyen and shirai (2015) extended rnn to obtain the representation of the target aspect by aggregating the syntactic information from the dependency and constituent tree of the sentence. @xcite proposed to use the distance between the context word and the aspect term along the dependency tree as the attention weight. some re-searchers @xcite @xcite employed gnns over dependency trees to aggregate information from syntactic neighbors. most recent work in @xcite proposed to reconstruct the dependency tree to an aspect-oriented tree. the reshaped tree only kept the dependency structure around the aspect term and got rid of all other dependency connections, which made the learned node representations not fully syntax-aware. @xcite designed a mutual biaffine module between transformer encoder and the gcn encoder to enhance the representation learning. the downside of applying gcn over dependency trees is that it cannot elegantly handle the long distance between aspect terms and opinion words. our proposed sa-gcn model effectively integrates the benefit of a gcn model over dependency trees and a self-attention sequence model to directly aggregate information from opinion words. the top-k self-attention sequence model selects the most important context words, which effectively sparsifies the fully-connected graph from self-attention. then we apply another gcn layer on top of this new sparsified graph, such that each of those important context words is directly reachable by the aspect term and the interaction between them could be learned. ## conclusions we propose a selective attention based gcn model for the aspect-level sentiment classification task. we first encode the aspect term and context words by pre-trained bert to capture the interaction between them, then build a gcn on the dependency tree to incorporate syntax information. in order to handle the long distance between aspect terms and opinion words, we use the selective attention based gcn block, to select the top-k important context words and employ the gcn to integrate their information for the aspect term representation learning. further, we adopt opinion extraction problem as an auxiliary task to jointly train with sentiment classification task. we conduct experiments on several semeval datasets. the results show that sa-gcn achieve better performances than previous strong baselines.