Improving Aspect-based Sentiment Analysis with Gated Graph Convolutional Networks and Syntax-based Regulation (2020)

Abstract: Aspect-based Sentiment Analysis (ABSA) seeks to predict the sentiment polarity of a sentence toward a specific aspect. Recently, it has been shown that dependency trees can be integrated into deep learning models to produce the state-of-the-art performance for ABSA. However, these models tend to compute the hidden/representation vectors without considering the aspect terms and fail to benefit from the overall contextual importance scores of the words that can be obtained from the dependency tree for ABSA. In this work, we propose a novel graph-based deep learning model to overcome these two issues of the prior work on ABSA. In our model, gate vectors are generated from the representation vectors of the aspect terms to customize the hidden vectors of the graph-based models toward the aspect terms. In addition, we propose a mechanism to obtain the importance scores for each word in the sentences based on the dependency trees that are then injected into the model to improve the representation vectors for ABSA. The proposed model achieves the state-of-the-art performance on three benchmark datasets.

## introduction aspect-based sentiment analysis (absa) is a finegrained version of sentiment analysis (sa) that aims to find the sentiment polarity of the input sentences toward a given aspect. we focus on the term-based aspects for absa where the aspects correspond to some terms (i.e., sequences of words) in the input sentence. for instance, an absa system should be able to return the negative sentiment for input sentence "the staff were very polite, but the quality of the food was terrible." assuming "food" as the aspect term. due to its important applications (e.g., for opinion mining), absa has been studied extensively in the literature. in these studies, deep learning has been employed to produce the state-of-the-art performance for this problem @xcite . recently, in order to further improve the performance, the syntactic dependency trees have been integrated into the deep learning models @xcite for absa (called the graph-based deep learning models). among others, dependency trees help to directly link the aspect term to the syntactically related words in the sentence, thus facilitating the graph convolutional neural networks (gcn) @xcite to enrich the representation vectors for the aspect terms. however, there are at least two major issues in these graph-based models that should be addressed to boost the performance. first, the representation vectors for the words in different layers of the current graph-based models for absa are not customized for the aspect terms. this might lead to suboptimal representation vectors where the irrelevant information for absa might be retained and affect the model's performance. ideally, we expect that the representation vectors in the deep learning models for absa should mainly involve the related information for the aspect terms, the most important words in the sentences. consequently, in this work, we propose to regulate the hidden vectors of the graph-based models for absa using the information from the aspect terms, thereby filtering the irrelevant information for the terms and customizing the representation vectors for absa. in particular, we compute a gate vector for each layer of the graph-based model for absa leveraging the representation vectors of the aspect terms. this layer-wise gate vector would be then applied over the hidden vectors of the current layer to produce customized hidden vectors for absa. in addition, we propose a novel mechanism to explicitly increase the contextual distinction among the gates to further improve the representation vectors. the second limitation of the current graph-based deep learning models is the failure to explicitly exploit the overall importance of the words in the sentences that can be estimated from the dependency trees for the absa problem. in particular, a motivation of the graph-based models for absa is that the neighbor words of the aspect terms in the dependency trees would be more important for the sentiment of the terms than the other words in the sentence. the current graph-based models would then just focus on those syntactic neighbor words to induce the representations for the aspect terms. however, based on this idea of important words, we can also assign a score for each word in the sentences that explicitly quantify its importance/contribution for the sentiment prediction of the aspect terms. in this work, we hypothesize that these overall importance scores from the dependency trees might also provide useful knowledge to improve the representation vectors of the graph-based models for absa. consequently, we propose to inject the knowledge from these syntaxbased importance scores into the graph-based models for absa via the consistency with the modelbased importance scores. in particular, using the representation vectors from the graph-based models, we compute a second score for each word in the sentences to reflect the model's perspective on the importance of the word for the sentiment of the aspect terms. the syntax-based importance scores are then employed to supervise the model-based importance scores, serving as a method to introduce the syntactic information into the model. in order to compute the model-based importance scores, we exploit the intuition that a word would be more important for absa if it is more similar the overall representation vector to predict the sentiment for the sentence in the final step of the model. in the experiments, we demonstrate the effectiveness of the proposed model with the state-of-the-art performance on three benchmark datasets for absa. in summary, our contributions include: ## related work sentiment analysis has been studied under different settings in the literature (e.g., sentence-level, aspect-level, cross-domain) @xcite @xcite @xcite . for absa, the early works have performed feature engineering to produce useful features for the statistical classification models (e.g., svm) @xcite . recently, deep learning models have superseded the feature based models due to their ability to automatically learn effective features from data @xcite @xcite . the typical network architectures for absa in the literature involve convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite , memory networks @xcite , attention @xcite and gating mechanisms @xcite . the current state-of-the-art deep learning models for absa feature the graph-based models where the dependency trees are leveraged to improve the performance. @xcite @xcite . however, to the best of our knowledge, none of these works has used the information from the aspect term to filter the graph-based hidden vectors and exploited importance scores for words from dependency trees as we do in this work. ## model the task of absa can be formalized as follows: given a sentence @xmath0, x 2 , . . . , x n ] of n words/tokens and the index t (1 ≤ @xmath1, the goal is to predict the sentiment polarity y * toward the aspect term x t for x. our model for absa in this work consists of three major components: (i) representation learning, (ii) graph convolution and regulation, and (iii) syntax and model consistency. one problem with the gcn hidden vectors h l i gcn is that they are computed without being aware of the aspect term x t . this might retain irrelevant or confusing information in the representation vectors (e.g., a sentence might have two aspect terms with different sentiment polarity). in order to explicitly regulate the hidden vectors in gcn to focus on the provided aspect term x i , our proposal is to compute a gate g l for each layer l of gcn using the representation vector e t of the aspect term: g @xmath2. this gate is then applied over the hidden vectors h l i of the l-th layer via the element-wise multiplication •, generating the regulated hidden vector hl i for h l i : ideally, we expect that the hidden vectors of gcn at different layers would capture different levels of contextual information in the sentence. the gate vectors g t for these layer should thus also exhibit some difference level for contextual information to match those in the gcn hidden vectors. in order to explicitly enforce the gate diversity in the model, our intuition is to ensure that the regu-lated gcn hidden vectors, once obtained by applying different gates to the same gcn hidden vectors, should be distinctive. this allows us to exploit the contextual information in the hidden vectors of gcn to ground the information in the gate vectors for the explicit gate diversity promotion. in particular, given the l-th layer of gcn, we first obtain an overall representation vector hl for the regulated hidden vectors at the l-th layer using the max-pooled vector: @xmath3, . . . , hl n ). afterwards, we apply the gate vectors g l from the other layers (@xmath4, resulting in the regulated hidden vectors hl,l for each of these other layers l , we also compute an overall representation vector hl,l with max-pooling: hl,@xmath5,l 1 , . . . , hl,l n ). finally, we promote the diversity between the gate vectors g l by enforcing the distinction between hl and hl,l for @xmath6. this can be done by minimizing the cosine similarity between these vectors, leading to the following regularization term l div to be added to the loss function of the model: in order to incorporate syntax-based scores syn i into the model, we first leverage the hidden vectors in gcn to compute a model-based importance score mod i for each word x @xmath7. afterwards, we seek to minimize the kl divergence between the syntaxbased scores syn 1 , . . . , syn n and the model-based scores mod 1 , . . . , mod n by introducing the following term l const into the overall loss function: the rationale is to promote the consistency between the syntax-based and model-based importance scores to facilitate the in-jection of the knowledge in the syntax-based scores into the representation vectors of the model. for the model-based importance scores, we first obtain an overall representation vector v for the input sentence x to predict the sentiment for x t . in this work, we compute v using the sentence representation vector s from bert and the regulated hidden vectors in the last layer of gcn: based on this overall representation vector v , we consider a word x i to be more contextually important for absa than the others if its regulated gcn hidden vector ĥl i in the last gcn layer is more similar to v than those for the other words. the intuition is the gcn hidden vector of a contextually important word for absa should be able capture the necessary information to predict the sentiment for x t , thereby being similar to v that is supposed to encode the overall relevant context information of x to perform sentiment classification. in order to implement this idea, we use the dot product of the transformed vectors for v and ĥl i to determine the model-based importance score for x i in the model: finally, we feed v into a feed-forward neural network with softmax in the end to estimate the probability distribution p (.|x, x t ) over the sentiments for x and x t . the negative log-likelihood l @xmath8, x t ) is then used as the prediction loss in this work. the overall loss to train the proposed model is then: @xmath9. as only the mams dataset provides the development data, we fine-tune the model's hyperparameters on the development data of mams and use the same hyper-parameters for the other datasets. the following hyper-parameters are suggested for the proposed model by the fine-tuning process: 200 dimensions for the hidden vectors of the feed forward networks and gcn layers, 2 hidden layers in gcn, the size 32 for the mini-batches, the learning rate of 0.001 for the adam optimizer, and 1.0 for the trade-off parameters α and β. finally, we use the cased bert base model with 768 hidden dimensions in this work. results: to demonstrate the effectiveness of the proposed method, we compare it with the following baselines: (1) the feature-based model that applies feature engineering and the svm model @xcite , (2) the deep learning models based on the sequential order of the words in the sentences, including cnn, lstm, attention and the gating mechanism @xcite @xcite @xcite , and (3) the graph-based models that exploit dependency trees to improve the deep learning models for absa @xcite @xcite @xcite . ablation study: there are three major components in the proposed model: (1) the gate vectors g l to regulate the hidden vectors of gcn (called gate), (2) the gate diversity component l div to promote the distinction between the gates (called div.), and (3) the syntax and model consistency component l const to introduce the knowledge from the syntax-based importance scores (called con.). ## conclusion we introduce a new model for absa that addresses two limitations of the prior work. it employs the given aspect terms to customize the hidden vectors. it also benefits from the overall dependency-based importance scores of the words. our extensive experiments on three benchmark datasets empirically demonstrate the effectiveness of the proposed approach, leading to state-of-the-art results on these datasets. the future work involves applying the proposed model to the related tasks for absa, e.g., event detection @xcite .