Aspect Based Sentiment Analysis Using Spectral Temporal Graph Neural Network (2021)

Abstract: The objective of Aspect Based Sentiment Analysis is to capture the sentiment of reviewers associated with different aspects. However, complexity of the review sentences, presence of double negation and specific usage of words found in different domains make it difficult to predict the sentiment accurately and overall a challenging natural language understanding task. While recurrent neural network, attention mechanism and more recently, graph attention based models are prevalent, in this paper we propose graph Fourier transform based network with features created in the spectral domain. While this approach has found considerable success in the forecasting domain, it has not been explored earlier for any natural language processing task. The method relies on creating and learning an underlying graph from the raw data and thereby using the adjacency matrix to shift to the graph Fourier domain. Subsequently, Fourier transform is used to switch to the frequency (spectral) domain where new features are created. These series of transformation proved to be extremely efficient in learning the right representation as we have found that our model achieves the best result on both the SemEval-2014 datasets, i.e., “Laptop” and “Restaurants” domain. Our proposed model also found competitive results on the two other recently proposed datasets from the e-commerce domain.

## introduction with the proliferation of online shopping consumers are relying more and more on ratings, and reviews available from past customers. aspectbased sentiment analysis (absa) tries to understand customers' granular opinion on different dimensions (aspects) of a product which helps in understanding its current limitations and planning for the next round of improvements. these reviews invariably talk about multiple aspects (sometimes in the same sentence) with the presence of mixed feedbacks, positive and negative. the presence of multiple aspects and opposite sentiments makes the task of absa challenging and since its birth in 2014 @xcite the task has seen a variety of different approaches and still enjoys considerable attention from the research community. the key to an improved performance in absa lies in the ability to identify the aspects and the corresponding sentiments with the help of connections between them. a typical example would be "the price is reasonable although the service is poor" where price and service are the aspects with positive and negative sentiments, respectively. it is clear from the sentence that the corresponding modifiers, reasonable and poor, are driving the respective sentiments. however, in another example, "i had the salmon dish and while it was fine, for the price paid, i expected it to have some type of flavor", it is not clear which words or phrases can be identified as responsible for the negative sentiment. similar examples are "prices are in line" (neutral sentiment) and "for the price, you can not eat this well in manhattan" (positive sentiment). thus, while one may hope that capturing the syntactical structure and dependency between words will help in improving the sentiment identification it is also important to understand the meaning and application of words in general setting and not to be gleaned from the limited examples present in the modest absa datasets (2328 and 3608 training examples for the popular "laptop" and "restaurant" domains, respectively). recent successes with bert and roberta based encoders indicate that a powerful word representation in the context of the surrounding words would help in establishing the required connections between the aspects and sentiments. while large model-based encoders provide a powerful initial representation, subsequent transfor-mations possibly involving attention and additional embeddings from parts-of-speech or dependency parser are equally important for the quality of the final prediction. recent trends also indicate a confluence of deep encoders and graph-based representation of words (nodes) where semantic relations are captured in the graph representations either by converting a dependency graph or by utilizing the nearest neighborhood of words. instead of working on a graph structure stemming from the language itself, a different approach of constructing an underlying graph can be by learning it from the encoded representations. this is motivated by recent advances in the forecasting literature where in absence of any obvious underlying graph, a graph is created from encoded representation by applying self-attention. armed with this new graph subsequent transformations including graph fourier transform (gft) and discrete fourier transform (dft) proved to be extremely efficient in understanding the correlation between different dimensions and increasing the overall performance of the model. motivated by the success of this approach in the forecasting domain, we apply the same for absa although we believe there can be many other applications. there are three key components in this technique, (a) create a graph from the encoded representation using self-attention, (b) use the graph laplacian to transform from the vertex domain to graph-spectral domain and (c) apply dft to transform from graph-spectral to frequency domain where convolution operator extracts features. to the best of our knowledge these components approach has not been considered in solving any natural language applications and would open numerous possibilities for further modifications and improvements. the organization of the paper is as follows. in the next section we provide a detailed literature survey on the techniques employed for absa. next, we present the details of the proposed model. subsequently, the model predictions and comparisons with other baseline methods are discussed. finally, conclusions are drawn and scope for future works is outlined. on the models based on memory network, @xcite and @xcite designed deep memory networks to with weighted memory mechanism to capture relations between aspects and sentiments separated by long distance. @xcite introduced dyadic memory network for absa where relevant memory information is adaptively used based on the input query. @xcite proposed hierarchical attention network to have separate aspect attention and sentiment attention that found better matching between previously unseen aspect and sentiment words. @xcite proposed a new mask memory network with semantic dependency that exploited inter-aspect relations for aspects in the same sentence. with the advent of transformer @xcite and strong baselines reported for different nlp tasks with bert @xcite based architectures there are quite a few bert based models for absa. hoang et al. (2019) used sentencepair classification task to reformulate aspect extraction and aspect polarity classification. zeng et al. (2019) used bert embeddings to create a local and global representation of the contexts that were further processed via multi-head self-attention. xu et al. (2019) created a novel task called review reading comprehension from the absa datasets and applied bert to answer the review questions. li et al. (2019) used bert as embedding layer together with crf for end-to-end absa. bert embeddings are also used by song et al. (2019) for absa with label smoothing regularization. sun et al. (2019a) constructed absa as sentence-pair classification task by constructing auxiliary sentences. phan and ogunbona (2020) combined pos embeddings, dependency embeddings and self-attention with roberta (liu et al., 2019) embeddings to further improve on the aspect classification results. there are not many applications of graph neural networks for absa available in the literature. zhang et al. (2019) and sun et al. (2019b) used graph convolution network (gcn) where the graph structure was learnt from the dependency tree. similarly, @xcite used graph attention network (gat) to establish the dependency between words without paying specific attention to the aspects and their opinions. @xcite modified the original dependency tree to create an aspect-oriented dependency tree that was used further in a relational gat (r-gat) where different relations contributed differently in the computation of nodal representations. while the above mentioned approaches rely on a graph structure that emerges naturally from the syntactic structure of the examples it is worth exploring if there is a possibility of learning the graph structure itself from the presented data. this idea is borrowed from forecasting literature where stateof-the-art models are based on gcn originated from the theory of graph fourier transform (gft). in addition to gcn and temporal modules like lstm or gru, it has also been shown that feature processing in the spectral domain can substantially improve the model performance @xcite . while the application of fourier transform is not common in the natural language processing (nlp) domain, it has been observed recently @xcite that the self-attention layer in the transformer can be replaced by a standard fourier transform and still achieving 92-97% of the original accuracy. ## methodology the overall architecture of the current method closely follows the architecture of spectral temporal graph neural network (stgnn) @xcite with some minor modifications (see fig. 1 ). however, for the sake of completeness the components of stgnn are described here. there are five major transformations that any sentence will be subjected to, (1) encoding by an embedding layer, (2) processing by an rnn (we call it the encoder) and create a graph structure, (3) transformation from vertex domain to graph spectral domain using the eigenvectors of this graph, (4) discrete fourier transform in the graph spectral domain and (5) filtering by convolution layers in the graph spectral frequency domain. subsequently, inverse fourier transform and inverse graph fourier transform are applied sequentially to bring the representation back to the graph vertex domain. these transformations can be broadly combined into three key components, (1) embedding layer, (2) latent correlation layer (lcl) and (3) spectral block layer (sbl). the details of each layer are given below: ## experiments in this section, we first describe the datasets used for the evaluation of our proposed method and the other baseline methods employed for comparison. then, we report the experimental results conducted from different perspectives. finally, error analysis and discussion are conducted with a few representative examples. ## results & analysis for all the baseline models, it is difficult to know the exact hyperparameter settings in order to reproduce the results. instead, we relied on the results that are obtained by @xcite . we have also included the originally reported results for the sake of completion and easy comparison. for all the datasets we have two sets of results, (a ## conclusion we present a novel application of graph fourier transform with spectral feature engineering hith-erto limited to forecasting domain. the model learns an underlying graph structure from the raw data created by a bert encoder. the advantage of this approach is that it does not require dependency parser based graph creation and thereby does not inherit any limitation of the parser. it is shown that the series of transformations involving gft, dft, convolution and glu create powerful representations of the text resulting in the superior performance on semeval-2014 datasets, namely "laptop" and "restaurants" domain. on the "laptop" dataset we achieved the best results while on the "restaurants" dataset our performance is at par with the current best prediction. on the recently released e-commerce datasets, our model performance is very competitive with a gap of 0.2-0.4 percent points. although we have not done a fullscale hyper-parameter tuning, the effect of different components like the initial encoder and the final layer is studied. it is observed that the same set of hyper-parameters and architecture will not generate the best result across all the datasets. there are several possible future directions of work. if we view the current model as a spectral graph transformer that takes sequential input and generates sequential output there could be several other applications like, sequence tagging or natural language generation. also, we have evaluated only bert for sentence encoding and in future, other language models like roberta and gpt can be explored.