Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification (2020)

Abstract: Aspect-based sentiment classification is a popular task aimed at identifying the corresponding emotion of a specific aspect. One sentence may contain various sentiments for different aspects. Many sophisticated methods such as attention mechanism and Convolutional Neural Networks (CNN) have been widely employed for handling this challenge. Recently, semantic dependency tree implemented by Graph Convolutional Networks (GCN) is introduced to describe the inner connection between aspects and the associated emotion words. But the improvement is limited due to the noise and instability of dependency trees. To this end, we propose a dependency graph enhanced dual-transformer network (named DGEDT) by jointly considering the flat representations learnt from Transformer and graph-based representations learnt from the corresponding dependency graph in an iterative interaction manner. Specifically, a dual-transformer structure is devised in DGEDT to support mutual reinforcement between the flat representation learning and graph-based representation learning. The idea is to allow the dependency graph to guide the representation learning of the transformer encoder and vice versa. The results on five datasets demonstrate that the proposed DGEDT outperforms all state-of-the-art alternatives with a large margin.

## introduction aspect-based or aspect-level sentiment classification is a popular task with the purpose of identifying the sentiment polarity of the given aspect @xcite @xcite . the goal is to predict the sentiment polarity of a given pair (sentence, aspect). aspects in our study are mostly noun phrases appearing in the input sentence. as shown in figure 1 , where the comment is about the laptop review, the sentiment polarities of two aspects battery life and memory are positive and negative, respectively. giving a specific aspect is crucial for sentiment classification owing to the situation that one sentence sometimes contains several aspects, and these aspects may have different sentiment polarities. modern neural methods such as recurrent neural networks (rnn), convolutional neural networks (cnn) @xcite have already been widely applied to aspectbased sentiment classification. inspired by the work @xcite which demonstrates the importance of modeling the semantic connection between contextual words and aspects, rnn augmented by attention mechanism @xcite @xcite is widely utilized in recent methods for exploring the potentially relevant words with respect to the given aspect @xcite @xcite . cnn based attention methods @xcite are also proposed to enhance the phrase-level representation and achieved encouraging results. although attention-based models have achieved promising performance on several tasks, the limitation is still obvious because attention module may highlight the irrelevant words owing to the syntactical absence. for example, given the sentence "it has a bad memory but a great battery life." and aspect "battery life", attention module may still assign a large weight to word "bad" rather than "great", which adversely leads to a wrong sentiment polarity prediction. to take advantages of syntactical information among aspects and contextual words, @xcite proposed a novel aspect-based gcn method which incorporates dependency tree into the attention models. actually, using gcn @xcite aspect: memory sentiment: negative aspect: battery life sentiment: positive welling, 2017) to encode the information conveyed by a dependency tree has already been investigated in several fields, e.g., modeling document-word relationships @xcite and tree structures @xcite . as shown in figure 1 , an annotated dependency tree of original sentence is provided, and we can observe that word-aspect pairs (bad, memory) and (great, battery life) are well established. direct application of dependency tree has two obvious shortcomings: (1) the noisy information is inevitably introduced through the dependency tree, due to imperfect parsing performance and the casualness of input sentence; (2) gcn would be inherently inferior in modeling long-distance or disconnected words in the dependency tree. it is reported that lower performance is achieved even with the golden dependency tree, by comparing against using only the flat structure @xcite . to address these two challenges, we propose a dependency graph enhanced dual-transformer network (named dgedt) for aspect-based sentiment classification. dgedt consists of a traditional transformer @xcite and a transformer-like structure implemented via a dependency graph based bidirectional gcn (bigcn). specifically, a dual-transformer structure is introduced in dgedt to fuse the flat representations learnt by the transformer and the graph-based representations learnt based on the dependency graph. these two kinds of representations are jointly refined through a mutual biaffine transformation process, where the dependency graph can guide and promote the flat representation learning. the final flat representations derived by the transformer is then used with an aspect-based attention for sentiment classification. we have conducted extensive experiments over five benchmark datasets. the experimental results demonstrate that the proposed dgedt achieves a large performance gain over the existing state-of-the-art alternatives. to the best of our knowledge, the proposed dgedt is the first work that jointly considers the flat textual knowledge and dependency graph empowered knowledge in a unified framework. furthermore, unlike other aspect-based gcn models, we aggregate the aspect embeddings from multiple aspect spans which share the same mentioned aspect before feeding these embeddings into submodules. we also introduce an aspect-modified dependency graph in dgedt. ## related work employing modern neural networks for aspectbased sequence-level sentiment classification task, such as cnns @xcite , rnns @xcite , recurrent convolutional neural networks (rcnns) @xcite , have already achieved excellent performance in several sentiment analysis tasks. many attention-based rnn or cnn methods @xcite @xcite are also proposed to handle sequence classification tasks. @xcite proposed a tree-lstm structure which is enhanced with dependency trees or constituency trees, which outperforms traditional lstm. @xcite proposed an adaptive recursive neural network using dependency trees. since being firstly introduced in @xcite , gcn has recently shown a great ability on addressing the graph structure representation in natural language processing (nlp) field. @xcite proposed a gcn-based model for semantic role labeling. @xcite and @xcite used gcn over dependency trees in document dating and relation classification, respectively. yao et al. (2019) introduced gcn to text classification task with the guidance of document-word and word-word relations. furthermore, @xcite introduced aspect-based gcn to cope with aspect-level sentiment classification task using dependency graphs. on the other hand, @xcite introduced and adapted capsule networks along with transfer learning to improve the performance of aspect-level sentiment classification. gao et al. (2019) ## preliminaries since transformer @xcite and gcn are two crucial sub-modules in dgedt, here we briefly introduce these two networks and illustrate the fact that gcn can be considered as a specialized transformer. assume that there are three input matrices @xmath0, @xmath1, @xmath2, which represent the queries, keys and values respectively. n and m are the length of two inputs. as for gcn, the computation can be conducted as follows when the adjacent matrix of each word in the input is explicitly provided. ## dgedt the network architecture of our proposed dgedt is shown in figure 2 . for a given input text, we first utilize bilstm or bidirectional encoder representations from transformers (bert) @xcite as the aspect-based encoder to extract hidden contextual representations. then these hidden representations are fed into our proposed dual-transformer structure, with the guidance of aspect-modified dependency graph. at last, we aggregate all the aspect representations via maxpooling and apply an attention module to align contextual words and the target aspect. in this way, the model can automatically select relevant aspectsensitive contextual words with the dependency information for sentiment classification. ## conclusion recently neural structures with syntactical information such as semantic dependency tree and constituent tree are widely employed to enhance the word-level representation of traditional neural networks. these structures are often modeled and described by treelstms or gcns. to introduce transformer into our task and diminish the error induced by incorrect dependency trees, we propose a dual-transformer structure which considers the connections in dependency tree as a supplementary gcn module and a transformer-like structure for self alignment in traditional transformer. the results on five datasets demonstrate that dependency tree indeed promotes the final performance when utilized as a sub-module for dual-transformer structure. in future work, we can further improve our method in the following aspects. first, the edge information of the dependency trees needs to be exploited in later work. we plan to employ an edgeaware graph neural network considering the edge labels. second and last, domain-specific knowledge can be incorporated into our method as an external learning source.