A structure-enhanced graph convolutional network for sentiment analysis (2020)

Abstract: Syntactic information is essential for both sentiment analysis(SA) and aspect-based sentiment analysis(ABSA). Previous work has already achieved great progress utilizing Graph Convolutional Network(GCN) over dependency tree of a sentence. However, these models do not fully exploit the syntactic information obtained from dependency parsing such as the diversified types of dependency relations. The message passing process of GCN should be distinguished based on these syntactic information. To tackle this problem, we design a novel weighted graph convolutional network(WGCN) which can exploit rich syntactic information based on the feature combination. Furthermore, we utilize BERT instead of Bi-LSTM to generate contextualized representations as inputs for GCN and present an alignment method to keep word-level dependencies consistent with wordpiece unit of BERT. With our proposal, we are able to improve the state-of-the-art on four ABSA tasks out of six and two SA tasks out of three.

## introduction sentiment analysis(sa), also known as opinion mining, is the task of determining the polarity of a piece of text. commonly the classification is whether the text is expressing a negative or positive attitude towards a topic or a product. fine-grained sentiment analysis involves more than two sentiment classes (very negative, negative, neutral, positive and very positive). aspect-based sentiment analysis(absa) is one step further by assigning sentiment polarities to specific aspects of an involved entity or a topic. for example, comment on a restaurant saying "the restaurant was expensive, but the menu was great" has positive and negative attitudes for two aspects food and price. much progress has been made recently to advance the state-of-the-art on shared sa and absa tasks. contributions mainly come from two research directions. one is to take advantage of the pre-trained language models such as elmo @xcite , bert @xcite and xlnet @xcite , which are typically employed to extract contextual features of a piece of text for the final classifer. these models effectively alleviate the heavy effort of feature engineering of earlier work on sa and absa. further inventions have been proposed to better fine-tune these models. for instance, a recent work @xcite converts absa to a sentence pair classification task, where an auxiliary sentence is generated. it then finetunes the pre-trained model from bert for this new task. promising experimental results are observed. second line of research is to exploit the syntactic structures of subjective sentences with a belief that interactions between words need to be considered in sentiment analysis, which however is not sufficiently captured by even the latest attention-based models. @xcite quotes a concrete example "its size is ideal and the weight is acceptable", where acceptable is often incorrectly identified by attention models as the most attentive word to size. previous works in @xcite @xcite propose a recursive tree-structured model to compose sentence representation from its constituent phrases. @xcite presents a novel rvnn architecture to dynamically integrate comprehensive syntactic information derived from the sentence parsing structure and linguistic tags on word level. models using a graph convolutional network(gcn) over the dependency tree of a sentence have shown evident effectiveness in absa tasks. the argument is that gcn captures longrange syntactic relations that are obscure from the surface @xcite @xcite . though these efforts have substantially pushed up the state-of-the-art accuracy of sa and absa, some challenges remain for sentiment classification. for example, the aforementioned gcn-based models are designed to encode the dependency tree of a sentence, where the adjacency matrix is binary with 1 representing if there is a dependency relationship between two corresponding words and 0 for others. however, types of dependency relations are diversified and the corresponding words of each relation may have different part-of-speech(pos) tags. these syntactic information should also influence the message passing process of gcn. as it is shown in figure 1 , the relationship("det(vehicle-3, a-2)") has less influence on polarity than the relationship("nsubj(worthwhile-14, film-11)") in the sentence "as a vehicle to savour binoche 's skill , the film is well worthwhile". besides, as @xcite points, pitfalls of sa and absa like sentiment shifters (such as negations, double negations and but clauses) have not been well handled by current models. in this paper, we are motivated to encode more syntactic features and leverage both the pre-trained models and the syntactic parsing in a compositional way. we believe these are complementary to tackle the long-standing challenges for sa and absa. more specifically, we propose a weighted graph convolutional network(wgcn) to work with bert. wgcn improves on top of gcn to model rich syntactic information. the adjacency matrix in wgcn represents not only the binary representations of dependency relations, but also the types of dependency relations as well as the part-ofspeech(pos) categories of the involved words. we argue that the pos tag of each word is the category assigned in accordance with its syntactic function , hence has influence on the overall sentiment of the sentence as well as sentiments of aspects. all weights and embeddings in wgcn are trainable. details of this model will be provided later in this paper. wgcn reply on bert to extract contextualized representations as inputs for the wgcn layers. one challenge is the inconsistency between the wordpiece unit of bert, and the word-pairs considered in the dependency tree. we propose an alignment method to bridge this chasm. our contributions are summarized as follows: we refer to the whole architecture as bert-wgcn. the rest of the paper is organized as follows. section 2 gives a brief review of bert and gcn. section 3 elaborates on our proposed overall model architecture that integrates wgcn and bert, as well as how the model is trained respectively for sa and absa tasks. section 4 reports our experiments and analysis. ## review of gcn and bert graph convolutional network(kipf and welling, 2016) is an adaptation of the convolutional neural network @xcite for encoding unstructured data. given a graph with k nodes, we can obtain an adjacency matrix a where a ij is obtained based on the connection between node i and node j. in an l-layer gcn where h l-1 represents the output feature matrix at (l -1)-th layer and h l represents the output feature matrix at the l-th layer, a graph convolutional operation can be written as: ã = a + i k is the adjacency matrix with selfloops, where i k is the identity matrix. @xmath0. d-1 2 ã d-1 2 is the normalized adjacency matrix. w l is a linear transformation weight,and σ is a nonlinear function(e.g., relu). in each graph convolution, each node collects and processes information from its neighboring nodes. bert @xcite is one of the key innovations in the recent progress of contextualized representation learning inspired by tranformer @xcite . given a sentence @xmath1, ..., w n }, its tokenized sequential representation is {t 1 , t 2 , ..., t k }. transformer creates three vectors (query, key and value) for each sequence position, and then applies the attention mechanism for each position x i , using the query vector for x i as well as key and value vectors for all other positions. this computation can be presented as: instead of performing a single attention function, @xcite found it is beneficial to have multiple attention heads. bert built on transformers contains a number of layers(transformer blocks) l . each layer is identical with a fixed number of hidden units h and a fixed number of multi-threading self-attention heads a. particularly we use the bert large model with @xmath2, @xmath3. ## approach figure 2 gives an overview of the whole architecture. our model consists of 3 main components. first, the input sequence of text is parsed into wordbased syntactic features as inputs for wgcn. at the same time, the text is also directly fed into bert for wordpiece contextualized representations. one challenge here is the inconsistency between the wordpiece unit of bert and word-based syntactic features for wgcn. the second part is the reform of gcn to exploit rich syntactic features. the third component is the sentiment classifer for sa and absa. the components will be introduced separately in the rest of the section. ## conclusion in this paper we propose a novel weighted graph convolutional networks(wgcn) to work with bert on sentiment analysis and aspect-based sentiment analysis tasks. wgcn improves on top of gcn to model rich syntactic information including dependency relations as well as pos tags. bert is used as a powerful tool to extract contextual representations, which are then used as inputs to wgcn to derive the final vectors for classification. we propose an alignment approach to solve the token inconsistency issue between wgcn and bert. our experimental results with visualizations show the success of our proposal comparing to the baseline and previous approaches in the literature.