BERT 4 GCN : Using BERT Intermediate Layers to Augment GCN for Aspect-based Sentiment Classification (2021)

Abstract: Graph-based Aspect-based Sentiment Classification (ABSC) approaches have yielded state-of-the-art results, expecially when equipped with contextual word embedding from pre-training language models (PLMs). However, they ignore sequential features of the context and have not yet made the best of PLMs. In this paper, we propose a novel model, BERT4GCN, which integrates the grammatical sequential features from the PLM of BERT, and the syntactic knowledge from dependency graphs. BERT4GCN utilizes outputs from intermediate layers of BERT and positional information between words to augment GCN (Graph Convolutional Network) to better encode the dependency graphs for the downstream classification. Experimental results demonstrate that the proposed BERT4GCN outperforms all state-of-the-art baselines, justifying that augmenting GCN with the grammatical features from intermediate layers of BERT can significantly empower ABSC models.

## introduction aspect-based sentiment classification (absc), a fine-grained sentiment classification task in the field of sentiment analysis, aims at identifying the sentiment polarities of aspects explicitly given in sentences. for example, given the sentence "the price is reasonable although the service is poor.", absc needs to correctly assign a positive polarity to price and a negative one to service. intuitively, matching aspects with their corresponding opinion expressions is the core of absc task. some of previous deep learning approaches @xcite @xcite use various types of attention mechanisms to model the relationship between aspects and opinion expressions in an implicit way. however, these attention-based models do not take * corresponding author (qingliang chen). in order to better pair aspects and their corresponding opinion expressions, the syntactic features must be taken into account. recent work such as @xcite @xcite @xcite employs either graph convolutional networks (gcns) @xcite or graph attention networks (gats) @xcite on the dependency graph, in which words in the sentence are nodes in the graph and grammatical relations are edge labels. the dependency graph they use can be obtained from either ordinary dependency tree parser or reshaping it using heuristic rules. these models can achieve promisingly better performance with this additional grammatical features, especially when they incorporate the contextual word embedding, such as bert @xcite . meanwhile, bertology researchers have investigated what linguistic knowledge can be learned from unlabeled data by language models @xcite @xcite . and it has been shown that bert captures a rich hierarchy of linguistic information, with surface features in lower layers, syntactic features in middle layers and semantic features in higher layers @xcite . inspired by @xcite , we propose a novel framework of bert4gcn to make the best of the rich hierarchy of linguistic information from bert in this paper. specifically, we firstly encode the context with a bilstm (bidirectional long short-term memory) to capture the contextual information regarding word orders. then, we use the hidden states of bilstm to initiate node representations and employ multi-layer gcn on the dependency graph. next, we combine the node representations of each layer of gcn with the hidden states of some intermediate layer of bert, i.e., for the neighborhood aggregation of gcn. in this way, bert4gcn can fuse grammatical sequential features with graph-based representations. besides, we further prune and add the edge of the dependency graph based on self-attention weights in the transformer encoder @xcite of bert to deal with parsing errors and make dependency graph better suit absc task. in addition, we develop a method which incorporates relative positional embedding in node representations to make gcn position-aware. our contributions in this paper are summarized as follows: more recently, a dual structure model has been proposed in @xcite , jointly taking the sequential features and dependency graph knowledge together. our bert4gcn model has a similar structure to that in @xcite , but ours utilizes the grammatical sequential features directly from intermediate layers in bert, instead of learning from scratch on other sources. ## conclusion in this paper, we propose a bert4gcn model which integrates the grammatical sequential features from bert along with the syntactic knowledge from dependency graphs. the proposed model utilizes intermediate layers of bert, which contain rich and helpful linguistic knowledge, to augment gcn, and furthermore, incorporates relative positional information of words to be positionaware. finally, experimental results show that our model achieves new state-of-the-art performance on prestigious benchmarks.