Learnable Dependency-based Double Graph Structure for Aspect-based Sentiment Analysis (2022)

Abstract: Dependency tree-based methods might be susceptible to the dependency tree due to that they inevitably introduce noisy information and neglect the rich relation information between words. In this paper, we propose a learnable dependency-based double graph (LD2G) model for aspect-based sentiment classification. We use multi-task learning for domain adaptive pretraining, which combines Biaffine Attention and Mask Language Model by incorporating features such as structure, relations and linguistic features in the sentiment text. Then we utilize the dependency enhanced double graph-based MPNN to deeply fuse structure features and relation features that are affected with each other for ASC. Experiment on four benchmark datasets shows that our model is superior to the state-of-the-art approaches.

## introduction aspect-based sentiment analysis (absa) is a finegrained sentiment analysis task for extracting the emotional orientation of the given one or more aspects in a sentence. absa is generally divided into two parts: aspect extraction (ae) and aspect sentiment classification (asc). we will focus on asc task in this paper. dependency-based approaches have illustrated promising performance by utilizing the spatial structure of the dependency tree (dt). early studies focus on the syntax feature enhanced aspectbased sentiment analysis (absa) models @xcite . dependencybased analysis currently tends to use graph neural networks to learn representation over dt due to that graph neural network can directly learn on graph data including dts @xcite . dt is regarded as an undirected tree structure to produce a symmetry adjacency matrix for graph convolutional networks (gcn) @xcite . @xcite use gnns to capture implicit relation between aspect words and others and achieved encouraged results. @xcite utilized reinforcement learning for aspectbased sentiment classification (asc). in this paper, we consider how to effectively fuse structure and relation features for asc. the following two questions: 1) how to utilize relations of edges between words for improving asc? most of existing approaches consider the structure information and neglected the rich edge relations between words with exception to the works @xcite @xcite . however, these works consider graph features (i.e., structure, edge relations, tree distance) as different input sources, and do not explore how to effectively fuse various features because these features are affected with each other. 2) how to effectively handle the multi-feature noisy information from dt for improving asc? existing approaches directly utilize the dt generated by a tool like spacy. direct application of dt is susceptible to noisy information in the dependency tree @xcite , which may achieve lower performance compared to using only the flat structure. dt includes 45 kinds of relations such as nsubj, amod and conj, and so on. dependency parsing requires effective way to reduce dt's noisy information. to handle the two problems, we propose a learnable dependency-based double graph (ld2g) model for aspect-based sentiment classification. we use multi-task learning (mtl) for domain adaptive pretraining and reducing dt's noisy information, by combining biaffine attention model (bam) and mask language model (mlm) and learnable dependency-based double graph structure for aspect-based sentiment analysis yinglong ma and yunhe pang school of control and computer engineering north china electric power university beinong road #2, beijing 102206, china {yinglongma, pangyunhe}@ncepu.edu.cn parsing structure, relations and linguistic features. we utilize the dependency enhanced double graphbased mpnn to deeply fuse structure features and relation features that are affected with each other for asc. experiment on four benchmark datasets shows that our model is superior to the state-of-theart approaches. to the best of our knowledge, ld2g is the first work that jointly considers the structure and relations features in a unified domain adaptive pretraining framework. the main contributions of this paper are as follows. ## related work generally, sentiment analysis aims to predict the sentiment of a sentence or a document. in contrast absa is a fine-grained task, which is to perform sentiment analysis on aspects by first identifying aspect words and then predicting the sentiment of the aspect. early methods mainly use machine learning methods, but these methods are more dependent on manual feature labeling @xcite . later, neural network approaches were found to have better performance. a common method was to model the sequence of aspect words and context. the popular lstm model and attention mechanism were commonly used to fuse the context semantics and obtain the important parts from it. in recent years, approaches based on pre-trained model has become the main method for contextual semantic integration, which brings considerable improvement to sentiment analysis. another kind of approach takes advantage of the spatial structure of syntax features by deep learning methods. early work has studied the syntax feature enhanced absa models @xcite . dependency tree and constituent tree can represent the syntactic information between the words, but these approaches based on them has not become more popular until deep learning method was applied. after the improvements brought by deep learning, syntax features can effectively improve the performance of absa task. in recent years, syntax approaches tend to use graph neural networks to represent the dependency tree, because gnns can directly model non-euclidean data which fits dependency tree well. @xcite combined bi-lstm and gcn @xcite to extract feature from both linear and spatial word representations, in which dependency tree is seemed as an undirected dependency tree to produce a symmetry adjacency matrix for gcn. @xcite generated an induced tree structure based on the pre-trained model instead of dependency tree, and the experiment shows that pre-trained models can optimize the induced tree in the following absa tasks. @xcite shortened the distance between aspect word and emotional words by learning a new aspect-centric tree. @xcite ) combined a dependency graph and an attention-based graph to learn representation based on gnn. there are some approaches such as @xcite @xcite , further explored the use of specific relation on the dependency trees. these works utilize the original features of the dependency tress. dependency-based methods add effective syntactic features to sentiment analysis tasks, but at the same time, they inevitably inherit wrong dependency tree information, which brings adverse influence on downstream task. therefore, it has become a crucial problem how to utilize relations of edges between words and handle the multi-feature noisy information effectively from dt for improving asc ## ld2g model the ld2g model overall is shown in figure 1 . we use a two-stage approach to implement aspect sentiment analysis: 1) domain adapted pretraining, and 2) double graph mpnn-based aspect sentiment classification. the whole model is divided into three modules: bert module, biaffine parser and message passing neural network. we first apply mtl on bert and biaffine parser module for domain adaptive pretraining by combining bam and mlm, then obtain the sparse score matrix of structure graph and the feature matrix of edge relations from bam. both them are further fused into a dependency enhanced double graph matrix. the ld2g model can learn from the dependency graphs and optimize the dependency graph structure during the training process. ## conclusions and future work in this paper, we proposed an ld2g model to fuse the syntactical structure and relation features for asc. we use an mtl method for domain adaptive pretraining by combining biaffine attention and mlm model. a dependency enhanced double graph-based mpnn is utilized to deeply fuse structure features and relation features that are affected with each other for asc. experiments results show that our model is superior to the state-of-the-art alternatives and the utilization of double graph-based dependency enhanced structure indeed contributes to asc performance improvement. in the future, we will analyze combinations of linguistic features and priori knowledge mining for absa.