Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification (2021)

Abstract: Recent work on aspect-level sentiment classification has demonstrated the efficacy of incorporating syntactic structures such as dependency trees with graph neural networks (GNN), but these approaches are usually vulnerable to parsing errors. To better leverage syntactic information in the face of unavoidable errors, we propose a simple yet effective graph ensemble technique, GraphMerge, to make use of the predictions from different parsers. Instead of assigning one set of model parameters to each dependency tree, we first combine the dependency relations from different parses before applying GNNs over the resulting graph. This allows GNN models to be robust to parse errors at no additional computational cost, and helps avoid overparameterization and overfitting from GNN layer stacking by introducing more connectivity into the ensemble graph. Our experiments on the SemEval 2014 Task 4 and ACL 14 Twitter datasets show that our GraphMerge model not only outperforms models with single dependency tree, but also beats other ensemble models without adding model parameters.

## introduction aspect-level sentiment classification is a finegrained sentiment analysis task, which aims to identify the sentiment polarity (e.g., positive, negative or neutral) of a specific aspect term in a sentence. for example, in "the exterior, unlike the food, is unwelcoming.", the polarities of aspect terms "exterior" and "food" are negative and positive, respectively. this task has many applications, such as assisting customers to filter online reviews or make purchase decisions on e-commerce websites. recent studies have shown that syntactic information such as dependency trees is very effective in capturing long-range syntactic relations that are obscure from the surface form @xcite . several successful approaches employed food the exterior unlike the , is unwelcoming , . figure 1 : an example where an incorrect parse (above the sentence) can mislead aspect-level sentiment classification for the term "food" by connecting it to the negative sentiment word "unwelcoming" by mistake. although having its own issues, the parse below correctly captures the main syntactic structure between the aspect terms "exterior", "food" and the sentiment word, and is more likely to lead to a correct prediction. in this paper, we propose graphmerge, a graph ensemble technique to help dependency-based models mitigate the effect of parsing errors. our technique is based on the observation that different parsers, especially ones with different inductive biases, often err in different ways. for instance, in figure 1 , the green parse under the sentence is incorrect around "unlike the food", but it nevertheless correctly associates "unwelcoming" with the other aspect term "exterior", and therefore is less likely to mislead model predictions. given dependency trees from multiple parses, instead of assigning each dependency tree a separate set of model parameters and ensembling model predictions or dependency-based representations of the same input, we propose to combine the different dependency trees before applying representation learners such as gnns. specifically, we take the union of the edges in all dependency trees from different parsers to construct an ensemble graph, before applying gnns over it. this exposes the gnn model to various graph hypotheses at once, and allows the model to learn to favor edges that contribute more to the task. to retain the syntactic dependency information between words in the original dependency trees, we also define two different edge typesparent-to-children and children-to-parent-which are encoded by applying relational graph attention networks (rgat) @xcite on the ensemble graph. our approach has several advantages. firstly, since graphmerge combines dependency trees from different parsers, the gnn models can be exposed to multiple parsing hypotheses and learn to choose edges that are more suitable for the task from data. as a result, the model is less reliant on any specific parser and more robust to parsing errors. secondly, this improved robustness to parsing errors does not require any additional computational cost, since we are still applying gnns to a single graph with the same number of nodes. last but not least, graphmerge helps prevent gnns from overfitting by limiting over-parameterization. aside from keeping the gnn computation over a single graph to avoid separate parameterization for each parse tree, graphmerge also introduces more edges in the graph when parses differ, which reduces the diameter of graphs. as a result, fewer layers of gnns are needed to learn good representations from the graph, alleviating the oversmoothing problem @xcite . to summarize, the main contribution of our work are the following: much recent work on aspect-level sentiment classification has focused on applying attention mechanisms (e.g., co-attention, self attention, and hierarchical attention) to sequence models such recurrent neural networks (rnns) @xcite @xcite @xcite @xcite @xcite . in a similar vein, pretrained transformer language models such as bert @xcite have also been applied to this task, which operates directly on word sequences @xcite @xcite . in parallel, researchers have also found syntactic information to be helpful for this task, and incorporated it into aspect-level sentiment classification models in the form of dependency trees @xcite as well as constituency trees @xcite . more recently, researchers have developed robust dependency-based models with the help of gnns that operate either directly on dependency trees @xcite @xcite , as well as reshaped dependency trees that center around aspect terms @xcite . while most recent work stack gnns on top of bert models, @xcite have also reported gains by jointly learning the two with a mutual biaffine attention mechanism. despite the success of these dependency-based models, they are usually vulnerable to parse errors since they rely on a single parser. @xcite used a dependency forest to combine multiple dependency trees, however they tackled the sentence-level sentiment analysis task instead, and their proposed ensemble technique is also significantly different from ours. furthermore, most prior work that leverage gnns to encode dependency information treats the dependency tree as an undirected graph, therefore ignores the syntactic relation between words in the sentence. ## proposed model we are interested in the problem of predicting the sentiment polarity of an aspect term in a given sentence. specifically, given a sentence of n words {w 1 , w 2 , . . . , w τ , . . . , w τ +t , . . . , w n } where {w τ , w τ +1 , . . . , w τ +t-1 } is the aspect term, the goal is to classify the sentiment polarity toward the term as positive, negative, or neutral. applying gnns over dependency trees is shown effective to solve this problem, however it is vulnerable to parsing errors. therefore, we propose a graph-merge technique to utilize multiple dependency trees to improve robustness to parsing errors. in this section, we will first introduce graphmerge, our proposed graph ensemble technique, then introduce the gnn model over graphmerge graph for aspect-level sentiment analysis. ## conclusion we propose a simple yet effective graph-ensemble technique, graphmerge, to combine multiple dependency trees for aspect-level sentiment analysis. by taking the union of edges from different parsers, graphmerge allows graph neural model to be robust to parse errors without additional parameters or computational cost. with different edge types to capture the original syntactic dependency in parse trees, our model outperforms previous state-of-theart models, single-parse models, as well as traditional ensemble models on three aspect-level sentiment classification benchmark datasets.