Syntax-Aware Opinion Role Labeling with Dependency Graph Convolutional Networks (2020)

Abstract: Opinion role labeling (ORL) is a fine-grained opinion analysis task and aims to answer “who expressed what kind of sentiment towards what?”. Due to the scarcity of labeled data, ORL remains challenging for data-driven methods. In this work, we try to enhance neural ORL models with syntactic knowledge by comparing and integrating different representations. We also propose dependency graph convolutional networks (DEPGCN) to encode parser information at different processing levels. In order to compensate for parser inaccuracy and reduce error propagation, we introduce multi-task learning (MTL) to train the parser and the ORL model simultaneously. We verify our methods on the benchmark MPQA corpus. The experimental results show that syntactic information is highly valuable for ORL, and our final MTL model effectively boosts the F1 score by 9.29 over the syntax-agnostic baseline. In addition, we find that the contributions from syntactic knowledge do not fully overlap with contextualized word representations (BERT). Our best model achieves 4.34 higher F1 score than the current state-ofthe-art.

## introduction opinion and sentiment analysis has a wide range of real-world applications like social media monitoring @xcite , stock market prediction @xcite , box office prediction @xcite , and general e-commerce applications @xcite @xcite . in particular, fine-grained opinion analysis aims to identify users' opinions in a text, including opinion expressions, holders of the opinions, targets of the opinions, target-dependent attitude, and intensity of opinions @xcite , which is very important for understanding political stance, customers' reviews, marketing trends, and other subjective information @xcite . as a typical fine-grained opinion mining task, opinion role labeling (orl) aims to identify different roles relevant to each opinion, i.e., who expressed what kind of sentiment towards what @xcite . due to the lack of large-scale labeled data, orl remains a challenging task to tackle. as a reference point, semantic role labeling (srl) is very similar to orl in the problem definition, but has 10 times more labeled data and thus achieves much higher performance than orl (80∼90 vs. 60∼70 in f1 score). motivated by the correlations between the two tasks, srl has been utilized to help the orl task by many previous studies @xcite @xcite . however, when opinion expressions and arguments compose complicated syntactic structures, it is difficult to correctly recognize the opinion arguments even with shallow semantic representation like srl @xcite . to compensate for the limited scale of labeled data for data-driven approaches, linguistic knowledge like syntax provides structural information representing human understanding of the text. naturally, dependency relations between words ease the discovering of opinion roles. taking the example in figure 1 , the target span is often incompletely recognized without syntactic dependency relations, missing either "facing chavez" or "challenge". for the similar srl task, many previous works have proposed to incorporate syntax into the neural models @xcite @xcite . in contrast, few studies in the recent years explore this line of research for orl. there are two barriers to apply syntactic dependency parsing to nlp tasks, i.e., 1) inaccuracy of the parsing results, and 2) error propagation of the processing pipeline. to overcome the first barrier, instead of employing the final discrete outputs (i.e., single 1-best dependency trees), we make use of the probability matrix of all dependency arcs (also can be viewed as an edge-weighted directed graph) before searching for the 1-best tree. such probabilistic representation of syntax provides more information while alleviating parsing errors. for the second barrier, considering that the pipeline methods are notorious for the error propagation problem, we introduce multi-task learning (mtl) frameworks, which have been widely used in many nlp models when predictions at various processing levels are needed @xcite . apart from the syntactic information, contextualized word representations like bert @xcite are widely used to compensate for the sparsity of task-specific training data. they compress distributional semantics of words from large corpora, making the local context fluent and natural. however, the long-distance dependencies between words are often ignored, which is ideally able to be captured by syntactic analysis. in summary, based on previous studies in using syntax to improve various tasks, this work investigates whether syntax can enhance the neural orl model. particularly, we try to answer the following three questions. based on our experiments, we observe that 1) compared with single 1-best parse trees, encoding the edge-weighted graphs achieves better re-sults, as the model is less sensitive to parsing errors while keeping richer structural information; 2) integrating various syntactic information, both explicit and implicit, boosts performance, and mtl framework can effectively alleviate the error propagation problem; and 3) contributions from syntactic information, especially from long-distance dependency relations, do not fully overlap with those from the contextualized word representations like bert. our overall model delivers a new stateof-the-art result on the benchmark mpqa corpus, with 4.34 absolute improvement over the previous best result. ## related work an opinion consists of several components, e.g., expressions, holders, and targets. some previous works focus on recognizing some components, whereas others try to recognize all components at the same time. @xcite and @xcite work entirely on labeling of the opinion expressions. @xcite and johansson and moschitti (2013) apply pipeline models to firstly predicting opinion expressions and then labeling holders and targets for each expression. joint models simultaneously identify all opinion components, predicting which role is related to which opinion @xcite @xcite . in this work, we follow the opinion role labeling (orl) task setting of @xcite and @xcite , and try to predict holders and targets for the given opinion expressions. previous works make use of srl resources to address the issue of data scarcity for orl, considering srl is highly related to orl and has a considerable amount of training data. inspired by the similarity between orl and srl in task definition, @xcite and @xcite address orl with a well-trained srl model by treating opinion expressions as semantic predicates, and opinion roles as semantic roles. @xcite take srl as an auxiliary task, and employ different mtl frameworks to learn the common grounds between orl and srl and distinguish task-specific knowledge. @xcite extract neural features from a welltrained srl model as srl-aware word representations, and then feed them into the input layer of orl, aiming to alleviate the error propagation problem. many previous works have shown that syntactic information is of great value for srl and other nlp tasks @xcite @xcite @xcite . @xcite use the relative position between predicate words and other words in a dependency tree to represent syntactic information, while roth and lapata (2016) employ lstm to obtain the embedding of a dependency path. @xcite and @xcite propose treelstm and graph convolution network (gcn) to encode the tree/graphstructural data respectively. both treelstm and gcn are commonly used techniques to encode parse trees @xcite @xcite . @xcite al. (2019a) and xia et al. (2019a) extract the hidden states from the lstm encoder of the parser model as syntax-aware word representations, and feed them to downstream tasks as extra inputs. in contrast, few works have proved that syntactic knowledge is useful in the neural orl models. @xcite integrate the shortest path features from dependency trees into a traditional crf-based orl model. to our best knowledge, this work is the first to investigate how to incorporate syntax into neural orl models. ## basic models the orl model aims to extract opinion-holdertarget structures from text by identifying the segments of these opinion arguments. the task can be modeled as a sequence labeling problem. we adopt the {bm eso} encoding schema to assign a tag for each word @xcite . following @xcite and @xcite , we focus on recognizing the holders and the targets for the given opinion expression and exploit a deep bilstm-crf-based model as our baseline. the figure 2-(a) shows the architecture of our orl baseline model, which is composed of three key components, i.e., the input layer, the bilstmbased encoder, and the crf-based decoder. given the input sentence @xmath0, w 2 , ..., w n and the opinion expression segment @xmath1, w s+1 , ..., w e (1 ≤ @xmath2, the input vector consists of the word embeddings and the expression-indicator embeddings as the following equation shows: the biaffine parser is the state-of-the-art dependency parser proposed by @xcite , as shown in figure 2-(b ). the parser contains a multi-layer bilstm layer for encoding the input sentence, followed by a biaffine transformation layer for computing the probabilities of all word pairs. then it searches for the highest-scoring and well-formed tree via the maximum spanning tree (mst) algorithm. the three cascaded layers, i.e., the bilstmbased encoder, the biaffine scorer, and the mst decoder, represent syntactic information at different levels. the encoder extracts the neural features from the input sentence and outputs hidden states (hdn), which can be regarded as implicit information. the 1-best output parse tree, on the other hand, conveys explicit syntactic structures. the biaffine scorer gives a probability matrix for all possible dependency arcs (also can be viewed as an edge-weighted directed graph), which represents richer explicit syntactic information than the 1-best parse tree. ## the syntax-aware approach despite of recent advances in dependency parsing @xcite , parsers still cannot output parse trees with high accuracy on out-ofdomain or irregular data. in this work, we exploit rich syntactic information contained in the edgeweighted graphs to mitigate the effects of parsing errors. specifically, we firstly employ graph convolutional networks (gcn) to encode the edgeweighted graphs, and then integrate them into different processing levels of orl with implicit parser hidden states. finally, we employ novel mtl frameworks to alleviate the error propagation problem further. ## results and analysis in this section, we first conduct experiments on the dev data to verify the effectiveness of our pro-2 in fact, we also investigate another typical use of bert, i.e., the fine-tuning method. however, the orl performance is much lower than the feature extraction method described above. besides, considering the training speed and flexibility in our proposed syntax-aware model, it is more flexible to adopt the feature extraction method, i.e., extracting bert outputs as extra word representations (frozen during training). posed approaches from several aspects: 1) how to effectively use explicit syntactic information; 2) usefulness of explicit vs. implicit syntax and their combination; 3) which mtl framework is most effective. then we present overall results on the test dataset, with and without bert. finally, we conduct detailed analysis to gain more insights.