Introducing Syntactic Structures into Target Opinion Word Extraction with Deep Learning (2020)

Abstract: Targeted opinion word extraction (TOWE) is a sub-task of aspect based sentiment analysis (ABSA) which aims to find the opinion words for a given aspect-term in a sentence. Despite their success for TOWE, the current deep learning models fail to exploit the syntactic information of the sentences that have been proved to be useful for TOWE in the prior research. In this work, we propose to incorporate the syntactic structures of the sentences into the deep learning models for TOWE, leveraging the syntax-based opinion possibility scores and the syntactic connections between the words. We also introduce a novel regularization technique to improve the performance of the deep learning models based on the representation distinctions between the words in TOWE. The proposed model is extensively analyzed and achieves the state-of-the-art performance on four benchmark datasets.

## introduction targeted opinion word extraction (towe) is an important task in aspect based sentiment analysis (absa) of sentiment analysis (sa). given a target word (also called aspect term) in the input sentence, the goal of towe is to identify the words in the sentence (called the target-oriented opinion words) that help to express the attitude of the author toward the aspect represented by the target word. for instance, as a running example, in the sentence "all warranties honored by xyz (what i thought was a reputable company) are disappointing.", "disappointing" is the opinion word for the target word "warranties" while the opinion words for the target word "company" would involve "reputable". among others, towe finds its applications in target-oriented sentiment analysis @xcite @xcite and opinion summarization @xcite . the early approach for towe has involved the rule-based and lexicon-based methods @xcite while the recent work has focused on deep learning models for this problem @xcite . one of the insights from the rule-based methods is that the syntactic structures (i.e., the parsing trees) of the sentences can provide useful information to improve the performance for towe @xcite . however, these syntactic structures have not been exploited in the current deep learning models for towe @xcite . consequently, in this work, we seek to fill in this gap by extracting useful knowledge from the syntactic structures to help the deep learning models learn better representations for towe. in particular, based on the dependency parsing trees, we envision two major syntactic information that can be complementarily beneficial for the deep learning models for towe, i.e., the syntax-based opinion possibility scores and syntactic word connections for representation learning. first, for the syntaxbased possibility scores, our intuition is that the closer words to the target word in the dependency tree of the input sentence tend to have better chance for being the opinion words for the target in towe. for instance, in our running example, the opinion word "disappointing" is sequentially far from its target word "warranties". however, in the de-pendency tree shown in figure 1 , "disappointing" is directly connected to "warranties", promoting the distance between "disappointing" and "warranties" (i.e., the length of the connecting path) in the dependency tree as an useful feature for towe. consequently, in this work, we propose to use the distances between the words and the target word in the dependency trees to obtain a score to represent how likely a word is an opinion word for towe (called syntax-based possibility scores). these possibility scores would then be introduced into the deep learning models to improve the representation learning for towe. in order to achieve such possibility score incorporation, we propose to employ the representation vectors for the words in the deep learning models to compute a model-based possibility score for each word in the sentence. the model-based possibility scores also aim to quantify the likelihood of being an opinion word for each word in the sentence; however, they are based on the internal representation learning mechanism of the deep learning models for towe. to this end, we propose to inject the information from the syntax-based possibility scores into the models for towe by enforcing the similarity/consistency between the syntax-based and model-based possibility scores for the words in the sentence. the rationale is to leverage the possibility score consistency to guide the representation learning process of the deep learning models (using the extracted syntactic information) to generate more effective representations for towe. in this work, we employ the ordered-neuron long short-term memory networks (on-lstm) @xcite to obtain the model-based possibility scores for the words in the sentences for towe. on-lstm introduces two additional gates into the original long short-term memory network (lstm) cells that facilitate the computation of the model-based possibility scores via the numbers of active neurons in the hidden vectors for each word. for the second type of syntactic information in this work, the main motivation is to further improve the representation vector computation for each word by leveraging the dependency connections between the words to infer the effective context words for each word in the sentence. in particular, motivated by our running example, we argue that the effective context words for the representation vector of a current word in towe involve the neighboring words of the current word and the target word in the dependency tree. for instance, consider the running example with "warranties" as the target word and "reputable" as the word we need to compute the representation vector. on the one hand, it is important to include the information of the neighboring words of "reputable" (i.e., "company") in the representation so the models can know the context for the current word (e.g., which object "reputable" is modifying). on the other hand, the information about the target word (i.e., "warranties" and possibly its neighboring words) should also be encoded in the representation vector for "reputable" so the models can be aware of the context of the target word and make appropriate comparison in the representation to decide the label (i.e., non-opinion word) for "reputable" in this case. note that this syntactic connection mechanism allows the models to de-emphasize the context information of "i" in the representation for "reputable" to improve the representation quality. consequently, in this work, we propose to formulate these intuitions into an importance score matrix whose cells quantify the contextual importance that a word would contribute to the representation vector of another word, given a target word for towe. these importance scores will be conditioned on the distances between the target word and the other words in the dependency tree. afterward, the score matrix will be consumed by a graph convolutional neural network (gcn) model @xcite to produce the final representation vectors for opinion word prediction. finally, in order to further improve the induced representation vectors for towe, we introduce a novel inductive bias that seeks to explicitly distinguish the representation vectors of the targetoriented opinion words and those for the other words in the sentence. we conduct extensive experiments to demonstrate the benefits of the proposed model, leading to the state-of-the-art performance for towe in several benchmark datasets. ## related work comparing to the related tasks, towe has been relatively less explored in the literature. in particular, the most related task of towe is opinion word extraction (owe) that aims to locate the terms used to express attitude in the sentences @xcite . a key difference between owe and towe is that owe does not require the opinion words to tie to any target words in the sentence while the opinion words in towe should be explicitly paired with a given target word. another related task for towe is opinion target extraction (ote) that attempts to identify the target words in the sentences @xcite @xcite @xcite . note that some previous works have also attempted to jointly predict the target and opinion words @xcite @xcite @xcite ; however, the target words are still not paired with their corresponding opinion words in these studies. as mentioned in the introduction, among a few previous work on towe, the main approaches include the rule-based methods (i.e., based on word distances or syntactic patterns) @xcite and the recent deep learning models @xcite . our model is different from the previous deep learning models as we exploit the syntactic information (i.e., dependency trees) for towe with deep learning. ## model the towe problem can be formulated as a sequence labeling task. formally, given a sentence w of n words: @xmath0, w 2 , . . . , w n with w t as the target word (1 ≤ @xmath1, the goal is to assign a label l i to each word w i so the label sequence @xmath2, l 2 , ..., l n for w can capture the target-oriented opinion words for w t . following the previous work @xcite , we use the bio tagging schema to encode the label l i for towe (i.e., l @xmath3, i, o} for being at the beginning, inside or outside of the opinion words respectively). our model for towe consists of four components that would be described in the following: (i) sentence encoding, (ii) syntax-model consistency, (iii) graph convolutional neural networks, and (iv) representation regularization. ## conclusion we propose a novel deep learning model for towe that seeks to incorporate the syntactic structures of the sentences into the model computation. two types of syntactic information are introduced in this work, i.e., the syntax-based possibility scores for words (integrated with the on-lstm model) and the syntactic connections between the words (applied with the gcn model with novel adjacency matrices). we also present a novel inductive bias to improve the model, leveraging the representation distinction between the words in towe. comprehensive analysis is done to demonstrate the effectiveness of the proposed model over four datasets.