Syntactically Aware Cross-Domain Aspect and Opinion Terms Extraction (2020)

Abstract: A fundamental task of fine-grained sentiment analysis is aspect and opinion terms extraction. Supervised-learning approaches have shown good results for this task; however, they fail to scale across domains where labeled data is lacking. Non pre-trained unsupervised domain adaptation methods that incorporate external linguistic knowledge have proven effective in transferring aspect and opinion knowledge from a labeled source domain to un-labeled target domains; however, pre-trained transformer-based models like BERT and RoBERTa already exhibit substantial syntactic knowledge. In this paper, we propose a method for incorporating external linguistic information into a self-attention mechanism coupled with the BERT model. This enables leveraging the intrinsic knowledge existing within BERT together with externally introduced syntactic information, to bridge the gap across domains. We successfully demonstrate enhanced results on three benchmark datasets.

## introduction a fundamental task of fine-grained sentiment analysis is aspect and opinion terms extraction. for example, in the sentence "the chocolate cake was incredible", the aspect term is chocolate cake and the opinion term is incredible. most of the work related to aspect and opinion term extraction is formulated as a supervised sequence-tagging task. rnn-based models @xcite and transformer-based models showed promising results in single-domain setups where the training and the testing data are from the same domain. however, these approaches typically do not scale across different domains, where only unlabeled data is available for the target domain, since aspect terms from two different domains are usually semantically different hence separated in the embedding space. for example, frequent aspect terms in the restaurant domain, like salad and dessert, have little or no semantic relatedness to frequent aspect terms in the laptop domain, like screen size and battery life. to date, only a handful of approaches for unsupervised domain adaptation of aspect and opinion term extraction have been proposed. it has been shown that syntactic information is important for identifying aspect and opinion terms @xcite . a recent line of work, based on non pre-trained models, encodes dependency-based aspect extraction rules @xcite or automatically-generated dependency relations @xcite , as auxiliary supervision for non pre-trained models. this recent line of work demonstrates effective domain adaptation by incorporating syntactic knowledge into non pre-trained models during their training step. subsequently, recent studies @xcite show that pre-trained transformer-based models such as bert @xcite and roberta @xcite already exhibit substantial linguistic knowledge. in this paper we examine whether the incorporation of external syntactic knowledge into pre-trained models, contributes to bridging the gap across domains. for this purpose, we propose an approach for unsupervised domainadaptation of aspect and opinion terms extraction based on incorporating linguistic knowledge into a pretrained bert model. specifically, inspired by @xcite , we incorporate externally-generated dependency relations into a self-attention mechanism that is coupled with the pre-trained bert model (stickland and murray, 2019), where the external information is introduced during the fine-tuning and testing stages of the model. ## motivation and background formally, the task of aspect and opinion terms extraction can be formulated as a sequence tagging task. the input is a sequence of tokens @xmath0, x 2 , .., x n } where the objective is to predict a corresponding sequence of labels @xmath1, y 2 , ..., y n } with y @xmath2, ia, bo, io, n }, where ba, bo, ia and io represent a beginning of aspect/opinion and inside of aspect/opinion, respectively, and n represents all other tokens. the goal of unsupervised domain adaptation is to predict the token-level labels y t i of unlabeled target domain sentences d @xmath3, given a set of labeled sentences from a source domain d @xmath4, y s j )}. it was observed that aspect and opinion terms maintain often-occurring syntactic patterns @xcite . consider for example, a sentence from the laptop domain "the display is absolutely wonderful" and a sentence from the restaurant domain "the cheesecake was simply wonderful". in the first sentence, an nsubj dependency relation exists between the opinion term ('wonderful') and the aspect term ('display'). assuming that the pattern aspect-nsubj-opinion is frequently observed in the laptop domain, then the term cheesecake can be extracted as an aspect term in the restaurant domain (figure 1 ). this domain-independent trait of the syntactic structure can be leveraged for transferring knowledge from a labeled source domain to an unlabeled target domain. based on this notion, @xcite proposed using dependency-based aspect extraction rules as auxiliary supervision for an rnn network. however, this method depends on the quality of manually-crafted rules. @xcite addressed this issue by automatically encoding dependency relations into the hidden representations of words, thus shifting the representations of different aspect terms having similar dependency relations, close to each other. @xcite have further enhanced this model by integrating a conditional domain-adversarial network that encodes both word features and syntactic parent relation types. analyses of pre-trained transformer-based models like bert reveal substantial syntactic information captured within their attention mechanisms; however, those analyses also show that for many syntactic relations bert only slightly improve over a simple baseline @xcite . our goal is to design a neural network model that leverages both the information captured in the pre-trained model, and externally introduced syntactic information, to bridge the gap between the source and target domains. ## the proposed model the basis for our model is a pre-trained bert-base model @xcite with a fully connected sequence tagging classifier on top. inspired by the work of @xcite , we incorporate dependency relations into a self-attention mechanism denoting a syntactically-aware attention head. our approach differs from previous approaches which modify an existing self-attention head within a transformer-based model and train it from scratch. our method modifies the bert function by adding syntactically-aware self-attention heads in parallel to the bert model (stickland and murray, 2019), and introduce the syntactic knowledge during the fine-tuning and testing stages. this leaves the original pre-trained model intact, enabling the model to utilize both the external linguistic information that is incorporated into the model and the intrinsic knowledge gained during the pre-training stage of the model. we refer to this model as syntactically-aware extended attention layer (sa-exal). multi-head self-attention. the basis of our implementation is bert's multi-head self-attention mechanism @xcite , which consists of i scaled dot-product attention heads. for each attention head i, the hidden token representations h @xmath5, at the input of layer l, are projected to key, query and value representations k i , q i and v i of dimensions t × d k , where t is the number of tokens in the input sequence and d @xmath6. attention head i denotes attention weights that are a distinct distribution of every input token over all other tokens in the sequence: the output of attention head i is denoted by m @xmath7, where m i is a t × t matrix, in which each row t, represents a weighted sum of the value representations of all other tokens with respect to token t. finally, the outputs of all i attention heads are concatenated and projected through a feed-forward(ff) network: @xmath8, m 2 , ..., m i ). ## conclusion we propose a method for incorporating external linguistic information into a self-attention mechanism coupled with the bert model. we demonstrate that this model leverages both the intrinsic knowledge existing within the pre-trained model and the externally introduced syntactic information, to bridge the gap across domains.