Attention Transfer Network for Aspect-level Sentiment Classification (2020)

Abstract: Aspect-level sentiment classification (ASC) aims to detect the sentiment polarity of a given opinion target in a sentence. In neural network-based methods for ASC, most works employ the attention mechanism to capture the corresponding sentiment words of the opinion target, then aggregate them as evidence to infer the sentiment of the target. However, aspect-level datasets are all relatively small-scale due to the complexity of annotation. Data scarcity causes the attention mechanism sometimes to fail to focus on the corresponding sentiment words of the target, which finally weakens the performance of neural models. To address the issue, we propose a novel Attention Transfer Network (ATN) in this paper, which can successfully exploit attention knowledge from resource-rich document-level sentiment classification datasets to improve the attention capability of the aspect-level sentiment classification task. In the ATN model, we design two different methods to transfer attention knowledge and conduct experiments on two ASC benchmark datasets. Extensive experimental results show that our methods consistently outperform state-of-the-art works. Further analysis also validates the effectiveness of ATN.

## introduction aspect-level sentiment classification (asc) is a fundamental task in sentiment analysis @xcite @xcite , which aims to infer the sentiment polarity (e.g. positive, neutral, negative) of a given opinion target in a review sentence. an opinion target, also known as aspect term, refers to a word or a phrase in review describing an aspect of an entity. for example, the sentence "the tastes are great, but the service is dreadful" consists of two opinion targets, namely "tastes" and "service". user's sentiment towards the opinion target "tastes" is positive while negative in terms of target "service". traditional methods usually focus on designing a set of features such as bag-of-words or sentiment lexicon to train a classifier (e.g., svm) for asc @xcite . motivated by the great success of deep learning in computer vision @xcite , speech recognition @xcite and natural language processing @xcite , recent works use neural networks to learn low-dimensional and continuous text representations without any feature engineering, and achieve competitive results on the asc task @xcite ). from the above example, we can see that a sentence sometimes refers to several opinion targets and they may express different sentiment polarities, thus one main challenge of asc is to separate different opinion contexts for different targets. to this end, abundant state-of-the-art works employ attention mechanism @xcite to capture sentiment words related to the given target, and then aggregate them to make sentiment prediction @xcite @xcite @xcite . despite the effectiveness of attention mechanism, we argue that it fails to reach the full potential due to the limited asc labeled data. it is well-known that the promising results of deep learning heavily rely on sufficient training data. however, the annotation of asc data is very labour-intensive and expensive in real-world scenarios, because annotators need to not only identify all opinion targets in a sentence but also determine their corresponding sentiment polarity. the difficulty of annotation leads to that existing public aspect-level datasets are all relatively small-scale, which finally limits the potential of attention mechanism. despite the lack of asc data, enormous labeled data of document-level sentiment classification (dsc) are available at online review sites such as amazon and yelp. these reviews contain substantial sentiment knowledge and semantic patterns. therefore, one meaningful but challenging research question is how to leverage resource-rich dsc data to improve the low-resource task asc. for this purpose, @xcite design the pret+mult framework to transfer sentiment knowledge from dsc data to asc task through sharing shallow embedding and lstm layer. inspired by the capsule network @xcite , @xcite propose transcap to share bottom three capsule layers, then separate two tasks only in the last classcap layer. fundamentally, pret+mult and transcap improve asc by sharing parameters and multi-task learning, but they cannot accurately control and interpret what knowledge to be transferred. in this work, we directly focus on the aforementioned attention issue in the asc task and propose a novel framework, attention transfer network (atn), to explicitly transfer attention knowledge from the dsc task for improving the attention capability of the asc task. compared with pret+mult and transcap, our model achieves better results and retains good interpretability. in the atn framework, we adopt two attention-based bilstm networks, respectively, as the dsc module and base asc module, and propose two different methods to transfer attention from dsc to asc. the first transfer approach is called attention guidance. specifically, we first pre-train an attentionbased bilstm on large-scale dsc data, then exploit the attention weights from the dsc module as a learning signal to guide the asc module to capture sentiment clues more accurately, thereby acheiving improvements. the second approach adopts the way of attention fusion, and directly incorporates the attention weights of the dsc module into the asc module. the two approaches work in different ways and have their different advantages. attention guidance aims to learn the attention ability of the dsc module and has faster inference speed, since it does not use external attention from dsc during the testing stage. in contrast, attention fusion can leverage the attention knowledge of the dsc module during the testing stage and make more comprehensive predictions. we conduct experiments on two benchmark datasets to evaluate different methods. the results indicate that the atn model can be substantially improved by incorporating the two attention transfer approaches, and outperforms all compared methods on the asc task. ## model figure 1 shows the overall architecture of the attention transfer network (atn). it mainly consists of four parts: the pre-trained dsc module, the base asc module, and two attention transfer approaches. in this section, we will first give the task formalization of asc and dsc, then introduce the attention-based pre-trained dsc module and base asc module. finally, we present the details of our proposed two attention transfer approaches, namely attention guidance and attention fusion. ## conclusion insufficient labeled data limits the effectiveness of attention-based models for the asc task. in this paper, we propose a novel attention transfer framework, in which two different attention transfer methods are designed to exploit attention knowledge from resource-rich document-level sentiment classification corpus to enhance the attention process of resource-poor aspect-level sentiment classification, finally achieving the goal of improving the performance of asc. experimental results indicate that our approaches outperform the state-of-the-art works. further analysis validates the effectiveness and benefits of transferring the attention knowledge from dsc data for the asc task.