Supervised Gradual Machine Learning for Aspect-Term Sentiment Analysis (2023)

Abstract: Recent work has shown that Aspect-Term Sentiment Analysis (ATSA) can be effectively performed by Gradual Machine Learning (GML). However, the performance of the current unsupervised solution is limited by inaccurate and insufficient knowledge conveyance. In this paper, we propose a supervised GML approach for ATSA, which can effectively exploit labeled training data to improve knowledge conveyance. It leverages binary polarity relations between instances, which can be either similar or opposite, to enable supervised knowledge conveyance. Besides the explicit polarity relations indicated by discourse structures, it also separately supervises a polarity classification DNN and a binary Siamese network to extract implicit polarity relations. The proposed approach fulfills knowledge conveyance by modeling detected relations as binary features in a factor graph. Our extensive experiments on real benchmark data show that it achieves the state-of-the-art performance across all the test workloads. Our work demonstrates clearly that, in collaboration with DNN for feature extraction, GML outperforms pure DNN solutions.

## introduction aspect-term sentiment analysis (atsa) is a classical fine-grained sentiment classification task @xcite . aiming to analyze detailed opinions towards certain aspects of an entity, it has attracted extensive research interests. in atsa, an aspect-term, also called target, has to explicitly appear in a review. for instance, consider the running example shown in the state-of-the-art solutions of atsa have been built upon pre-trained language models, such as lcf-bert @xcite , bat @xcite , ph-sum @xcite , and roberta+mlp @xcite , to name a few. it is noteworthy that the efficacy of these deep solutions depends on the independent and identically distributed (i.i.d.) assumption. however, in real scenarios, there may not be sufficient labeled training data; even if provided with sufficient training data, the distributions of training data and target data are almost certainly different to some extent. to alleviate the limitation of the i.i.d assumption, a solution based on the non-i.i.d paradigm of gradual machine learning (gml) has recently been proposed for atsa @xcite . gml begins with some easy instances, which can be automatically labeled by the machine with high accuracy, and then gradually labels more challenging instances by iterative knowledge conveyance in a factor graph. without exploiting labeled training data, the current unsupervised solution relies on sentiment lexicons and explicit polarity relations indicated by discourse structures to enable knowledge conveyance. an improved gml solution leverages unsupervised dnn to extract sentiment features beyond lexicons @xcite . it has been empirically shown that even without leveraging any labeled training data, unsupervised gml can achieve competitive performance compared with many supervised deep models. however, unsupervised sentiment the food is sinful. s @xcite the staff was really friendly. a supervised dnn can usually effectively separate the instances with different polarities. as a result, two instances appearing very close in its embedding space usually have the same polarity. therefore, we leverage a polarity classifier for the detection of polarity similarity between close neighbors in an embedding space. it can also be observed that in natural languages, there are many different types of patterns to associate opinion words with polarities. however, a polarity classifier may put the instances with the same polarity but different association patterns in far-away places in its embedding space. in comparison, metric learning can cluster the instances with the same polarity together while separating those with different polarities as far as possible @xcite ; it can thus align different association patterns with the same polarity. therefore, we also employ a siamese network for metric learning, which has been shown to perform well on semantic textual similarity tasks @xcite , to detect complementary polarity relations. a siamese network can detect both similar and opposite polarity relations between two arbitrary instances, which may be far away in an embedding space. finally, our proposed approach fulfills knowledge conveyance by modeling polarity relations as binary features in a factor graph. in our implementation, we use the state-of-the-art dnn model for atsa, roberta+mlp @xcite , to capture neighborhood-based polarity similarity while adapting the siamese network @xcite , the classical model of deep metric learning @xcite , to extract arbitrary polarity relations. it is worth pointing out that our work is orthogonal to the research on polarity classification dnns and siamese networks in that the proposed approach can easily accommodate new polarity classifiers and siamese network models. the main contributions of this paper can be summarized as follows: our extensive experiments have shown that it consistently achieves the state-of-the-art performance across all the test datasets. ## related work sentiment analysis at different granularity levels, including document, sentence, and aspect levels, has been extensively studied in the literature @xcite . at the document (resp., sentence) level, its goal is to detect the polarity of the entire document (resp., sentence) without regard to the mentioned aspects @xcite @xcite . the state-of-the-art solutions for document-level and sentence-level sentiment analysis have been built upon various dnn models @xcite @xcite . however, they cannot be directly applied to the finer-grained aspect-level sentiment analysis because a document or sentence may express different polarities towards different aspects. the task of aspect-level sentiment analysis has been further classified into two finer subtasks, aspect-term sentiment analysis (atsa) and aspect-category sentiment analysis (acsa) @xcite . atsa aims to predict the sentiment polarity associated with an explicit aspect term appearing in the text while acsa deals with both explicit and implicit aspects. in this paper, we focus on the far more popular subtask of atsa. but, as shown in our experimental evaluation, our proposed approach is also applicable to acsa. even though early work on deep learning for atsa employed non-attention models @xcite , more recent proposals leveraged various attention mechanisms to output aspect-specific sentiment features, such as interactive attention networks @xcite , recurrent attention network @xcite , content attention model @xcite , multi-grained attention network @xcite , segmentation attention network @xcite , attention-over-attention neural networks @xcite , and effective attention modeling @xcite to name a few. most recently, the focus has experienced a considerable shift towards how to leverage pre-trained language models for atsa, e.g., bert-spc @xcite , aen-bert (attentional encoder network) @xcite , and lcf-bert (local context focus) @xcite . since bert is trained on wikipedia articles and has limited ability to understand review texts, xu proposed to first post-train bert on both domain knowledge and task knowledge, and then fine-tune the resulting model of bert-pt on supervised domain data @xcite . since then, many models built upon bert-pt have been proposed @xcite . other variants of bert for atsa include adapted bert (bert-ada) @xcite , robustly optimized bert (roberta) @xcite , and bert with disentangled attention (deberta) @xcite . since syntax structures are helpful for aspects to find their contextual words, many syntax-enhanced models have been recently proposed for atsa, such as proximity-weighted convolution network (pwcn) @xcite , relational graph attention network (rgat) @xcite , graph convolutional networks (gcn) @xcite , dependency graph enhanced dual-transformer network (dgedt) @xcite , type-aware graph convolutional networks (t-gcn) @xcite , and knowledge-aware gated recurrent memory network with dual syntax graph (kagrmn-dsg) @xcite . they focused on how to exploit explicit syntactic information provided by dependency-based parse trees. other proposals investigated how to induce implicit syntactic information from pre-trained models @xcite . the gml paradigm was first proposed for the task of entity resolution @xcite . since then, it has also been applied to the task of atsa @xcite . without exploiting labeled training data, the performance of unsupervised gml is usually limited by inaccurate and insufficient knowledge conveyance. in this paper, we focus on how to leverage labeled examples to improve gradual learning for atsa. ## the gml framework in this section, we illustrate the gml framework by the existing unsupervised gml solution for atsa @xcite . given a corpus of reviews, r, the goal of atsa is to predict the sentiment polarity of each aspect unit in r, t @xmath0, s k , a l ), where r j denotes a review, s k denotes a sentence in the review r j , and a l denotes an explicit aspect appearing in the sentence s k . in this paper, we suppose that an aspect polarity is either positive or negative. as shown in figure 1 , the framework consists of the following three essential steps: ## empirical evaluation in this section, we empirically evaluate the performance of the proposed approach, denoted by s-gml, on real benchmark data. we compare s-gml with the existing gml solution as well as the state-of-the-art dnn models. even though the focus of this paper is on atsa, the proposed approach can also be applied to the task of acsa. therefore, we also compare s-gml with its alternatives on acsa. the rest of this section is organized as follows: section 5.1 describes the experimental setup. section 5.2 presents the evaluation results on atsa. section 5.3 presents the evaluation results of parameter sensitivity. section 5.4 presents the evaluation results on acsa. ## conclusion and future work in this paper, we have proposed a novel supervised gml approach for atsa that can effectively exploit labeled examples to improve gradual learning. it leverages both polarity classification dnn and siamese network to extract implicit polarity relations between instances, and then instills them into a factor graph to enable supervised knowledge conveyance. our extensive empirical study has validated its efficacy. our work has demonstrated clearly that in collaboration with dnn for feature extraction, gml can outperform pure dnn solutions. for future work, it can be observed that even though the proposed solution is built upon the specific polarity classifier and siamese network for aspect-level sentiment analysis, similar classifiers and siamese networks are readily available or can be constructed for other binary classification tasks, especially nlp tasks. therefore, the proposed collaboration approach of dnn and gml can be potentially generalized to other binary classification tasks.