Incorporating Dynamic Semantics into Pre-Trained Language Model for Aspect-based Sentiment Analysis (2022)

Abstract: Aspect-based sentiment analysis (ABSA) predicts sentiment polarity towards a specific aspect in the given sentence. While pre-trained language models such as BERT have achieved great success, incorporating dynamic semantic changes into ABSA remains challenging. To this end, in this paper, we propose to address this problem by Dynamic Re-weighting BERT (DR-BERT), a novel method designed to learn dynamic aspect-oriented semantics for ABSA. Specifically, we first take the Stack-BERT layers as a primary encoder to grasp the overall semantic of the sentence and then fine-tune it by incorporating a lightweight Dynamic Re-weighting Adapter (DRA). Note that the DRA can pay close attention to a small region of the sentences at each step and re-weigh the vitally important words for better aspect-aware sentiment understanding. Finally, experimental results on three benchmark datasets demonstrate the effectiveness and the rationality of our proposed model and provide good interpretable insights for future semantic modeling.

## introduction aspect-based sentiment analysis is a branch of sentiment analysis, which aims to identify sentiment polarity of the specific aspect in a sentence @xcite . for example, given a sentence "the restaurant has attentive service, but the food is terrible.", the task aims to predict the sentiment polarities towards "service" and "food", which should be positive and negative respectively. as a fundamental technology, the absa task has broad applications, such as recommender system @xcite and question answering @xcite . therefore, a great amount of research has been attracted from both academia and industry. among them, deep neural networks (dnn) @xcite @xcite , attention mechanism @xcite and graph neural/attention networks @xcite @xcite have significantly improved the performance through deep feature alignment between the aspect representations and context representations. recently, the large-scaled pre-trained language models, such as bidirectional encoder representations from transformers (bert) @xcite , realize a breakthrough for improving many language tasks, which further attracts considerable attention to enhance the semantic representations. in absa, @xcite designed bert-pt, which explores a novel post-training approach on the bert model. @xcite further proposed a text pair classification model bert-spc, which prepares the input sequence by appending the aspects into the contextual sentence. although great success has been achieved by the above studies, some critical problems remain when directly applying attention mechanisms or fine-tuning the pre-trained bert in the task of absa. specifically, most of the existing approaches select all the important words from a contextual sentence at one time. however, according to neuroscience studies, the essential words during semantic comprehension are dynamically changing with the reading process and should be repeatedly considered @xcite @xcite . for example, when judging the sentiment polarity of the aspect "system memory" in a review sentence "it could be a perfect laptop if it would have faster system memory and its radeon would have ddr5 instead of ddr3", the important words should change from general sentiment words {"faster", "perfect", "laptop"} into aspectaware words {"would have", "faster", "could", "be", "perfect"}. through these dynamic changes, the sentiment polarity will change from positive to the ground truth sentiment label negative. meanwhile, simply initializing the encoder with a pre-trained bert does not effectively boost the performance in absa as we expected @xcite @xcite . one possible reason could be that training on two specific tasks, i.e., next sentence prediction and masked lm, with rich resources leads to better semantic of the overall sentences. however, the absa task is conditional, which means the model needs to understand the regional semantics of sentences by fully considering the given aspect. for instance, bert tends to understand the global sentiment of the above sentence "it could be a perfect laptop ... of ddr3" regardless of which aspect is given. but in absa, the sentence is more likely to be different sentiment meanings for different aspects (e.g., negative for "system memory" while positive for "ddr5"). therefore, the vanilla bert is hardly to pay closer attention to relevant information for the specific aspect, especially when there are multiple aspects in one sentence. to equip the pre-trained models with the ability to capture the aspect-aware dynamic semantics, we present a dynamic re-weighting bert (dr-bert) model, which considers the aspect-aware dynamic semantics in a pre-trained learning framework. specifically, we first take the stack-bert layers as primary sentence encoder to learn overall semantics of the whole sentences. then, we devise a dynamic re-weighting adapter (dra), which aims to pay most careful attention to a small region of the contextual sentence and dynamically select and re-weight one critical word at each step for better aspect-aware sentiment understanding. finally, to overcome the limitation of vanilla bert mentioned above, we incorporate the light-weighted dra into each bert encoder layer and fine-tune it to adapt to the absa task. we conduct extensive experiments on three widely-used datasets where the results demonstrate the effectiveness, rationality and interpretability of the proposed model. ## dynamic re-weighting bert in this section, we introduce the technical detail of dr-bert. specifically, we start with the problem definition, followed by an overall architecture of dr-bert as illustrated in figure 1 . problem definition in absa, a sentence-aspect pair (s, a) is given. in this paper, the sentence is represented as @xmath0, w s 2 , ..., w s ls } which consists of a series of l s words. the specific aspect is denoted as @xmath1, w a 2 , ..., w a la } which is a part of s. l a is the length of aspect words. the goal of absa is to learn a sentiment classifier that can precisely predict the sentiment polarity of sentence s for specific aspect a. as the aspect-related information plays a key role in the prediction @xcite , this paper aims to dynamically select and encode the aspect-aware semantic information through the proposed model. overall architecture dr-bert mainly contains two components (i.e., bert encoder and dynamic re-weighting adapter), together with two modules (i.e., the embedding module and sentiment prediction module). the technical details of each part will be elaborated on as follows. ## computation time comparison we also compared the computation runtime of three baseline methods. all of the models are performed on a linux server with 64 intel(r) cpus and 4 tesla v100 32gb gpus. from the results shown in ## conclusion and future works this paper introduced a new approach named dynamic re-weighting bert (dr-bert) for aspectbased sentiment analysis. specifically, we first employed the bert layers as a base encoder to learn the overall semantic features of the whole sentence. then, inspired by human semantic comprehension, we devised a new dynamic re-weighting adapter (dra) to enhance aspect-aware semantic features in the sentiment learning process. in addition, we inserted the dra into the bert layers to address the limitations of the vanilla pre-trained model in absa task. extensive experiments on three benchmark datasets demonstrated the effectiveness and interpretability of the proposed model, with good semantic comprehension insights for future nature language modeling. moreover, the error analysis was performed on incorrectly predicted examples, leading to some insights into the absa task. we hope our research can help boost excellent work for aspect-based sentiment analysis from different perspectives. in the future, we plan to extend our method to other tasks like sentence semantic matching, relation extraction, etc., which can also benefit from utilizing the dynamic semantics. besides, we will explore whether dr-bert can make any positive changes based on previous mistakes during the dynamic semantic understanding.