An Empirical Study of Sentiment-Enhanced Pre-Training for Aspect-Based Sentiment Analysis (2023)

Abstract: Aspect-Based Sentiment Analysis (ABSA) aims to recognize fine-grained opinions and sentiments of users, which is an important problem in sentiment analysis. Recent work has shown that Sentiment-enhanced Pre-Training (SPT) can substantially improve the performance of various ABSA tasks. However, there is currently a lack of comprehensive evaluation and fair comparison of existing SPT approaches. Therefore, this paper performs an empirical study to investigate the effectiveness of different SPT approaches. First, we develop an effective knowledge-mining method and leverage it to build a large-scale knowledge-annotated SPT corpus. Second, we systematically analyze the impact of integrating sentiment knowledge and other linguistic knowledge in pre-training. For each type of sentiment knowledge, we also examine and compare multiple integration methods. Finally, we conduct extensive experiments on a wide range of ABSA tasks to see how much SPT can facilitate the understanding of aspect-level sentiments.

## introduction aspect-based sentiment analysis (absa) is an important problem in sentiment analysis @xcite . its goal is to recognize opinions and sentiments towards specific aspects from usergenerated content @xcite . traditional absa approaches generally develop several separate models @xcite @xcite or a joint model @xcite , establishing interactions between different sentiment elements through specific model structures. in recent years, pre-trained models @xcite @xcite have yielded excellent results in extensive nlp tasks. this inspires many research efforts that leverage pre-training techniques to learn sentiment-aware representations. among them, @xcite reveal that pre-training on the sentiment-dense corpus through masked language modeling alone could result in significant improvements on three downstream absa tasks. further, researchers undertake many explorations on integrating sentiment knowledge (e.g., sentiment words) in the pre-training phase @xcite @xcite @xcite , as sentiment knowledge has been widely demonstrated to be helpful in various absa tasks @xcite @xcite @xcite . despite significant gains in various absa tasks, there has not been a comprehensive evaluation and fair comparison of existing sentiment-enhanced pre-training (spt) approaches. therefore, this paper conducts an empirical study of spt-absa to systematically investigate and analyze the effectiveness of the existing approaches. we mainly concentrate on the following questions: (a) what impact do different types of sentiment knowledge have on downstream absa tasks?; (b) which knowledge integration method is most effective?; and (c) does injecting non-sentiment-specific linguistic knowledge (e.g., part-of-speech tags and syntactic relations) into pre-training have positive impacts? based on the experimental investigation of these questions, we eventually obtain a powerful sentiment-enhanced pre-trained model. we evaluate it on a wide range of absa tasks to see how much spt can facilitate the understanding of aspect-level sentiments. our contributions can be concluded as follows: ## method given a review x of length t , a pre-trained model produces its word-level contextualized representations and review-level representation, which can be generally formulated as follows: general-purpose pre-training @xcite @xcite mostly learns parameters through masked language modeling (mlm). in mlm, a certain proportion of words c in the review is masked, and the masked review x is then input to the pre-trained 4 the trained pipeline we use is en_core_web_sm 3.3.0. existing spt approaches integrate sentiment knowledge in two main ways: (1) knowledgeguided masking prioritizes masking sentiment knowledge in reviews and leverages mlm to increase the model's awareness of sentiment knowledge; (2) knowledge supervision directly converts sentiment knowledge into labels and then predicts them by the word-level representations and the review-level representation. ## conclusion in this paper, we perform an empirical study of sentiment-enhanced pre-training (spt). our study investigates the impacts of integrating sentiment knowledge and other linguistic knowledge in pre-training on aspect-based sentiment analysis (absa). to enable our study, we first develop an effective knowledge-mining approach, leverage it to build a large-scale spt corpus, and then select a range of absa tasks as the benchmark to systematically evaluate a pre-trained model's understanding of aspect-level sentiments. experimental results reveal the following findings: (1) integrating aspect words brings general benefits to downstream tasks; its statistic is presented in dataset for cross-domain absa is provided by @xcite . we derive it from https:// github.com/nustm/bert-uda . its statistic is presented in mams dataset is presented by @xcite . we download it from https://github.com/ siat-nlp/mams-for-absa . its statistic is presented in