Boosting Large Language Models with Continual Learning for Aspect-based Sentiment Analysis (2024)

Abstract: Aspect-based sentiment analysis (ABSA) is an important subtask of sentiment analysis, which aims to extract the aspects and predict their sentiments. Most existing studies focus on improving the performance of the target domain by fine-tuning domain-specific models (trained on source domains) based on the target domain dataset. Few works propose continual learning tasks for ABSA, which aim to learn the target domain’s ability while maintaining the history domains’ abilities. In this paper, we propose a Large Language Model-based Continual Learning (LLM-CL) model for ABSA. First, we design a domain knowledge decoupling module to learn a domain-invariant adapter and separate domain-variant adapters dependently with an orthogonal constraint. Then, we introduce a domain knowledge warmup strategy to align the representation between domain-invariant and domain-variant knowledge. In the test phase, we index the corresponding domain-variant knowledge via domain positioning to not require each sample’s domain ID. Extensive experiments over 19 datasets indicate that our LLM-CL model obtains new state-of-the-art performance.

## introduction aspect-based sentiment analysis (absa) @xcite @xcite plays an important role in the field of natural language processing. this task can be divided into two sub-tasks: aspect extract (ae), which aims to identify the aspects in the sentence and aspectbased sentiment classification (absc) @xcite , which aims to infer the polarities of the corresponding aspects. for example, in the review "the service is bad but the food is delicious!", the user expresses negative and positive sentiments for aspects "service" and "food" respectively. there are still several challenges to continual learning for absa. first (c1), this task requires rich commonsense knowledge to infer the sentiment. for example, the word "hot" expresses a negative sentiment polarity for the aspect "cpu" in the laptop domain and has a positive sentiment for the aspect "pizza" in the restaurant domain (see figure 1 ). second (c2), the sentiment knowledge is inconsistent among different domains. the knowledge in each domain can be divided into domain-invariant knowledge (e.g., good, happy) and domain-variant knowledge (e.g., long, hot, fast). for instance, the general sentiment words are domain-invariant knowledge, which does not change among various domains. to address these problems, we propose large language model-based continual learning (llm-cl) for absa. particularly, for c1, we integrate llms to utilize the large-scale commonsense knowledge in the model. existing work has proved that llms can serve as a knowledge base @xcite . then, for c2, we individually consider the domain-invariant and domainvariant knowledge via a domain knowledge decoupling module with an orthogonal constraint. all the domains learn separate adapters for different domains with a shared adapter. also, we propose a domain knowledge warmup mechanism to align the domain-invariant and -variant representation using replay data. in the test phase, we design a domain positioning strategy to index the correct domain-variant knowledge without knowing the domain the sample belongs to. in the experiments, we first analyze the catastrophic forgetting problem of llms for absa. although llms can reduce the catastrophic forgetting problem, it is still challenging for llms. comparing our llm-cl model on absc, ae, and joint tasks with several strong baselines, our model obtains new state-of-the-art performance on 19 datasets. the ablation studies show the effectiveness of the main components consisting of our llm-cl model. the key contributions are summarized as follows: ## our method in this paper, we propose an llms-based cl framework for absa, which consists of domain knowledge decoupling, domain knowledge warmup and domain positioning (figure 2 ). our framework is based on an llms-based absa model, which trains a generative model using instruction tuning. we first introduce a domain knowledge decoupling module to learn a domaininvariant adapter with individual domain-variant adapters for each domain. then, we align the domain-invariant and domain-variant representa-tions via a domain knowledge warmup strategy. finally, we utilize a domain positioning mechanism to index the domain-variant adapter without requiring the domain id of each sample in the test stage. formally, given a sequence of domains {d 1 , d 2 , ..., d n }, we aim to sequentially learn a model f to maximize the function f at the domain d i and history domains d 1 , ...