CLASSIC : Continual and Contrastive Learning of Aspect Sentiment Classification Tasks (2021)

Abstract: This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks in a particular CL setting called domain incremental learning (DIL). Each task is from a different domain or product. The DIL setting is particularly suited to ASC because in testing the system needs not know the task/domain to which the test data belongs. To our knowledge, this setting has not been studied before for ASC. This paper proposes a novel model called CLASSIC. The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing. Experimental results show the high effectiveness of CLASSIC.

## introduction continual learning (cl) learns a sequence of tasks incrementally. after learning a task, its training data is often discarded @xcite . the cl setting is useful when the data privacy is a concern, i.e., the data owners do not want their data used by others @xcite @xcite . in such cases, if we want to leverage the knowledge learned in the past to improve the new task learning, cl is appropriate as it shares only the learned model, but not the data. in our case, a task is a separate aspect sentiment classification (asc) problem of a product or domain (e.g., camera or phone) @xcite . asc is stated as follows: given an aspect term (e.g., sound quality in a phone review) and a sentence containing the aspect (e.g., "the sound quality is poor"), asc classifies whether the sentence expresses a positive, negative, or neutral opinion about the aspect. there are three cl settings (van de ven and tolias, 2019): class incremental learning (cil), task incremental learning (til), and domain incremental learning (dil). in cil, the tasks contain non-overlapping classes. only one model is built for all classes seen so far. in testing, no task information is provided. this setting is not suitable for asc as asc tasks have the same three classes. til builds one model for each task in a shared network. in testing, the system needs the task (e.g., phone domain) that each test instance (e.g., "the sound quality is great") belongs to and uses only the model for the task to classify the instance. requiring the task information (e.g., phone domain) is a limitation. ideally, the user should not have to provide this information for a test sentence. that is the dil setting, i.e., all tasks sharing the same fixed classes (e.g., positive, negative, and neutral). in testing, no task information is required. this work uses the dil setting to learn a sequence of asc tasks in a neural network. the key objective is to transfer knowledge across tasks to improve classification compared to learning each task separately. an important goal of any cl is to overcome catastrophic forgetting (cf) @xcite , which means that in learning a new task, the system may change the parameters learned for previous tasks and cause their performance to degrade. we solve the cf problem as well; otherwise we cannot achieve improved accuracy. however, sharing the classification head for all tasks in dil makes cross-task interfere/update inevitable. without task information provided in testing makes dil even more challenging. previous research has shown that one of the most effective approaches for asc @xcite is to fine-tune the bert @xcite using the training data. however, our experiments show that this works poorly for dil because the fine-tuned bert on a task captures highly task specific features that are hard to use by other tasks. in this paper, we propose a novel model called classic (continual and contrastive learning for aspect sentiment classification) in the dil setting. instead of fine-tuning bert for each task, which causes serious cf, classic uses the idea of adapter-bert in @xcite to avoid changing bert parameters and yet achieve equally good results as bert fine-tuning. a novel contrative continual learning method is proposed (1) to transfer the shareable knowledge across tasks to improve the accuracy of all tasks, and (2) to distill the knowledge (both shareable and not shareable) from previous tasks to the model of the new task so that the new/last task model can perform all tasks, which eliminates the need for task information (e.g., task id) in testing. existing contrastive learning @xcite ) cannot do these. task masks are also learned and used to protect task-specific knowledge to avoid forgetting (cf). extensive experiments have been conducted to show the effectiveness of classic. in summary, this paper makes the following contributions: (1) it proposes the problem of domain continual learning for asc, which has not been attempted before. (2) it proposes a new model called classic that uses adapters to incorporate the pretrained bert into the asc continual learning, a novel contrastive continual learning method for knowledge transfer and distillation, and task masks to isolate task-specific knowledge to avoid cf. ## related work several researchers have studied lifelong or continual learning for sentiment analysis. early works are done under lifelong learning (ll) @xcite @xcite . two naive bayes (nb) approaches were proposed to improve the new task learning @xcite . @xcite proposed a voting based approach. all these systems work on document sentiment classification (dsc). @xcite used ll for aspect extraction. these works do not use neural networks, and have no cf problem. l2pg @xcite uses a neural network but improves only the new task learning for dsc. @xcite worked on asc, but since they improve only the new task learning, they did not deal with cf. each task uses a separate network. existing cl systems srk @xcite and kan @xcite are for dsc in the til setting, not for asc. b-cl @xcite is the first cl system for asc. it also uses the idea of adapter-bert in @xcite and is based on capsule network. more importantly, b-cl works in the til setting. the proposed clas-sic system is based on contrastive learning and works in the dil setting for asc, which is a more realistic setting for practical applications. general continual learning (cl): cl has been studied extensively in machine learning @xcite . existing work mainly focuses on dealing with cf. there are several main approaches. (1) regularization-based approaches such as those in @xcite add a regularization in the loss to consolidate previous knowledge when learning a new task. these methods are for overcoming cf in the cil or til setting of cl. limited work has been done on knowledge transfer, which is our goal. there is little work in the dil setting except the replay method der++ @xcite , which saves some past data. classic saves no past data. contrastive learning @xcite is the base of our contrastive continual learning method. however, there is a major difference. existing contrastive learning uses various transformations (e.g., rotation and cropping) of the existing data (e.g., images) to generate different views of the data. however, we use the hidden space information from the previous task models to create views for explicit knowledge transfer and distillation. existing contrastive learning cannot do that. ## proposed classic method state-of-the-art asc systems all use bert @xcite or other language models as the base. the proposed technique classic adopts the bert-based asc formulation in @xcite , where the aspect term (e.g., sound quality) and review sentence (e.g., "the sound quality is great") are concatenated via @xcite . the sentiment polarity is predicted on top of the [cls] token. as indicated earlier, although bert can achieve state-of-the-art performance on a single task, its architecture and fine-tuning are unsuitable for cl (see sec. 1) and perform very poorly (sec. 4.4). we found that the bert adapter idea in @xcite is a better fit for cl. ## conclusion this paper studied domain incremental learning (dil) of a sequence of asc tasks without knowing the task ids in testing. our method classic uses adapters to exploit bert and to deal with bert cf in fine-tuning, and the proposed contrastive continual learning to transfer knowledge across tasks and to distill knowledge from previous tasks to the current task so that the last model can be used for all tasks in testing and no task id is needed. our experimental results show that classic outperforms the state-of-the-art baselines. finally, we believe that the idea of classic is also applicable to some other nlp tasks. for example, in named entity extraction, we can build a better model to extract the same types of entities from text of different domains. each domain works on the same task but no data sharing (the data may be from different clients with privacy concerns). since this is an extraction task, the backbone model needs to be switched to an extraction model. @xcite . lifelong learning memory networks for aspect sentiment classification. in ieee international conference on big data. rui xia, jie jiang, and huihui he. 2017. distantly supervised lifelong learning for large-scale social media sentiment analysis. ieee trans. affective computing, 8(4):480-491. hu xu, bing liu, lei shu, and philip s. yu. 2019. bert post-training for review reading comprehension and aspect-based sentiment analysis. in naacl-hlt. hu xu, sihong xie, lei shu, and philip s. yu. 2018. dual attention network for product compatibility and function satisfiability analysis. in aaai. wei xue and tao li. 2018. aspect based sentiment analysis with gated convolutional networks. in acl. guanxiong zeng, yang chen, bo cui, and shan yu. 2019. continuous learning of context-dependent processing in neural networks. nature machine intelligence. han zhang, ian j. goodfellow, dimitris n. metaxas, and augustus odena. 2018. self-attention generative adversarial networks.