Class Incremental Learning for Intent Classification with Limited or No Old Data (2022)

Abstract: In this paper, we explore class-incremental learning for intent classification (IC) in a setting with limited old data available. IC is the task of mapping user utterances to their corresponding intents. Even though class-incremental learning without storing the old data yields high potential of reducing human and computational resources in industry NLP model releases, to the best of our knowledge, it hasn’t been studied for NLP classification tasks in the literature before. In this work, we compare several contemporary class-incremental learning methods, i.e., BERT warm start, L2, Elastic Weight Consolidation, RecAdam and Knowledge Distillation within two realistic class-incremental learning scenarios: one where only the previous model is assumed to be available, but no data corresponding to old classes, and one in which limited unlabeled data for old classes is assumed to be available. Our results indicate that among the investigated continual learning methods, Knowledge Distillation worked best for our class-incremental learning tasks, and adding limited unlabeled data helps the model in both adaptability and stability.

## introduction in real-world scenarios, nlp models are regularly updated to incorporate new functionality that either covers new data distributions or includes new output classes @xcite . we focus on the latter scenario in this work, and we consider a feature expansion use case for a spoken language understanding (slu) task in a voice assistant, such as siri or alexa. in particular, we focus on the task of intent classification (ic), which is a common sub-task in slu that aims to map an input user utterance to an intent supported by the system (for instance, playmusic for play the best songs from madonna). with feature expansion, new intents * work done at amazon alexa ai are to be added to the slu model over time to support new output classes that corresponds to new user-facing features in a voice assistant. a regular feature expansion process in production results in a series of consecutive slu model releases that are trained for the same intent classification task, in which new output classes are being added to the model. having regular model releases, it is wasteful of computing and human resources to start from scratch every time and not re-use the previous release model in one form or another. moreover, as our research community moves towards building nlp systems that are environmentally friendly (e.g., requiring less training time and computational resources), class incremental learning (c-il) becomes an important research direction. hence, the problem of class incremental learning (c-il) has been studied by the community for a while now @xcite ; yet, most of the previous work has focused on computer vision tasks @xcite with limited attention to nlp @xcite . a c-il setup for regular releases of a model offers both challenges and opportunities. on the one hand, updating the previous model with new classes introduces a stability-plasticity problem @xcite , where the new model should both retain the knowledge about the old classes and learn the new classes on the same level of accuracy. on the other hand, re-using the old model to build the next release might help to reduce the reliance on old training data and speed up the training of the next release @xcite . in this work, we compare contemporary solutions for incremental learning on a c-il problem for the ic task following a production-inspired feature expansion cycle. we assume a number of consecutive model releases, where each one adds a number of new output classes and the corresponding training data. the training data for the old classes is either limited or not available at all in line with the privacy requirements of a real-life setup, where there might be restrictions on storing or using older data. we simulate this setup with the large intent classification oos (out-of-scope) dataset @xcite . furthermore, we restrict our experiments to methods that do not increase the capacity of the final model with time. this is an important restriction in a real-world scenario where runtime constraints do not allow to make the model larger. our main contributions are: ## challenges the fundamental obstacles to effective c-il are conceptually simple, but in practice very challenging to overcome. these challenges originate from the sequential training of tasks and the requirement that at any moment the learner must be able to classify all classes from all previously learned tasks. incremental learning methods must balance retaining knowledge from previous tasks while learning new knowledge for the current task. this problem is called the stability-plasticity dilemma @xcite . a naive approach to class incremental learning which focuses solely on learning the new task will suffer from catastrophic forgetting: a drastic drop in the performance on previous tasks. preventing catastrophic forgetting leads to a second important problem of class incremental learning, i.e., the problem of intransigence: the resistance to learn new tasks. class incremental learning methods need to balance between keeping knowledge about old classes and learning new classes at the same level of accuracy. ## method we consider a c-il scenario in which an existing nlp classification model is updated over time. we assume a number of consecutive model updates, and for each update, a certain number of new classes and corresponding training data become available, which need to be supported. since, to the best of our knowledge, c-il has not yet been explored for nlp classification tasks, this study includes a diverse set of techniques, which have shown promising results on other incremental learning tasks. in the following, we first describe our c-il scenario in more detail. subsequently, we discuss the considered basic classification model architecture and the c-il strategies afterwards. ## experimental set up in our experiments, we study two c-il scenarios: without any old data available and assuming that a small amount of unlabeled data for old classes can be accessed. in the following, we first discuss both scenarios in more detail. subsequently, we describe the dataset and class-incremental learning simulation, and finally the experimental settings. ## experimental results recall that the core challenge with class incremental learning methods is to balance retaining knowledge from old exemplars while learning new knowledge for the new exemplars. in our experiments, we investigate how the different incremental learning methods perform on both preventing catastrophic forgetting and intransigence i.e., resistance to learn accuracy 0 25 50 75 100 @xmath0: 51-70 classes, test : 1-50 classes @xmath1: 71-90 classes, test : 1-70 classes @xmath2: 91-110 classes, test : 1-90 classes @xmath3: 111-130 classes, test : 1-110 classes @xmath4: 131-150 classes, test : 1-130 classes bert ewc kd recadam oracle l2 figure 1: average accuracy of the c-il methods on all seen classes so far until the previous incremental phase for the core c-il scenario. test classes accuray 0 25 50 75 100 1-50 classses @xmath5. where appropriate, we also include performance of an "oracle"; contrasting with the c-il techniques, the oracle has access to all of the previously available labeled datasets. it should be seen as an upper bound, indicating what performance would be possible if older labeled data could be kept for model training. ## conclusion in this paper, we explored class-incremental learning for the intent classification task. in particular, we compared several methods, i.e., bert warm start, l2, elastic weight consolidation, recadam and knowledge distillation. we compared performance within two class-incremental learning scenarios: one where only the previous model was assumed to be available, but no data corresponding to old classes, and one in which limited unlabeled data for old classes was assumed to be available. we are the first to benchmark these methods in the challenging incremental learning setup for intent classification motivated by real-life restrictions where no old data might be available. we presented extensive experiments on the out-of-scope dataset for intent classification. among the investigated continual learning methods, knowledge distillation worked best for our class-incremental learning tasks, and adding limited unlabeled data helped the model in both adaptability and stability. we plan to add token-level slot prediction task to our setup in the future and include further mlm models beyond just the bert architecture.