Class Lifelong Learning for Intent Detection via Structure Consolidation Networks (2023)

Abstract: Intent detection, which estimates diverse intents behind user utterances, is an essential component of task-oriented dialogue systems. Previous intent detection models are usually trained offline, which can only handle predefined intent classes. In the real world, new intents may keep challenging deployed models. For example, with the prevalence of the COVID-19 pandemic, users may pose various issues related to the pandemic to conversational systems, which brings many new intents. A general intent detection model should be intelligent enough to continually learn new data and recognize new arriving intent classes. Therefore, this work explores Class Lifelong Learning for Intent Detection (CLL-ID), where the model continually learns new intent classes from new data while avoiding catastrophic performance degradation on old data. To this end, we propose a novel lifelong learning method, called Structure Consolidation Networks (SCN), which consists of structure-based retrospection and contrastive knowledge distillation to handle the problems of expression diversity and class imbalance in the CLL-ID task. In addition to formulating the new task, we construct 3 benchmarks based on 8 intent detection datasets. Experimental results demonstrate the effectiveness of SCN, which significantly outperforms previous lifelong learning methods on the three benchmarks.

## introduction task-oriented dialogue systems provide a natural interface to help users accomplish a wide range of tasks, such as playing music, handling money transfer business, and providing information about the covid-19 pandemic. intent detection is an essential component of task-oriented dialogue systems, which aims to accurately estimate diverse user intents for downstream modules @xcite . for example, given the user utterance "tell me some ways to avoid coronavirus", an intent detection model should classify it into the intent class "how to protect yourself ". existing intent detection models usually perform once-and-for-all training on a fixed dataset and can only handle predefined intent classes. however, this setting may not be practical enough in the real world, as new intent classes continually emerge after the model is deployed. for example, with the prevalence of the covid-19 pandemic, users may pose various issues related to the pandemic to conversational systems, which brings many new intents, such as "how to protect yourself " and "the latest number of infections". a general intent detection model should be able to flexibly and efficiently learn new intents round by round. therefore, this work proposes a realistic and challenging task, class lifelong learning for intent detection (cll-id). this task continually trains an intent detection model using new data to learn new intents. at any time, the updated model should be able to perform accurate classification for all intents observed so far. in the cll-id task, it is often infeasible to retrain the model from scratch with the data of all seen classes due to computational cost and data privacy @xcite . for example, the time to train a model with all data of the clinc benchmark @xcite is approximately 9.8 times longer than the time to train the same model with only new data. in practice, intent: how to protect yourself? user 1: how can i protect against the virus? user 2: do any medications protect against the virus? user 3: will wearing gloves help me avoid covid-19? user 4: what kind of face mask helps? figure 1 : an example of expression diversity. different users have different expressions for the same intent. a plain lifelong learning method is to fine-tune a model pre-trained on old data directly on new data. however, this method usually suffers from catastrophic performance degradation on old data, also known as catastrophic forgetting @xcite . to cope with this issue, current mainstream lifelong learning methods usually maintain a memory to store a small number of representative old data @xcite @xcite . however, when directly applying existing lifelong learning methods to the cll-id task, we find two severe problems: expression diversity and class imbalance. expression diversity: in the intent detection task, there are various expression types for the same intent class, as shown in figure 1 . previous methods usually preserve similar old samples that involve only a few expression types and are inconsistent with the original data distribution. these samples are not conducive to maintaining the performance of the old intent classes. class imbalance: at each step of the lifelong learning process, there is generally a large amount of new data, yet only a small amount of old data is preserved due to the memory capacity limitation, leading to a severe imbalance between the new and old intent classes. in this case, the model will be significantly biased towards learning new data, leading to catastrophic forgetting on old data. to address the above two problems, we propose structure consolidation networks (scn), which contains two core components: (1) to handle the problem of expression diversity, we propose structure-based retrospection, which selects and preserves diverse and informative old data based on the spatial structure of features; (2) to cope with the class imbalance problem, we propose contrastive knowledge distillation, which preserves the knowledge of the model trained at the previous step and improves the generalization between the old and new intent classes through contrastive learning. for the cll-id task, we constructed 3 benchmarks based on 8 widely used intent detection datasets. experimental results show that scn significantly outperforms previous lifelong learning methods. in summary, the contributions of this work are as follows: ## task formulation the traditional intent detection task is usually formulated as a text classification task, which predicts an intent class for each input utterance @xcite . the cll-id task adopts a realistic setting where the intent detection model is continually trained on new data to learn new intents. that is, new data arrives in a stream form, denoted as (d 1 , d 2 , ..., d k ). each data d i has its own training/validation/test set (d train i , d valid i , d test i ), as well as its own label set c i . the label set c i contains one or multiple new classes that do not appear in the previous steps. when new data arrives, the intent detection model is updated using the new training set d train i , and uniformly classifies each sample according to all observed intents (i.e., @xmath0. the updated model should perform well on all seen classes. therefore, in the testing stage of the i-th step, we evaluate the updated model on the test data of all observed classes (i.e., dtest @xmath1. the arrival of new data round by round will constantly change the original data distribution, which makes it increasingly difficult for intent detection models to achieve high performance on old data. we experimentally demonstrate this claim in section 4.4. thus, how to alleviate catastrophic performance degradation on old data is a central research point of the cll-id task. ## method in this work, we propose structure consolidation networks to handle the cll-id task. the overall framework of scn is shown in figure 2 . scn consists of two core components, i.e., structurebased retrospection and contrastive knowledge distillation. structure-based retrospection preserves diverse and informative samples to deal with the problem of expression diversity. contrastive knowledge distillation alleviates the negative effects of class imbalance through knowledge distillation and contrastive learning. ## conclusion in this paper, we introduce class lifelong learning into intent detection and further propose structure consolidation networks to overcome catastrophic forgetting. to cope with expression diversity, we propose structure-based retrospection to select diverse and informative samples. to alleviate the negative effects of class imbalance, we propose contrastive knowledge distillation to preserve the knowledge of the original model. experimental results on three benchmarks demonstrate the effectiveness of our method.