Incremental Intent Detection for Medical Domain with Contrast Replay Networks (2022)

Abstract: Conventional approaches to medical intent detection require fixed pre-defined intent categories. However, due to the incessant emergence of new medical intents in the real world, such requirement is not practical. Considering that it is computationally expensive to store and re-train the whole data every time new data and intents come in, we propose to incrementally learn emerged intents while avoiding catastrophically forgetting old intents. We first formulate incremental learning for medical intent detection. Then, we employ a memory-based method to handle incremental learning. We further propose to enhance the method with contrast replay networks, which use multilevel distillation and contrast objective to address training data imbalance and medical rare words respectively. Experiments show that the proposed method outperforms the state-of-the-art model by 5.7% and 9.1% of accuracy on two benchmarks respectively.

## introduction medical intent detection aims to identify intents of medical queries and classify them into specific categories @xcite @xcite . in medical scenarios, understanding query intent is very important for medical question answer systems @xcite . typically, to perform medical intent detection, existing methods pre-define a class set of fixed medical intent categories in advance, and train the whole collected dataset with the fixed class set. however, novel medical intents incessantly emerge with new data in the real world. when given a query with new intent category that is out of the pre-defined class set, these models can do nothing about it. a straightforward solution is to store and re-train the whole data every time new data and intents come in. however, it is almost infeasible with limited storage budget and computation cost in practice learn intent medical fees learn intent treatment plan learn intent cause analysis data for treatment plan data for medical fees data for cause analysis ( @xcite . consider the above problems, we propose to address this issue in a incremental learning way @xcite , where the number of intent categories is allowed to incrementally increase and the system can incessantly learn emerged novel intents from continually arriving data of new intents, which is illustrated as figure 1 . naturally, one simple incremental method is to directly finetune the model by new intents data. however, this method suffers from the serious catastrophic f orgetting problem @xcite @xcite . after fitting emerged data of new intent, the performance of the model on old classes will inevitably drop a lot. there are some studies that have made effects to overcome the catastrophic forgetting problem. typically, they are divided into parameter-based methods that preserve parameters important to the previous classes when updating @xcite , and memory-based methods that store a few examples for each old class and replay them with arriving data of new classes @xcite @xcite . due to the simplicity and effectiveness, memory-based methods dominated this field. however, when applying memory-based methods to incremental intent detection for medical domain, these methods face two new challengestraining data imbalance and medical rare words. training data imbalance: while the amount of new classes data is often large, only a few examples for old classes are stored in the limited memory. therefore, the model can be drastically altered by the richer new classes data and ignore old classes @xcite @xcite . medical rare words: compared with common domains, the medical domain typically contains many domain-specific rare words foot_0 . these rare words are usually not well learned by the model and bring disturbance to representations of examples, which is adverse to selecting representative examples as memory for old classes replay. considering the above problems, we propose contrastive replay networks to enhance incremental intent detection for medical domain. specifically, to address training data imbalance, we devise multilevel distillation to make the current model mimic the behaviors of the original model. to address medical rare words, we devise contrastive objective to push examples from the same intent close and examples from different intents further apart. experimental results demonstrate that our method outperforms previous state-of-the-art models. ## problem definition medical intent detection (mid) is a classification task. in real medical applications, new intent classes incessantly emerge. therefore, a practical mid system should be able to incrementally learn new query intent classes. we introduce a new problem, incremental intent detection. suppose that there is a class-incremental data stream, denoted as and its own intent class set c (k) . note that any two intent class sets are disjoint 2 , i.e., c (i) ∩ c (j) = ∅(@xmath0. at the k-th step, the mid model optimizes its parameters using the training data x (k) train and the updated model should still perform well on historical classes, i.e., classes from 1 to k -1. thus, for testing at the k-th step, we evaluate the updated model on the testing data of all old classes, i.e., ∪ k @xmath1 ## method in this paper, we propose contrastive reply networks for incremental medical intent detection (crn). crn consists of three components: 1) intent classifier with memory, 2) multilevel distillation and 3) contrastive objective.