Novel Slot Detection With an Incremental Setting (2023)

Abstract: Current dialogue systems face diverse user requests and rapid change domains, making quickly adapt to scenarios with previous unseen slot types become a major challenge. Recently, researchers have introduced novel slot detection (NSD) to discover potential new types. However, dialogue system with NSD does not bring practical improvements due to the system still cannot handle novel slots in subsequent interactions. In this paper, we define incremental novel slot detection (INSD), which separates the dialogue system to deal with novel types as two major phrases: 1) model discovers unknown slots, 2) training model to possess the capability to handle new classes. We provide an effective model to extract novel slots with set prediction strategy and propose a query-enhanced approach to overcome catastrophic forgetting during the process of INSD. We construct two INSD datasets to evaluate our method and experimental results show that our approach exhibits superior performance.

## introduction slot filling (sf), a crucial component of taskoriented dialogue systems, aims to identify specific spans in user utterances @xcite @xcite @xcite . as dialogue systems are widely used in daily life, traditional slot filling needs to be quickly adapted to scenarios with previous unseen types @xcite @xcite , which is known as a novel slot detection (nsd) task @xcite . figure 1 : an example of how dialogue systems respond to the user request containing an unknown slot 'tooth will out'. without nsd, system misclassifies slot type (the right type is 'movie name') and delivers an incorrect reply, with nsd, system returns an inability to handle the request every time. with insd, system can provide a satisfying response in subsequent interactions. however, prior research in nsd only focuses on identifying new slots @xcite , which is too idealized and does not bring practical improvements to the dialogue system. consider the user's request contained an unknown type of slot 'tooth will out' in figure 1 , the dialogue system with novel slot detection just provides a reply that is not wrong but less than satisfactory, and will deliver an inability to handle it all the time. actually, dialogue systems need a specialized procedure to deal with discovered new slot or entity types. in this paper, we focus on a more practical setting: after the model recognizes new types, the dialogue system will view them as known and can return satisfying replies to user requests concerning these types in subsequent interactions, as shown at the bottom of figure 1 . we start with a collection of annotated examples with some pre-defined types. our goal is to identify examples with novel types (i.e., types not contained in the pre-defined set), and then, the new types are seen as known, and we should train model to have the ability to identify entities of these types, therefore resulting in incremental learning @xcite @xcite . we separate the dialogue system to deal with novel types as two major phrases: 1) model discovers slots with unknown classes, 2) training model to possess the capability to handle new classes. we call this unified process incremental novel slot @xcite . for identifying new slots, we employ the set prediction method to generate a great number of triples (start, end, type), and subdivide the non-predefined entity triple into the non-entity triple and the novel-entity triple to extract novel spans. given that the nsd task lacks a clear objective for fitting due to unseen information of new classes at the training stage, we further provide two strategies to make the model better suit this task, which can improve robustness and encourage model to generate diverse outputs. for the insd task, we construct two datasets based on slot filling dataset @xcite . it is generally challenging to mitigate the catastrophic forgetting @xcite during the process of insd, inspired by previous works @xcite @xcite , we propose a queryinduced strategy to realize explicitly knowledge learning from stored data and implicitly knowledge transferring from old model together. we evaluate the proposed method on two common benchmarks and two datasets we constructed. from the results, our approach exhibits promising results on insd and nsd ( § 5). we also perform ablation studies to assess our method of discovering new classes and solving catastrophic forgetting ( § 6.1). finally, we conduct further experiments to explore the nsd model's ability to identify indomain and novel slots ( § 6.2). to summarize, the contributions of our work are three-fold: the set prediction is an end-to-end method that enables model to directly generate the final expected set, which was proposed in detr @xcite to solve object detection task. researchers have introduced it to natural language processing (nlp). @xcite propose a sequence-to-set network to handle the complexity of nested relationships in named entity recognition task. @xcite obtain multimodal inputs representation by detr's decoder. incremental learning. incremental learning @xcite @xcite has been introduced to simulate the setting of type/class-increasing in real-world scenarios, which is mainly researched in information extraction (ie) @xcite . different from assuming data arrived with definite entity types at each stage @xcite , we define a new task incremental novel slot detection that type is unknown and needs to discover in the new data stream. catastrophic forgetting @xcite ) is a long-standing problem in incremental learning. current approaches to overcome this issue can be roughly classified into two groups. the first is knowledge distillation (kd) @xcite @xcite @xcite , which transfers the knowledge from the previous model into the current model. the second is data retrospection @xcite @xcite , which samples old data and stores them in a memory buffer with a fixed capacity. in this paper, we propose a query-induced strategy to enhance kd and data retrospection. ## experimental setups experimental setting. we conduct experiments on two slot filling datasets: snips @xcite and atis @xcite . snips is annotated with 13,084 sentences and has 39 slot types. atis provides 4,478 training examples with 44 slot types. the statistics of the datasets are shown in for nsd task, we follow the data setup in @xcite . we employ the remove processing strategy that randomly selects several classes as unknown and the sentences containing unknown slots are deleted from the training set. for metrics, we apply the span-level precision (p), recall (r), and f1-scores (f1). for insd task, we construct two multi-stage datasets based on snips and atis. we divide the official training set into k subsets {d 1 , d 2 , ..., d k } and assign slot types {t p1 , t p2 , ..., t pk } for each stage. we process the d i as follows: 1) directly remove the sentences including types in t p• (•>i), 2) label the slot values belonging to t p• (•<i) with 'o', and the corresponding text tokens are replaced with 'mask', 3) d i (@xmath0. we exploit four metrics to assess model performance: micro-averaged f1 and macro-averaged f1 for current known types, and micro-averaged f1 and macro-averaged f1 for all types. implementations. we employ bert @xcite as contextualized encoder to implement our model. for constructed multi-stage insd dataset, the k is set to 5 in snips and 3 in atis. the model is trained for 30 epochs with a learning rate of 1e-5 and a batch size of 8. the query number is selected in {20, 40, 60, 80} and the weight to generate novel slot span is chosen from {1e-1, 1e-3, 1e-5}. we set the capacity of memory as 100. baselines. for insd task, we compare our approach to the knowledge distillation model kl @xcite and the data retrospection model kcn @xcite . for nsd task, we compare the proposed method to the pipeline model gda @xcite with two distance strategies gda-min and gda-different. ## overall results in this section, we present the main results of incremental novel slot detection (insd) framework and novel slot detection (nsd) model. ## conclusion in this paper, we introduce a new task incremental novel slot detection (insd) and establish a benchmark with two datasets and four metrics. moreover, we propose a query-induced strategy to alleviate the problem of catastrophic forgetting during incremental learning and provide an effective model to identify novel slots. our proposed approach achieves state-of-the-art performance in all datasets and scenarios.