Novel Slot Detection: A Benchmark for Discovering Unknown Slot Types in the Task-Oriented Dialogue System (2021)

Abstract: Existing slot filling models can only recognize pre-defined in-domain slot types from a limited slot set. In the practical application, a reliable dialogue system should know what it does not know. In this paper, we introduce a new task, Novel Slot Detection (NSD), in the task-oriented dialogue system. NSD aims to discover unknown or out-of-domain slot types to strengthen the capability of a dialogue system based on in-domain training data. Besides, we construct two public NSD datasets, propose several strong NSD baselines, and establish a benchmark for future work. Finally, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future directions.

## introduction slot filling plays a vital role to understand user queries in personal assistants such as amazon alexa, apple siri, google assistant, etc. it aims at identifying a sequence of tokens and extracting semantic constituents from the user queries. given a large scale pre-collected training corpus, existing neural-based models @xcite @xcite @xcite @xcite @xcite have been actively applied to slot filling and achieved promising results. existing slot filling models can only recognize pre-defined entity types from a limited slot set, which is insufficient in the practical application scenario. a reliable slot filling model should not only predict the pre-defined slots but also detect potential unknown slot types to know what it doesn't agent: you don't have a playlist called "is this my world" × √ figure 1 : an example of novel slot detection in the task-oriented dialogue system. without nsd, the dialogue system gives the wrong response since it misunderstands the unknown slot "is this my world" as the indomain playlist type. in contrast, nsd recognizes "is this my world" as ns and the system gives a fallback response. meanwhile, with human-in-the-loop annotation, the system can increase its functions or skills. in this paper, we define novel slot (ns) as new slot types that are not included in the pre-defined slot set. nsd aims to discover potential new or out-of-domain entity types to strengthen the capability of a dialogue system based on in-domain precollected training data. there are two aspects in the previous work related to nsd, out-of-vocabulary (oov) recognition @xcite @xcite @xcite and out-of-domain (ood) intent detection @xcite @xcite . oov means many slot types can have a large number of new slot values while the training set only obtains a tiny part of slot values. oov aims to recognize unseen slot values in training set utterance play is this my world by leo arnaud slot filling labels o b-album i-album i-album i-album o b-artist i-artist novel slot detection labels o ns ns ns ns o b-artist i-artist for pre-defined slot types, using character embedding @xcite , copy mechanism @xcite , few/zero-shot learning @xcite @xcite , transfer learning @xcite and background knowledge @xcite , etc. compared to oov recognition, our proposed novel slot detection task focuses on detecting unknown slot types, not just unseen values. nsd faces the challenges of both oov and no sufficient context semantics (see analysis in section 6.2), greatly increasing the complexity of the task. another line of related work is ood intent detection @xcite @xcite @xcite which aims to know when a query falls outside the range of predefined supported intents. the main difference is that nsd detects unknown slot types in the token level while ood intent detection identifies out-of-domain intent queries. nsd requires a deep understanding of the query context and is prone to label bias of o (see analysis in section 5.3.1), making it challenging to identify unknown slot types in the task-oriented dialog system. in this paper, we first introduce a new and important task, novel slot detection (nsd), in the task-oriented dialogue system (section 2.2). nsd plays a vital role in avoiding performing the wrong action and discovering potential new entity types for the future development of dialogue systems. then, we construct two public nsd datasets, snips-nsd and atis-nsd, based on the original slot filling datasets, snips @xcite and atis @xcite (section 2.2). from the perspective of practical application, we consider three kinds of dataset construction strategies, replace, mask and remove. replace denotes we label the novel slot values with all o in the training set. mask is to label with all o and mask the novel slot values. remove is the most strict strategy where all the queries containing novel slots are removed. we dive into the details of the three different construction strategies in section 3.2 and perform a qualitative analysis in section 5.3.1. be-sides, we propose two kinds of evaluation metrics, span-level f1 and token-level f1 in section 3.4, following the slot filling task. span f1 considers the exact matching of a novel slot span while token f1 focuses on prediction accuracy on each word of a novel slot span. we discuss performance comparison between the two metrics and propose a new metric, restriction-oriented span evaluation (rose), to combine the advantages of both in section 5.3.3. then, we establish a fair benchmark and propose extensive strong baselines for nsd in section 4. finally, we perform exhaustive experiments and qualitative analysis to shed light on the challenges that current approaches faced with nsd in section 5.3 and 6. our contributions are three-fold: (1) we introduce a novel slot detection (nsd) task in the task-oriented dialogue system. nsd helps avoid performing the wrong action and discovering potential new entity types for increasing functions of dialogue systems. (2) we construct two public nsd datasets and establish a benchmark for future work. (3) we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future nsd work. ## dataset since there are not existing nsd datasets, we construct two new datasets based on the two widely used slot filling datasets, snips @xcite and atis @xcite . we first briefly introduce snips and atis, then elaborate on data construction and processing in detail, and display the statistic of our nsd datasets, snips-nsd and atis-nsd. finally, we define two evaluation metrics for the nsd task, span f1 and token f1. ## methodology in this section, we introduce the nsd models proposed in this paper and illustrate the differences between the various parallel approaches during the training and test stage. ## discussion in this section, we empirically divide all the error samples into three categories. each type of problem contains two aspects, corresponding to nsd precision and recall, respectively. we present the relative proportions of several types of errors in ## conclusion in this paper, we defined a new task, novel slot detection(nsd), then provide two public datasets and establish a benchmark for it. further, we analyze the problems of nsd through multi-angle experiments and extract the key challenges of the task. we provide some strong models for these problems and offer possible solutions for future work.