A Framework to Generate High-Quality Datapoints for Multiple Novel Intent Detection (2022)

Abstract: Systems like Voice-command based conversational agents are characterized by a pre-defined set of skills or intents to perform user specified tasks. In the course of time, newer intents may emerge requiring retraining. However, the newer intents may not be explicitly announced and need to be inferred dynamically. Thus, there are two important tasks at hand (a). identifying emerging new intents, (b). annotating data of the new intents so that the underlying classifier can be retrained efficiently. The tasks become specially challenging when a large number of new intents emerge simultaneously and there is a limited budget of manual annotation. In this paper, we propose MNID (Multiple Novel Intent Detection) which is a cluster based framework to detect multiple novel intents with budgeted human annotation cost. Empirical results on various benchmark datasets (of different sizes) demonstrate that MNID, by intelligently using the budget for annotation, outperforms the baseline methods in terms of accuracy and F1-score.

## introduction the conversational agents such as amazon alexa, apple siri are characterised by the skill of understanding intents which help them to efficiently handle a user's query. for example, the query 'will it be colder in ohio' requires getting the weather updates for the city 'ohio' and would be associated to the intent getweather. the agents are trained with a pre-defined set of intents such as {getweather, ratebook, bookrestaurant} so as to perform the goal-oriented user tasks. but with the passage of time, a user may be interested in performing newer tasks adding hitherto unknown intents. for example, 'play some music from 1954' would be * equal contribution † work done while the author was a student at iit kharagpur associated to the intent playmusic that may not be a part of the set of pre-defined intents. emergence of novel intent detection has been periodically checked by different models in the last decade. there are works on incremental learning in dynamic environment for evolving new classes @xcite @xcite . there are also several approaches @xcite @xcite @xcite to detect new classes in the form of outlier detection but they do not distinguish among multiple new class labels so are not effective in novel multi-class detection. @xcite ; @xcite detect user intents using zeroshot generalized intent detection framework. however, they assume that the unseen intent class la-bels are already known, while in our case neither the number of unseen intent classes, nor the corresponding class labels are known. the other line of works @xcite supply the system with new intents, albeit with a limited amount of tagged data per class and then have an efficient algorithm to incrementally learn new classes. these models work on the assumption that some instances of these new classes would be provided for model building. however, in a realistic setting, the system may not have any knowledge of the number and types of new intents appearing, it may at most understand that some new out-of-domain samples are generated. so, the problem statement is to probe the incoming data wisely and use minimum human intervention to identify all types of novel intents emerging and intelligently tag a limited set of data covering all discovered intents, which can be be fed into a model for retraining. more concretely the system is at first trained with an initial set of known intents; side-by-side an out-of-distribution (ood) detector classifier is also trained to identify datapoints which do not fit the known intents. when substantial amount of such points are detected, the task is to (a) identify whether the points are originating from introduction of a single novel intent or multiple and (b) choose (a limited number of) samples to annotate so that the classifier can be retrained efficiently. in order to determine the number of novel intents present in the ood data, we undertake a clustering based approach with the idea that each cluster would represent a novel intent. by increasing the cluster number progressively, we can make a highly accurate estimate of the number of novel intents. if sample points of an intent mainly correspond to a well formed cluster, the implication is that without much probing we can shortlist enough training samples (through silver tagging) for that class. on the other hand, if the sample points of an intent tend to intertwine with other intent points in the feature space, these can be considered as uncertain points and require human intervention for tagging (gold tagging). with this intuition in place, we design a mix of silver and gold tagging to produce high-quality training samples which can be used to retrain the classifier. our proposed framework of multiple novel intent detection (mnid) is compared with competitive baselines and evaluated across several standard public datasets in nlu domain where it performs substantially better. we use datasets with different number of intent classes. snips @xcite and atis @xcite are smaller datasets, consisting of less number intent classes -7 and 21, respectively. hwu @xcite , banking @xcite and clinc @xcite consist of large number of intent classes -64, 77 and 150, respectively. the paper is organized as follows. we discuss the problem setting and solution overview in section 2. our algorithmic framework is described in section 3. we present the datasets with experimental statistics and data pre-processing in section 4. in section 5, we discuss the experimental design and baselines. detail evaluation results with different algorithmic variations are in section 6. we conclude with a summary in section 7 1 . ## problem setting and solution overview problem setting: to formally describe the problem setting, let there be a dataset w containing overall n classes. however, the value of n is not the overall mnid framework with different algorithmic modules is shown in fig. 1 . ## mnid: solution detail the proposed framework for multiple novel intent detection (mnid) is explained through algorithm 1. as highlighted in the overview, the algorithm consists of data pre-processing step, followed by three stages, each of them are discussed below. the total budget of (gold) annotation is b. besides the system can undertake unlimited silver annotation. the advantage of silver strategy is that it is free as no human probing is required. however, it is also likely to bring in noise if used indiscriminately. pre-process: ood detection (oodd): for the dataset (d), this module (algorithm 2) takes the initial labelled data (d init ) as input and predicts the out-of-domain (ood) samples on the remaining data, (d -d init ). we call the set of ood samples predicted as os. this is a part of the data preprocessing. stage 1. novel class detection (ncd): in this sub-module (algorithm 3), we aim at finding all the new classes, n new . on the ood samples (os), obtained in the previous sub-module (algorithm 2), we do clustering using k-means. we start the algorithm with @xmath0, n @xmath1. we perform -(i) k-means clustering. (ii) annotate x points from each cluster, add those points to l and identify n ′ new classes. (iii) increase new class count (n new + n ′ ). (iv) double the number of cluster count (we compare n new with k/2). we execute the above steps until cluster count exceeds the new intent count. the algorithm returns current annotations (l), newly discovered class count (n new ) and newly formed clusters (cl). the budget spent in this step is b 1 . stage 2. cluster quality based annotation (cqba): in this step (algorithm 4), we evaluate the quality of each of the clusters obtained by the previous algorithm. we annotate p points from each of these clusters and if all the p points belong to the same class, we term it as a good cluster or else a bad cluster. an example of a bad cluster in banking would be the one containing data points from multiple classes, which may have high similarity, such as: declined_cash_withdrawal and pending_cash_withdrawal. for the bad clusters, we annotate q more points. all these annotated points are then added to the labelled data, l. the budget spent in this step is b 2 . hence the remaining budget b -(b 1 + b 2 ) is used in the next step. ## dataset and pre-processing we perform our experiments on a variety of datasets, which are widely used as benchmarks for natural language understanding tasks. the datasets are snips @xcite , atis @xcite , hwu @xcite , bank-ing @xcite and clinc @xcite . snips (7) and atis (21) are smaller datasets consisting of less number of intents (in bracket) where hwu (64), banking (77) and clinc (150) are larger datasets with many intents. atis is the most imbalanced, skewed dataset. in banking data -several intents are highly similar among themselves. the detailed statistics of these datasets including our experimental framework are shown in in the pre-processing step, we filter the out-ofdomain samples. we consider four algorithms for detecting out-of-domain samples. i) softmax prediction probability (msp) @xcite predicts out-of-domain samples based on a threshold on the softmax prediction scores. ii) deep open classification (doc) @xcite ) method builds a multi-class classifier with one vs rest layer of sigmoids. iii) large margin cosine loss (lmcl) @xcite ) trains a network with margin loss and predictions are then fed into an algorithm called local outlier factor (lof) for outlier detection. iv) few-shot ood (fs-ood) @xcite uses a proto-typical network to detect ood examples and classifying in-domain examples with few-shot examples from the in-domain class. we fine-tune bert embeddings using all these out of domain sample detection algorithms. we use bert-base-uncased for these methods. we set the threshold for msp as 0.5 as in @xcite , @xcite . the results of all these algorithms are shown in ## experimental setup the efficacy of the algorithm needs to be tested on two aspects. (a). the number of unknown intents identified. (b). the accuracy achieved when the data is annotated by our algorithm, mnid. to test the accuracy, we use state-of-the-art several classification algorithms used for intent detection. different neural models: we explore different neural models to evaluate mnid as discussed next: the unsupervised algorithms are: (i) k-means (km) @xcite other than km and ag, all the other unsupervised methods along with some of the semisupervised methods such as dtc and cdac need the information of the ground truth number of clusters for training and we provide them so (it is an extra advantage for them). for semi-supervised methods such as kcl, mcl, dtc and dal, we start with double the number of ground truth clusters and let the method determine the number of clusters. hyper-parameters and settings: for post-processing annotation strategy of mnid, we set the cosine similarity threshold, τ as 0.8 and the confidence threshold, t h as 0.5 6 . for all datasets, 5 code: https://github.com/thuiar/textoir 6 this combination of τ and t h provides the best results among different experimented results. ## experimental results in this section, we discuss the experimental outcomes for mnid and competing baselines. we also show results of different clustering algorithms and variations of distinct components of mnid. ## conclusion we have developed mnid (multiple novel intent detection), an end-to-end framework to identify multiple novel intents within a fixed annotation cost. the algorithm intelligently uses the concept of clusters to first discover the classes and then estimate the nature in which datapoints of a class is distributed, that is, whether the datapoints of a class congregate strongly within themselves and separate from other classes or are entangled with datapoints of other classes. in the two types of situations, we propose two different strategies, silver strategy to take advantage of the clusters so that we can annotate many points without any extra human cost and gold strategy to annotate highly uncertain points. this two-pronged approach helps us to annotate highly precise points automatically while annotating the most uncertain (with respect to the class it belongs) points using human assistance. we have done a very rigorous analysis/experimentation to establish the core idea of our algorithm. we observe that the accuracy of classifiers when fed with the dataset created by mnid can beat the standard best few-shot setting where it is assumed that 'κ' instances of each class are provided and annotated by human whereas in our case we have to first discover the classes and then have to find the instances of each class. one limitation of mnid is that it is not able to detect intents where classes are very similar to each other. for example, the query "can you explain why my payment is still pending?" in banking dataset is from the "pending transfer" category but our system detects as "pending card payment" intent as both intents are quite similar. we shall try to address this issue in future. we have presently worked on a setting where novel intents appear in one step, we would strive to extend this framework to explore the dynamics of periodically evolving intents.