Redwood: Using Collision Detection to Grow a Large-Scale Intent Classification Dataset (2022)

Abstract: Dialog systems must be capable of incorporating new skills via updates over time in order to reflect new use cases or deployment scenarios. Similarly, developers of such ML-driven systems need to be able to add new training data to an already-existing dataset to support these new skills. In intent classification systems, problems can arise if training data for a new skill’s intent overlaps semantically with an already-existing intent. We call such cases collisions. This paper introduces the task of intent collision detection between multiple datasets for the purposes of growing a system’s skillset. We introduce several methods for detecting collisions, and evaluate our methods on real datasets that exhibit collisions. To highlight the need for intent collision detection, we show that model performance suffers if new data is added in such a way that does not arbitrate colliding intents. Finally, we use collision detection to construct and benchmark a new dataset, Redwood, which is composed of 451 categories from 13 original intent classification datasets, making it the largest publicly available intent classification benchmark.

## introduction as task-oriented dialog systems like alexa and siri have become more and more pervasive, tools enabling developers to build custom dialog systems have followed suit. such tools-like microsoft's luis foot_0 , twilio's autopilot foot_1 , rasa foot_2 , and google's dialogflow foot_3 -enable engineers and dialog designers to craft dialog systems composed of intents, or core categories of competencies or skills in which the system is knowledgeable and to which the system can respond intelligently. new intents may be added periodically to the dialog system as part of its development and maintenance cycle, or dialog system models may be combined together (e.g., @xcite ). these phenomena may occur especially in realworld deployments, where datasets for dialog models may be developed, grown, and modified by large (and even disparate) teams over the span of a project's lifetime. furthermore, dialog system models and their corresponding training datasets are sometimes offered as-a-service or "off-the-shelf" to dialog system builders who might not be fully familiar with the breadth or scope of the pre-existing dataset or model. if the builder adds a new intent to the dataset that overlaps with an existing intent, then the re-trained model's performance can suffer. as such, there is a need for tools and algorithms to help detect when a new intent overlaps-that is, collides-with an already-existing intent category. in this paper, we introduce the challenge of intent collision detection, and develop several algorithms for determining whether a candidate intent category collides with another intent category. to do so, we curate and release a meta-dataset of 722 intents from 13 existing datasets. this graph-like metadataset consists of annotations indicating tuples of colliding intent pairs (examples of colliding intents can be seen in we also use intent collision detection to build redwood, a new intent classification dataset of 451 intent categories. redwood is built by combining 13 smaller datasets. as a comparison, we also build redwood-naïve, which is constructed by naïvely joining together all 13 datasets without arbitrating colliding intents. we find that classifier performance on redwood-naïve to be substantially worse than redwood, showcasing the negative effect of not addressing intent collisions in data. upon official release, redwood will by far be the largest openly available intent classification dataset in terms of breadth of intent categories. our hope is that the new redwood dataset serves as a showcase for intent collision detec2 related work the collision detection task. we discuss three areas of related work related to our proposed intent collision detection task: generalized zero-shot learning, open set classification, and out-of-domain (or out-of-scope) sample identification. in generalized zero-shot learning (e.g., zhang et al. (2022) ), a model is trained with data from a set of "seen" label classes (e.g., intents) and, during inference, must identify test samples as belonging to either a "seen" label class or an "unseen" class for which the model has limited auxiliary knowledge (e.g., descriptions of unseen classes, but no concrete training examples). both open set classification and out-of-domain sample identification refer to the modeling task of classifying inference samples among label classes seen during training or to identify if the sample belongs to an unknown or undefined label class (e.g., @xcite ; @xcite ). slotfilling models that are trained on b/i/o tags naturally predict the unknown class label as o tags, but for intent classifiers the task is much more challenging since it requires curating viable training data for an out-of-domain category (i.e., it is challenging to know in advance what types of out-of-domain inputs a system might encounter). our proposed task of intent collision detection differs from the aforementioned tasks because "inference" samples need not be considered one at a time, but can instead be grouped together into entire candidate intent categories. this enables considering entirely different modeling tasks like those discussed in section 3.3. nevertheless, both our meta-dataset of intent collisions and redwood allow for the evaluation of both zero-shot and generalized zero-shot learning models, and the redwood intent classification dataset includes a substantial number of out-of-domain samples for evaluating open set classification and out-of-domain sample detection. intent classification corpora. there are several smaller corpora for evaluating intent classification models, some spanning broad domains (e.g., @xcite al. (2019), larson et al. (2019b), li et al. (2021) ) and others focusing fine-grained evaluation of individual domains (e.g., the banking-77 corpus @xcite with respect to the personal banking domain). while most datasets are constructed via crowdsourcing, our new redwood dataset is constructed from both (1) already existing datasets and (2) newly crowdsourced intents. dataset derivation and combination. datasets are sometimes formed from other datasets, either what is the weather? is it supposed to rain tomorrow? how hot is it outside? what's the weather like? do i need an umbrella today? tell me the weather forecast how's the weather look? what's the temperature out? tell me today's weather a b c figure 1: transitive collisions. by deriving a new dataset from an existing one, or by combining datasets together. the former category include translations of dialog datasets (e.g., (upadhyay et al., 2018; xu et al., 2020)) as well as re-formulations of existing datasets into new tasks (e.g., converting a semantic role labeling (srl) dataset to open information extraction (oie) data as done in solawetz and larson (2021) ). dataset combination has been used in other fields beyond dialog systems and conversational ai. for instance, @xcite combined several speech recognition datasets together to form their speechstew dataset. as there are no target labels analogous to intents in automatic speech recognition, the creators of speechstew did not have to consider collisions among intent categories. in this paper, our focus is primarily on dataset combination, but we also derive intent classification data from several turn-based dialog corpora (metalwoz and dstc-8, discussed in section 3.4). ## detecting collisions in this section we discuss our proposed challenge, intent collision detection. we begin with a motivating example showing why detecting collisions is important, as well as a formal problem statement. then, we introduce and evaluate several collision detection baselines on our meta-dataset. ## building the redwood dataset with tools addressing the problem of intent collision detection in hand, we now turn our attention to combining the individual datasets from ## conclusion and future work this paper introduces the task of intent collision detection when constructing or updating an intent classification model's dataset to incorporate additional intents. using 13 individual datasets, we constructed a meta-dataset to track intent collisions between the datasets, and then introduced and evaluated two intent collision detection techniques and found that both perform effectively at the collision detection task. to help measure and address this problem, we constructed redwood, a large-scale intent classification dataset consisting of 451 intents and over 60,000 queries. we used redwood to benchmark several intent classification models on the task of in-scope query prediction and outof-scope detection, the new redwood dataset is the largest publicly available intent classification benchmark, in terms of number of intents, and will be made publicly available. future work will include annotating slots to extend redwood to joint intent classification and slot-filling, and it is likely that new tools will have to be developed for doing so. additionally, using the collision detection methods introduced in this paper, redwood can be periodically updated with new intents whenever other new intent classification datasets are published.