Towards Open Environment Intent Prediction (2023)

Abstract: Out-of-Domain (OOD) Intent Classification and New Intent Discovering are two basic and critical tasks in the Task-Oriented Dialogue System, which are typically treated two independent tasks. Classification focuses on identifying intents beyond the predefined set of the dialog system, but it will not further differentiate detected OOD intents in fine granularity. Discovering focuses on how to cluster unlabeled samples according to their semantic representation, which relies heavily on prior knowledge and can not provide label information for the formed clusters. To be closer to the real user-facing scenarios, we introduce a task paradigm to extend Classification with Discovering referred as Open Environment Intent Prediction, which is to make a further fine-grained discovery of OOD based on OOD Intent Classification. Using various widely-used generative models as an archetype, we propose a general scheme for Open Environment Intent Prediction. In a nutshell, we first perform intent detection to identify the In-domain (IND) samples and then generate labels for those identified as OOD. With these generated labels, we can discover new general intents and provide label information for them. We develop a suite of benchmarks on the existing intent datasets and present a simple yet effective implementation. Extensive experiments demonstrate that our method establishes substantial improvement compared to the baselines.

## introduction ood intent classification, also known as ood intent detection (oid), and new intent discovery (nid), as two basic tasks of the task-oriented dialogue system, have been two areas of active research. the purpose of ood intent classification @xcite ; zhou discovery gpt / bart / t5 my account looks to be blocks. book-flight generate labels book a flight from... can you get me a table… schedule me a table… my account looks to be blocks… account-lock/… can i speed up your voice… adjust-speech/… when will the covid…. covid-pandemic/… intent discovery on generated labels id ood ood fine-grained discovery et al., 2022a) is to identify utterances with not supported intents to prevent them from being wrongly post-processed. however, in the setting of oid, all ood samples, which contain a lot of valuable corpus with different meaningful intents, are just grouped into one rejected class and are not distinguished in a fine-grained way. at the same time, how to effectively identify intents under the generative paradigm has been underdeveloped. new intent discovery @xcite focuses on how to cluster unlabeled data according to their learned semantic representation. however, existing research on new intent discovery needs strong prior knowledge @xcite to learn the semantic representation that can adapt to subsequent clustering, which also depends on unacceptable assumptions in real scenarios, such as knowing the number of categories of ood intents in advance. in addition, its processing procedure is usually cumbersome with multiple dependent processing stages, which often suffers the dilemma that the knowledge learned in the previous is often forgotten in the follow-up as demonstrated in @xcite and the generated clusters usually lack semantic label information. further, since unlabeled data usually contains a large number of samples with known intents, a closer look at the process of nid will reveal that it pays a lot of costs, but in many cases, it is just gathering a large number of samples with definite intents into clusters but cannot provide labels and not fully commit to discovering new intents. to be closer to the realistic scenarios, we first strengthen a combined generative task paradigm based on the characteristics of the above two tasks-open environment intent prediction, which is to make a further fine-grained discovery of ood based on ood intent classification and not only gives the specific categories of ind samples but also further gives the label information of ood. this paradigm can reduce the "burden" of existing nid tasks by avoiding clustering a large number of known intent samples and focusing on discovering new intents while giving specific label information. compared with oid and nid, our proposed task paradigm is more general and practical, whose whole process is shown in figure 1 . then we offer a general implementation based on the generative models. specifically, with a generative model in hand, we carry out oid according to the learned semantic representation and give the corresponding predefined label for ind. at the same time, labels are generated for the samples identified as ood, which also can help to discover more general intents in fine granularity. for more general and practical, we expect to not rely on any assumptions or prior about ood and directly provide high-quality ood labels, which also becomes more challenging. especially for the label generation, since only ind samples are available in the training set, fine-tuning the model directly (model-tuning) will cause the generated labels to overfit training labels, making it a poor choice for the open environment intent prediction. therefore, we adopt prefix-tuning (li and liang, 2021) to retain the general knowledge learned during pretraining to avoid shifting toward the training labels to generate more diverse labels. on this basis, to discover more general intents, we reformulate the intent discovery as a minimum cost multi-cut problem, which can automatically divide samples belonging to a general intent into a cluster according to the similarity of labels. further, to mitigate the impact of inherent label uncertainty @xcite on the open environment intent prediction, with the help of large pre-trained models such as gpt-3 @xcite or chatgpt 2 , we intro-2 https://openai.com/blog/chatgpt/ duce a simple yet effective method of enriching the expression of intents and generate multiple related labels for each intent in the training set. the contributions can be summarized as follows: firstly, this paper strengthens a combined generative task paradigm, which can not only give the specific category of ind but also give the label information of ood and can further discover more general intents. secondly, this paper offers an effective implementation for such a paradigm without relying on any prior about ood and provides a novel solution for enriching the expression of intents and a general method for intent discovery. thirdly, to evaluate the effectiveness and generality of our method, we establish a suite of benchmarks across widely-used generative models and datasets. the experimental results demonstrate our method not only performs better classification but also makes an effective discovery. to this end, lester et al. (2021) inject tunable prompts into the input layer. li and liang (2021); liu et al. (2022) go a step further and put tunable prompts on each internal layer of the model to achieve better results. within training labels outside of training labels (a) generated words distribution pre-trained lms (e.g., chatgpt ) please generate many labels similar to card-not-working, which is related to following samples. template why cannot i use my card? my card cannot physically work. training examples ptms expand labels -card-not-functioning -card-malfunctioning (b) label extension by ptms ## proposed method a natural solution to solve the open environment intent prediction is to carry out full model tuning, i.e., fine-tune all the parameters of the generative models, by taking generating labels for ind sam-ples as the downstream task. however, model tuning could lead to a certain degree of "degradation" of the vocabulary generated by the fine-tuned generative model, which means that the generated labels overfit the labels in the training set. specifically, as shown in figure 2 (a), almost all the words generated by the fine-tuned model fall in the vocabulary composed of the labels in the training set (solid red line in figure 2(a) ), and few words beyond the vocabulary (dotted red line in figure 2(a) ) can be generated, which will fail to generate correct labels for ood samples. prompt-based prefix tuning to retain the general knowledge (avoid shifting towards training labels) obtained by pre-training in large-scale corpus while adapting the model to the downstream task that generates diverse labels, we achieve it by prompting the model with tunable instructions to retain the main parameters of the model unchanged. specifically, we adopt the prefix-tuning (li and liang, 2021; @xcite training paradigm to prepend continuous tunable tokens p @xmath0, denoting @xmath1, p 2 , ..., p l ] as the whole prefixes in all layers. in addition, to steer generative models to generate labels according to the content of samples, we formulate the input x to model with natural language prompts (such as "it was [mask]", which a crafted template of prompt) into t (x) = {x.it was @xcite .|@xmath2. the optimization objective is formulated as follows: label extension with pre-trained models both the oid and nid tasks face a real dilemma. because of the inherent defects of annotation and diversity of intent expression, only one label given in the dataset usually can not accurately reflect the true intents behind the samples or even is wrong for some samples, which can be called inherent label uncertainty (ilu). ilu not only affects the defi- ## analysis figure 5 : plots show the impact of the length of the prefix. our method (red) is also better than prefixtuning (blue) under various prefix lengths. ## conclusion in this paper, we strengthen a combined generative task paradigm to expand the two basic tasks of the task-oriented dialogue system, which is more general and practical. further, without relying on prior knowledge about ood, we provide an effective and efficient implementation based on the generative model. at the same time, we introduce an effective method of intent expansion to alleviate inherent label uncertainty and provide a method for constructing multi-label intent datasets to inspire further research. extensive experiments across different models and different datasets have verified effectiveness and generality.