An Open Intent Discovery Evaluation Framework (2024)

Abstract: In the development of dialog systems the discovery of the set of target intents to identify is a crucial first step that is often overlooked. Most intent detection works assume that a labelled dataset already exists, however creating these datasets is no trivial task and usually requires humans to manually analyse, decide on intent labels and tag accordingly. The field of Open Intent Discovery addresses this problem by automating the process of grouping utterances and providing the user with the discovered intents. Our Open Intent Discovery framework allows for the user to choose from a range of different techniques for each step in the discovery process, including the ability to extend previous works with a human-readable label generation stage. We also provide an analysis of the relationship between dataset features and optimal combination of techniques for each step to help others choose without having to explore every possible combination for their unlabelled data.

## introduction a major first task for a goal-oriented dialogue system is to identify the intent behind the user's utterance using a natural language understanding module. this module is often implemented as a classifier, trained on a set of pre-defined intent labels @xcite @xcite @xcite . discovering these intents in real-world systems can be a laborious and timeconsuming task involving a domain expert exploring the dataset and curating a representative set of labels. this task will also need to be repeated regularly as new intents emerge through time. the field of oid seeks to automatically discover unknown intents in a set of unlabelled/partially labelled utterances without requiring such manual effort. there exists an issue in the current literature in that many works focus only on the development of clustering algorithms to identify utterances of figure 1 : an example of the automated discovery and labelling of intents in an given dataset of unlabelled/partially labelled text utterances. first, the utterances are clustered for similar semantic intent, then human-readable labels are generated for each identified cluster. evaluation methods are also inconsistent across the field. some works report on classification or clustering metrics while others evaluate quality of generated labels, but rarely are these reported for the same datasets. there are also differences in the definition of 'intent' and the features of the datasets used for evaluation. some works consider intents in the abstract such as 'customerservice' or 'baggage' in the air travel domain. other works take a much stricter definition e.g. only an action(verb)-object(noun) pair. some datasets contain a mixture of these intent types. these issues make it difficult to identify a truly state-of-the-art (sota) technique for different domains and features of dataset. we present an oid framework which views oid as a two stage process: semantic clustering, and intent label generation (see figure 1 ). we focus on the generation of high quality labels for an unlabelled/partially labelled dataset, produced by combining a semantic representation method, clustering algorithm, candidate extraction method and a label selection method. we evaluate 20 combinations of representation/clustering/extraction/selection methods on 9 datasets. our key contributions include: (1) we introduce our novel oid framework providing a choice of a number of different techniques at every step in the process. foot_0 (2) we extend previous oid work to include a human-readable intent stage. (3) a rigorous investigation into instantiating choice of representation/clustering model/extraction/selection which reveals the optimal settings for datasets and target intents. ## related work state-of-the-art oid techniques utilise semisupervised learning such as in dsscc (deep semi-supervised contrastive clustering) @xcite and deepaligned @xcite . a portion of intents are known in advance and these are used to aid the clustering stage in discovering both the known intent clusters and estimate a number of new, unknown intents. @xcite take a different approach, by pre-training a representation model with a labelled dataset from the same domain as the target unlabelled dataset and then using unsupervised kmeans clustering on the target dataset to discover intents. there are several works which attempt to solve the problem in an unsupervised fashion. chatterjee and sengupta (2020) adapted the dbscan clustering algorithm @xcite in an attempt to handle discovering new intents in datasets with unbalanced distributions, while others such as @xcite use simple kmeans clustering. @xcite are one of the few oid works which include a label generation stage. each cluster has candidate intent labels extracted using a dependency parser to find action(verb)-object(noun) pairs within the utterances and the most common pair is assigned as an auto-generated, humanreadable label for the cluster. their technique discovered the correct number of clusters for the snips dataset and produced labels which were clearly semantically similar to the ground-truth intents, however no quantitative evaluation was conducted. a more challenging dataset would prove more difficult both to cluster and to evaluate by manual inspection. @xcite looked at intent discovery as a sequence tagging task. a neural model sequence tagger is trained to tag action and object words in text utterances. this technique differs in that it will produce an intent for every text utterance and may produce many distinct pairs that express the same intent. in our concurrent work, we presented experimental results for different combinations of candidate extraction and intent label selection techniques against a large generative plm @xcite . in order to produce fine-grained intents, we also proposed an extension to the action-object extraction method used in @xcite which captures more detail from the utterances by including compound nouns or adjectives that are related to the object, and negations related to the action. @xcite introduced a platform for open intent recognition. they combine the related tasks of open intent detection and discovery to both identify the known intents and discover new ones. the detection module identifies known intent samples and groups unknown samples into a single class of open intent. the discovery module then performs clustering to group the unknown samples and present them as new intents. our framework differs in that we focus only on discovery and not detection. we also include a human-readable label generation stage while textoir provides keywords to represent their discovered intents. these keywords are helpful, however, out of context they would be difficult to fully understand without further analysing the utterances themselves. ## methods many current oid techniques can fit into the same two stage pattern (see figure 2 ). stage 1 consists of semantic clustering and is split into two steps. first, semantic representations are obtained for each utterance, then these are grouped with a clustering algorithm to identify semantically similar intents. stage 2 involves the generation of a natural language label for each cluster. first, candidate labels are extracted or generated for each cluster, finally, a label is chosen from these candidates. at each step in the process there are many different options for a researcher to choose from. at the semantic representation step, choices include using bert, universal sentence encoder or one of many other embedding options. for clustering, one could choose kmeans, dbscan etc. when looking for candidate labels, possibilities include an extraction method, such as the action-object extraction used by @xcite , or a label could be generated by prompting a pre-trained language model (plm) such as chatgpt or t0pp. finally, a label must be chosen from the candidates e.g. by choosing the most frequent candidate or even by prompting a plm, specifying the candidates to choose from. our framework allows for any combination of options to be evaluated. most related works do not progress to stage 2, and simply present the clusters of semantically similar texts as the found intents. using the framework, we are able to extend these with stage 2 techniques allowing us to evaluate the quality of the final natural language labels for clusters found by all oid techniques. for each cluster, we measure both the cosine similarity and the bartscore between the most common ground truth label in the cluster and the generated label. one of the goals of this work is to find common patterns in the configurations for datasets with similar features. it is hoped that this will help others to choose the best configuration for their own datasets rather than having to perform a brute force search, or best guess. the framework implements each step as a python module. each can be run individually provided they are given any input required. when chained together, they execute the entire oid process end-to-end. ## datasets we intentionally select a group of datasets with different features to analyse the correlation between features and optimal configurations. snips @xcite , askubuntucorpus and we-bapplications corpus @xcite all contain the action-object format of intents and are queries/commands in conversational style. db-pedia14 sampled and stackoverflow @xcite are labelled for topic. dbpedia14_sampled dataset intent type number of samples number of intents intent balance average number of words vocabulary size askubuntu action-object small (162) small (5) imbalanced (7.13) short (7.94) small (474) snips action-object large (13784) small (7) slightly imbalanced (1.03) short (9.15) large (13418) webapplications action-object small (89) small (8) imbalanced (23.00) short (8.01) small (300) banking77 mixed large (13083) large (77) imbalanced (3.03) short (11.71) medium (3027) chatbotcorpus mixed small (206) small (2) slightly imbalanced (1.64) short (7.70) small (173) clinc mixed large (22500) large (150) balanced (1.00) short (8.31) medium (6420) personalassistant mixed large (20735) medium (46) imbalanced (247.96) short (6.84) medium (7896) dbpedia14 sampled topic large (14000) medium (14) balanced (1.00) long (46.29) xlarge (75214) stackoverflow topic large (20000) medium (20) balanced (1.00) short (8.32) large (16773) vocabulary size small (<500), medium (>=500, <10,000) large (>=10,000,<50,000), xlarge (>=50,000) ## conclusions and future work we have shown that our framework for oid can produce high quality labels for many datasets of differing intent type. the modular nature of the framework allows for further improvements to be utilised when new techniques are discovered for each step. we have evaluated a number of configurations based on the final generated label quality, including extending previous oid works which originally do not generate a human-readable intent label. we have also presented an initial analysis of the mapping between dataset features and the optimal configuration to use for a new, unlabelled dataset which can help reduce the initial effort required to choose the combination of techniques. in future work, we plan to add our action-object extension technique (proposed in @xcite ) to the framework and update the optimal configuration results. we also hope to curate more intent datasets of varying features in order to develop a model for predicting a 'best guess' configuration, given a new dataset's features, rather than having to try every one in turn.