Novel Feature Discovery for Task-Oriented Dialog Systems (2023)

Abstract: A novel feature represents a cluster of semantically equivalent novel user requests e.g., requests to play a song on a service or read user’s messages. Detecting and supporting novel features is crucial towards wider adoption of dialog systems by end users. Intuitively, features are represented by a combination of intents, slot types and/or their values. For example, while playing a song is a feature represented by a single intent (PlayMusic) only, playing a song on a service is another feature represented by the combination of PlayMusic intent and ServiceName slot type. Prior work on novelty detection limits the scope of features to those represented by novel single intents, leading to (1) giant clusters spanning several user-perceived fine-grained features belonging to the same intent, (2) incoherent interpretation of clusters from users’ perspective (no direct connection to some user-perceived feature), and (3) missing those features spanning several intents. In this work, we introduce feature discovery as opposed to single intent discovery, which aims at discovering novel features spanning a combination of intents and slots, and present a technique for discovering novel features from user utterances. Experiments on two datasets demonstrate the effectiveness of our approach and consistently show its ability to detect novel features.

## introduction advances in natural language understanding (nlu) have led to accelerated adoption of dialog systems such as apple siri and amazon alexa by end users. standing at the core of a dialog system is an nlu model for parsing and understanding user utterances. two of the key tasks of an nlu model are (1) intent classification, which classifies an utterance into a fixed set of intent labels, and (2) slot labeling, which classifies slot values into a predefined set of slot types @xcite . determining the intent guides the dialog system to perform the proper actions as response to user's utterance. for example, user utterances "play some music" and "play despacito" express the intent playmusic, while "how is the weather?" and "is it raining today" express the intent getweather. intents can be further grouped into domains, for instance, playmusic and ratesong intents belong to music domain. detecting slots and their corresponding values within an utterance gives information about objects upon which the actions should be performed. for example, 'despacito' in "play despacito" is of type songname. a feature represents a user experience with the dialog system, for example, playing a song on a service or reading user's messages. over time, users build up expectations about the features/experiences that the dialog system offers. unsupported features cause friction and degrade user's experience. in terms of nlu, a novel feature could be mapped to a new combination of domain(s), intent(s), slot(s) and/or their values, where each is not necessarily novel. for example, while playmusic intent was seen by the dialog system, the combination of playmusic and servicename is never seen before, causing friction with the nlu model when parsing utterances like "play despacito on spotify". consequently, it becomes crucial to discover such features that are frequently requested by the users but are still unsupported by the nlu model, which we address in this work. the task of novel intent discovery has been intensively studied in prior work, by harnessing unsupervised techniques (e.g., @xcite or semi-supervised methods for incorporating existing knowledge from labeled data (e.g., @xcite @xcite . existing work limits the scope of novel features to those represented by novel single intents. however, this does not cover all types of features that naturally span several domains, intents, slots and their values. consider the following user utterances, "play despacito", "play despacito on spotify" and "play despacito in 30 minutes". while the utterances belong to the same intent playmusic, handling each of them is different. the first requires the dialog system to play a song on the device itself, the second asks for playing the same song on a specific service, namely spotify, while the third requires playing the song after a certain amount of time is elapsed. moreover, each request corresponds to a different user experience and requires a different underlying implementation for responding. assuming playmusic intent is novel, applying standard novel intent discovery will group the three utterances in a single cluster, which creates two issues: (1) having many different user-perceived experiences within the same intent (play song on device, on specific service or after some time), results in a giant cluster that needs manual inspection to be decomposed into smaller sub-groups to make meaningful business decisions, and (2) while the cluster represents a novel intent, it might not correspond to a user perceived experience -what the end users are looking for. on the other hand, assuming playmusic intent is not novel, intent discovery might miss those features at more fine-grained levels. for instance, playing a song on a service might not be detected as novel. additionally, intent discovery cannot handle those features spanning several intents. to allow for general feature discovery, and close the gap between user requests and the underlying models, we define a feature to be any combination of domain(s), intent(s) and slot(s) and/or their value(s) and move towards discovering clusters of utterances with novel feature definitions rather than only focusing on novel single intents. novel intent discovery can be seen as a special case of feature discovery, where new features correspond to new intents unseen before. in this paper, we present dnf (discovery of novel features), a semi-supervised approach for discovering novel features from a given set of user utterances with respect to an underlying nlu model. our method consists of a cascaded system with two steps: feature clustering and novelty detection. first, we employ state-of-the-art language model bert @xcite with multi-stage fine-tuning to produce feature/experience-aware representations of user utterances. then, user utterances are clustered into features. second, we classify each resulting feature cluster as either novel or already supported by the nlu model. the salient contributions of our paper are: experimental results demonstrate the effectiveness of our method across the two datasets. ## related work intent classification and slot labeling are two fundamental tasks in spoken language understanding, dating back to early 90's @xcite . with the rise of task-oriented dialog systems, the two tasks have seen more attention, and progress has been made by applying various deep learning approaches (e.g., @xcite @xcite @xcite . discovering novel domains and intents from a pool of user utterances has been well addressed in earlier works, with fully unsupervised and semisupervised settings. these include clustering user utterances with novel domains or intents individually, using various techniques such as constrained deep adaptive clustering @xcite , deep aligned clustering @xcite , contrastive learning @xcite , capsule network @xcite , open intent extraction @xcite , and others (e.g., @xcite @xcite . alternatively, vedula et al., 2020a explore the joint discovery of domains and intents, using hierarchical linking to form an intentdomain taxonomy. the task is also performed jointly with slot filling in recent works @xcite @xcite @xcite . all of the above works require a small amount of labeled data as prior knowledge to guide the discovery process. the most prominent work for detecting novel intents without any prior knowledge was proposed by @xcite , which employs a pre-trained network for generating sentence embeddings, and k-means for intent clustering. however, due to the limited supervision, the method is shown to perform poorly on novel domains. all of the above methods are limited to a discovery objective at intent level, and thus fail to operate on more fine-grained levels (e.g., slot type and/or value) within the same intent. moreover, they fail to detect features composed of multiple domains, intents and/or slots. ## methodology given a set of user utterances {u 1 , u 2 , ..., u n }, we aim to detect a set of clusters {c 1 , ..., c m } where each c j is a cluster of utterances pertaining to a novel feature and m is the total number of novel features. as depicted in figure 1 , our method consists of two components, feature clustering and novelty detection. first, we assign each utterance u i a feature label and eventually produce a set of feature-labeled utterances (u 1 , f 1 ), ..., (u n , f k ), where f 1 , f 2 , ..., f k is the list of k unique features. this is performed by employing an utterance representation model specifically trained with both feature-labeled and -unlabeled data to project the input utterances into a feature-aware vector space that helps clustering the input utterances into k features. second, a feature novelty detection model is used to classify each of the k feature clusters as either novel or already supported by the dialog system. we measure the novelty of a feature by exploiting signals from the nlu model. with dnf being a semi-supervised technique, we distinguish between two types of training data supervision: feature-labeled and feature-unlabeled utterances: ## experimental setup we evaluate the performance of each component of dnf independently as well as the overall discovery system using two datasets: snips and a large french internal dataset. ## experimental results we evaluate (1) the effect of our fine-tuning strategies to produce feature-aware representation on feature clustering, and (2) our feature novelty detection model. ## conclusion we introduced feature discovery, where, given a set of user utterances and a trained nlu model, we extract clusters of novel features composed of a combination of domains, intents, slots and/or their values. to this end, we presented dnf, a semi-supervised approach for extracting novel user features from a set of raw utterances, utilizing minimal feature knowledge from labeled data combined with feature-unlabeled data. dnf supports several fine-tuning strategies to improve utterance representation and make them separable in the feature space. we evaluated dnf on two datasets and observed significant improvements over baselines, showing the effectiveness of our method. in the future, we plan to explore various fine-tuning strategies for better utterance representations, as well as extending dnf to support different languages. 7 limitations while we empirically showed that our approach performs well for novel feature discovery and generalizes across different datasets, we can identify avenues for improvement in terms of efficiency and model training. with dnf relying on good feature-aware sentence representations, obtaining such representations requires expensive fine-tuning steps. for example, using a single gpu, our cascaded fine-tuning strategy takes on average 7 hours on our internal dataset to reach convergence. moreover, in workflows where nlu model refresh is frequent, the model's intent classification and slot tagging confidence distribution can shift over time. with dnf relying on observing confidence signals from the nlu model to determine feature novelty, a retraining/tuning of the novelty detection component would have to be performed frequently. furthermore, our approach requires a small manually annotated feature-labeled dataset (feature labels in addition to intent and slot labels). these additional annotations require expertise and time, which poses a challenge during the data collection phase.