Intent Discovery for Enterprise Virtual Assistants: Applications of Utterance Embedding and Clustering to Intent Mining (2022)

Abstract: A key challenge in the creation and refinement of virtual assistants is the ability to mine unlabeled utterance data to discover common intents. We develop an approach to this problem that combines large-scale pre-training and multi-task learning to derive a semantic embedding that can be leveraged to identify clusters of utterances that correspond to unhandled intents. An utterance encoder is first trained with a language modeling objective and subsequently adapted to predict intent labels from a large collection of cross-domain enterprise virtual assistant data using a multi-task cosine softmax loss. Experimental evaluation shows significant advantages for this multi-step pre-training approach, with large gains in downstream clustering accuracy on new applications compared to standard sentence embedding approaches. The approach has been incorporated into an interactive discovery tool that enables visualization and exploration of intents by system analysts and builders.

## introduction to build an enterprise virtual assistant capable of providing effective interaction with customers, intent detection -automatically detecting the customer's intent based on their input -is an indispensable component. large-scale pre-trained language models have shown promising abilities in few-shot classification, and high accuracy intent detection can now be obtained with limited amounts of labeled data @xcite @xcite . however, there are still situations where no labeled data is available. this situation is commonly encountered when designing a dialogue system for a brand new application. in this case, we would like to identify common intents from any available unlabeled data in the domain, such as call transcripts or chat logs. also, for deployed applications, intent discovery can be * work completed at interactions llc applied to 'no-match' data; that is, utterances that the current system does not handle. at first glance, the problem of intent discovery from unlabeled data appears similar to text clustering, which is well-studied in the literature. one natural approach for clustering is to use transformerbased sentence embedding methods @xcite to represent each utterance as a fixed-length vector, and then apply standard clustering methods such as k-means to extract intent clusters @xcite . however, these models are mostly pre-trained on generic data such as wikipedia or natural language inference (nli) datasets, which are quite different from typical enterprise customer service data. hence there is a domain mismatch between the pre-trained model and the downstream application task. in our experience, enterprise virtual assistant data has several key characteristics. first, utterances are mostly short, containing only a few words. additionally, they contain some level of noise from automatic speech recognition (asr) transcriptions. moreover, the data distribution is affected by the customer service communication channel. for example, there are many calls asking to speak to a live agent in order to bypass the system. finally, the semantics differ at some points from ordinary language, because of business logic and design constraints. for example, the two utterances "my screen is broken" and "power button not responding" may be treated as containing the same semantic intent of "tech-support", while in common datasets (e.g., nli), they would likely be considered different. for these reasons, directly applying a generically pre-trained transformer encoder (such as bert @xcite , or even an adapted model like sentence-bert @xcite ) may not be optimal for the intent discovery problem. however, in our data lake we have accumu-lated large amounts of utterance data from a large number of applications across multiple business verticals, all with intent labels either from production understanding models or human annotators. we, therefore, hypothesized that continuing pretraining on our existing virtual assistant data, to obtain a domain-specific utterance encoder, could be beneficial for downstream intent discovery tasks. we propose a three-step solution (figure 1 ) to the intent discovery problem: 1) generic transformer pre-training, 2) domain-adaptive pretraining, 3) downstream embedding and clustering. this approach is related to the don't-stoppretraining paradigm proposed in @xcite , in which a generic language model (lm) is adapted to the target domain (or task) through domain (or task) adaptive pre-training. however, a key difference is that we leverage supervised data for the domain-adaptive pre-training in the second step; on the contrary, they only leverage unlabeled data for the continued pre-training. recently @xcite proposed convfit where supervised contrastive learning is performed after generic lm pre-training. convfit uses a small amount of labeled data from the same task in the adaptive pre-training step, which can be viewed as supervised task-adaptive pre-training for few-shot learning. in contrast, we use a collection of labeled data across a large number of customer service use cases in the second step, which can be viewed as supervised domain-adaptive pre-training for downstream clustering tasks. notice that no access to the downstream data is needed for our domain-adaptive pre-training step, which makes our model reusable for new and unseen applications and use cases. ## model description we describe here the model formulation for step 2 of our intent discovery pipeline. we collected pre-training data from @xmath0, and for each application m we sample m) } is the intent label for that utterance, the total number of intents c (m) for application m is about 1000, and the number of samples n (m) for application m is around one million. our multi-application intent classifier is a multitask learning model where the utterance encoder is shared across applications but the classifier is built separately for each application. mathematically the model can be formulated by minimizing the following loss function ## experiments 4.1 experimental methodology to monitor the quality of the domain-adaptive pretraining in step 2, we randomly select 4% from the supervised pre-training data as a validation set and predict the intent following the multi-task cosine softmax loss proposed in section 3, step 2. the domain-adapted transformer model achieved an accuracy of 91.5% on this validation set, which demonstrates the consistency of the true intent labels and the high quality of the adapted transformer model. however, the ultimate goal of our model is not to test the intent classification accuracy for our existing applications but to perform intent discovery on a new application. it is not possible to directly apply equation (3) for intent discovery, as we do not have the intent embeddings θ(m) for the new application. a natural approach for this problem is to embed all utterances from a new application t using the pre-trained application-independent encoder h(x (t) i ) and apply a clustering algorithm. then, we simultaneously identify the centroids as the discovered intent embeddings and the cluster indices as intent predictions. this is the step 3 in our intent discovery pipeline. for simplicity and a fair comparison with other methods, we use the k-means algorithm to perform clustering on the new application's data, from equation (4), we can see that we are essentially using an enterprise virtual assistant domain adapted utterance encoder for this clustering problem. we will evaluate the performance of the proposed intent discovery pipeline objectively and subjectively. ## conclusion this paper proposed a practical three-step solution to the challenge of intent discovery for virtual assistants. from our experiments, we found that a domain-adaptive pre-training step is essential for achieving good downstream clustering performance. through supervised pre-training, enterprise virtual assistant data characteristics and associated business logic are all distilled into the resulting utterance encoder. this three-step approach can be viewed as an extension of the don't-stoppretraining paradigm @xcite to the downstream clustering tasks. we also found that state-of-the-art generic sentence encoders may not yield the best sentence representations to specific industrial applications such as enterprise virtual assistants. a consequence of this domainadaptive pre-training is that the resulting sentence encoder is no longer generic, hence cannot necessarily be applied to data beyond the domain of enterprise virtual assistants. however, the same methodology could be applied to any enterprise database, so that people can pre-train domainspecific sentence encoders to match their own needs.