Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems (2022)

Abstract: The rapidly growing market demand for automatic dialogue agents capable of goal-oriented behavior has caused many tech-industry leaders to invest considerable efforts into task-oriented dialog systems. The success of these systems is highly dependent on the accuracy of their intent identification – the process of deducing the goal or meaning of the user’s request and mapping it to one of the known intents for further processing. Gaining insights into unrecognized utterances – user requests the systems fails to attribute to a known intent – is therefore a key process in continuous improvement of goal-oriented dialog systems. We present an end-to-end pipeline for processing unrecognized user utterances, deployed in a real-world, commercial task-oriented dialog system, including a specifically-tailored clustering algorithm, a novel approach to cluster representative extraction, and cluster naming. We evaluated the proposed components, demonstrating their benefits in the analysis of unrecognized user requests.

## introduction the development of task-oriented dialog systems has gained much attention in both the academic and industrial communities over the past decade. compared with open-domain dialog systems aimed at maximizing user engagement @xcite , task-oriented (also referred to as goal-oriented) dialog systems help customers accomplish a task in one or multiple domains @xcite . a typical pipeline system architecture is divided into several components, including a natural language understanding (nlu) module, which is responsible for classifying the first user request into potential intents, performing a decisive step that is required to drive the subsequent conversation with the virtual assistant in the right direction. goal-oriented dialog systems often fail to recognize the intent of natural language requests due figure 1 : natural language understanding (nlu) module. based to the intent classifier's confidence level, first user utterances are 'recognized' and associated with an execution flow, or stored in an unhandled pool. to system errors, incomplete service coverage, or insufficient training @xcite . 1 in practice, these cases are normally identified using intent classifier uncertainty. here, user utterances that are predicted to have a level of confidence below a certain threshold to any of the predefined intents, are identified and reported as unrecognized or unhandled. figure 1 presents the nlu module from a typical taskoriented dialog system: the user utterance is either transformed into an intent with an appropriate flow of subsequent actions, or labeled as unrecognized and stored in the unhandled pool (figure 1 ). unhandled utterances often carry over various aspects of potential importance, including novel examples of existing intents, novel topics that may introduce a new intent, or seasonal topical peaks that should be monitored but not necessarily modeled within the system. in large deployments, the number of unhandled utterances can reach tens of thousands on a daily basis. despite their evident importance for continuous bot improvement, tools for gaining effective insights into unhandled utterances have not been developed sufficiently, leaving a vast body of knowledge, as well as a range of potentially actionable items, unexploited. gaining insights into the topical distribution of unrecognized requests can be achieved using unsupervised text analysis tools, such as clustering or topic modeling. indeed, identifying clusters of semantically similar utterances can surface topics of interest to a conversation analyst. we show that traditional clustering algorithms result in sub-optimal performance due to the unique traits of unhandled utterances in dialog systems: an unknown number of expected clusters and a very long tail of outliers. consequently, we propose and evaluate a radiusbased variant of the k-means clustering algorithm @xcite , that does not require a fixed number of clusters and tolerates outliers gracefully. we show that it outperforms its out-of-the-box counterparts on a range of real-world customer, as well as public datasets. the algorithm has recently been evaluated on the task of intent discovery in the context of large-scale, production chatbot, being ranked first (out of 4) at coverage metrics, and second at utterance partitioning @xcite . we propose an end-to-end pipeline for surfacing topical clusters in unhandled user requests, including utterance cleanup, a designated clustering procedure and its extensive evaluation, a novel approach to cluster representatives extraction, and cluster naming. we approach this task in a realworld setting of commercial task-oriented dialog systems, and demonstrate the benefits of the suggested approach on multiple publicly available, as well as proprietary, datasets. ## clustering of unrecognized requests consider a virtual assistant aimed to attend to public questions about covid-19. the rapidly evolving situation with the pandemic means that novel requests are likely to be introduced to the bot on a daily basis. as such, changes in international travel regulations would entail requests related to pcr test availability, and the decision to offer booster shots for seniors might cause a spike in questions about vaccine appointments for elderly citizens. monitoring and prompt detection of these topics are fundamental for continuous bot improvement. ## selecting cluster representatives contemporary large-scale deployments of virtual assistants must cope with increasingly high volumes of incoming user requests. a typical large task-oriented system can accept over 100k requests (i.e., user utterances) per day, where the amount of conversations that pass the initial step of intent identification varies between 40% and 80%. consequently, tens of thousands of requests can be identified as unrecognized on a daily basis. clustering these utterances would result in large clusters that are often impractical for manual processing. providing conversation analysts with a limited set of cluster representatives is a fundamental step toward extracting value from the unrecognized data. ## cluster naming assigning cluster with names, or labels, is an essential step toward their consumability. common approaches to this task resort to simple but reliable techniques based on word n-gram extraction, such as tf-idf ; many of these techniques made their way into the first large-scale information retrieval (ir) systems @xcite . here, we distinguish between the task of cluster naming (extracting a coherent phrase reliably reflecting a cluster's content) and the task of keyword extraction (providing a sequence of one or more words for a compact representation of a document). common approaches to cluster naming include extracting one of the cluster's members to reflect the cluster's content; extracting such a member can be done by by naively selecting the most frequent member in the cluster or by choosing a member satisfying maximum cosine similarity to the cluster's centroid @xcite . in other cases, a good name may not occur directly as one of the cluster's members, and hence requires different handling. some works were trying to investigate the contribution of external knowledge-bases for cluster naming, by incorporating wikipedia pages' meta-data corresponding to the cluster's content @xcite , while others were trying to generate clusters' queries, as a mixture of cluster-internal and differential labeling @xcite . contemporary large pretrained large language models can also be used for the task of keyword extraction. here we make use of key-bert -an approach based on bert @xcite -for identifying key phrases in a cluster, and evaluate the outcome against tf-idf. cluster labeling with tf-idf we treat all utterances in individual clusters from a set @xmath0, c 2 , ..., c k ) as distinct documents. we first applied lemmatization to these documents using the spacy toolkit @xcite , excluded stopwords, and further ranked all ngram token sequences of length n (for @xmath1, 2, 3)) by their tf-idf score. the ngram with the highest score was selected as the cluster name. cluster labeling with keybert treating each cluster as a document, we first extract document-level representation using a pretrained bert language model. 5 we further extract ngram representations for all unique word ngrams in the document, and compute semantic similarity between each ngram's embedding and that of the document. ngram with the highest cosine similarity to the document is selected as the cluster name. 6 evaluation adhering to the same evaluation paradigm as section 2.2, we use the six intent classification datasets for assessing the quality of cluster naming techniques. a common practice for building an intent training dataset involves assigning each class in the training set with a meaningful name, typically mirroring the semantics of the class. as such, an intent class grouping example requests about covid-19 testing information in @xcite , is named 'testing information'. for each class in the intent training set, we compare the automatically extracted class name to that assigned to the class by the dataset creator, where the similarity is obtained by encoding the two phrases -the original class name and the candidate one -and computing their cosine similarity. ## conclusions and future work analyzing unrecognized user requests is a fundamental step toward improving task-oriented dia-log systems. we present an end-to-end pipeline for clustering, representatives selection, and cluster naming -procedures that facilitate the effective and efficient exploration of utterances unrecognized by the nlu module. we propose a clustering variant of the popular k-means algorithm, and show that outperforms its out-of-the-box alternatives on multiple metrics. we also suggest a novel approach to extracting representative utterances while simultaneously optimizing their centrality and diversity. our future work includes the evaluation of our clustering approach with additional datasets, exploration of additional approaches to representative set selection, and advanced techniques for cluster naming. leveraging clustering results to automatically identify actionable recommendations for conversation analyst is another venue of significant practical importance, we plan to pursue.