Intent Mining from past conversations for Conversational Agent (2020)

Abstract: Conversational systems are of primary interest in the AI community. Organizations are increasingly using chatbot to provide round-the-clock support and to increase customer engagement. Many commercial bot building frameworks follow a standard approach that requires one to build and train an intent model to recognize user input. These frameworks require a collection of user utterances and corresponding intent to train an intent model. Collecting a substantial coverage of training data is a bottleneck in the bot building process. In cases where past conversation data is available, the cost of labeling hundreds of utterances with intent labels is time-consuming and laborious. In this paper, we present an intent discovery framework that can mine a vast amount of conversational logs and to generate labeled data sets for training intent models. We have introduced an extension to the DBSCAN algorithm and presented a density-based clustering algorithm ITER-DBSCAN for unbalanced data clustering. Empirical evaluation on one conversation dataset, six different intent dataset, and one short text clustering dataset show the effectiveness of our hypothesis.

## introduction in the past few years, there is a growing community and business interest in conversational systems (chatbots primarily). a key step towards designing a task-oriented conversational model is to identify and understand the intention of a user utterance. an intent in a conversational model maps semantically similar sentences to a high-level abstraction for a chatbot that can generate a similar response or perform an action. for example, "unable to log-in to the system", "can not log in", "facing issue during sign-in" -though linguistically different, are all interpreted as intents related to login issue. the current crop of bot-building frameworks require annotated data for building an intent model. many commercial chatbot building frameworks such as microsoft azure bot service 2 , ibm watson assistant 3 support intent training in a supervised setting. the developers and domain experts typically consider past interaction logs between human-human or human-computer as a valuable resource and carry out an extensive manual process of intent labeling. the process of intent discovery and training data creation is by large manual and effort-intensive and carried out by domain experts. existing dialog corpora contains pre-defined intent and dialog state defined, and consequently, most of the work @xcite ignores intent discovery during conversation design. previous work @xcite on intent identification focuses on clustering single user query/ question using supervised or unsupervised clustering. but the tasks do not consider conversational data. @xcite discusses the realistic complexity of user intent space in a complex domain such as customer support and health care and use the conversational data for clustering and intent induction. but the previous works on intent discovery use pre-decided number of intents as a parameter ## related work intent discovery and analysis is a fundamental step to build intelligent task-oriented conversational agents. intents are a sequence of words which are mapped to predefined categories to comprehend user request. recent works point to two directions to build quality intent models. re-using available conversation log, to bootstrap intent model building process @xcite @xcite . the other is to allow domain experts to build an intent model by working on the model definition, labeling, and evaluation through user interfaces @xcite . our work is at the intersection of these two approaches, in the sense that we mine candidate clusters in an unsupervised way and then allow domain experts to review and label the clusters (intent discovery). gathering good quality labeled data for any machine learning process is expensive. there have been significant efforts to reduce labeling effort; including work on clustering @xcite @xcite , semi-supervised learning @xcite , active learning @xcite , transfer learning @xcite and also recently proposed data programming frameworks @xcite . semi-supervised, transfer learning and active learning require seed training data for processing. clustering is primarily used to collect the initial seed data. most clustering algorithms fail to discover classes in a highly skewed distribution. we overcome these challenges to obtain labels on noisy data by applying a novel clustering algorithm for seed data collection and subsequently propagating labels to generate high-quality training data. various work focus on using existing chat logs to build intent models. a transfer learning-based system has been proposed @xcite to learn from low resource settings. data programming based @xcite systems provide an interface to write labeling function for labeled data generation. however, one underlying assumption of using these methods is that they all require the intents to be known beforehand. this pre-condition is very difficult to meet in real-world cases. clustering is also an active research area for pattern mining. popular algorithms such as centroid based clustering algorithms @xcite ), density-based algorithm (dbscan @xcite ), @xcite ), are very useful in practical applications. although k-means is very fast and mostly used for clustering, it requires one to define the number of clusters as a parameter to the algorithm. among the existing clustering approaches, a density-based algorithm particularly dbscan (density-based spatial clustering with noise) and its variations, is more efficient for detecting clusters with arbitrary shapes from the noisy dataset where there is no prior knowledge about the number of clusters @xcite . many improved versions of this algorithm are also available (such as ng-dbscan @xcite ) to overcome the scalability issues of densitybased clustering, but they fail to address the ineffectiveness of density-based approaches in sparse data setting. although density-based clustering has limitations, it is a powerful tool for automatic data exploration and pattern mining. a key contribution of our work is to provide a better exploration strategy in unbalanced data settings. we search the feature space for different density clusters by adjusting the density definition of dbscan algorithm over iterations. this allows us to generate cluster with different densities and hence to find intents with low frequencies from the past chat log. clusters are explicitly labeled by the expert to collect training data for the intent model. we apply this methodology in the publicly available intent classification dataset with highly skewed class distribution to understand the effectiveness of our clustering algorithm for intent discovery. ## methodology in this section, we describe the methods used for the intent mining framework. ## data existing task oriented dialog corpora such as multiwoz dataset @xcite @xcite , microsoft dialog dataset @xcite , atis @xcite comprise of dialog intent defined in narrow domain like train, restaurant, car booking. recently, perkins (2019) published a curated complex human-human conversation dataset gathered from twitter airline customer support. the tweets comprise conversations between customer support agents of some airline companies and their customers. the conversations constitutes various topics for intent mining task. we also consider various open-source short text dataset and evaluate the generalization of our algorithm. in , we present the overview of the datasets. ## experiments in this section, we evaluate iter-dbscan on the 8 datasets discussed in section 4. we compare iter-dbscan algorithm with state-of-the-art density-based clustering algorithms such as dbscan and hdbscan. for the conversation dataset, we also evaluate our results with a recently published multi-view clustering, av-kmeans algorithm. we use use sentence embedding method to convert the natural language to numerical feature. ## conclusion and feature work in this work, we presented a framework that can cluster textual data using a state-of-art sentence representation method with our algorithm. that provides better intent discovery for complex conversation datasets and short text datasets. our algorithms are able to identify intents from imbalanced dataset with greater accuracy than previous state of the art algorithms. we also presented a feature extraction method using dialog act classification model to extract a short description from conversations for intent mining task. in future, we would like to extend our work by incorporating various other features from conversations, such as different form of questions asked by the agent to resolve a functional task and other linguistic features, for improved clustering. we would also like to explore the use of the neural network to learn generic conversation representation from chat logs for better feature representation.