Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue (2023)

Abstract: The focus of this work is to investigate unsupervised approaches to overcome quintessential challenges in designing task-oriented dialog schema: assigning intent labels to each dialog turn (intent clustering) and generating a set of intents based on the intent clustering methods (intent induction). We postulate there are two salient factors for automatic induction of intents: (1) clustering algorithm for intent labeling and (2) user utterance embedding space. We compare existing off-the-shelf clustering models and embeddings based on DSTC11 evaluation. Our extensive experiments demonstrate that the combined selection of utterance embedding and clustering method in the intent induction task should be carefully considered. We also present that pretrained MiniLM with Agglomerative clustering shows significant improvement in NMI, ARI, F1, accuracy and example coverage in intent induction tasks. The source codes are available at https://github.com/Jeiyoon/dstc11-track2.

## introduction why intent induction? we humans are generalists. during a conversation, we listen to the other person's utterance and naturally grasp which intent of the utterance is. with the skyrocketing demand for conversational ai, however, the more user utterances a dialogue system encounters, the more unknown intents it does. predefining user intent is expensive and it is impossible to annotate all the user intents. since provided user utterances are unlabeled, intent induction @xcite @xcite @xcite focuses on discovering user intents from user utterances. however, previous studies did not conduct an in-depth analysis of the application of existing models to intent induction that might cause performance degradation problems @xcite . agent: hello, you are currently speaking with rivetown insurance customer service. my name is julian, how may i be of service to you? customer: the services i have being receiving from your company has been encouraging and my intent is to increase or enroll for more plans with you. agent: whoa, that is such an encouraging word coming from you. so, which of the plans do you wish to register for? customer: i would like to enroll for the life insurance policy. our approach. intuitively, for good intent induction, user utterances must be well represented in the embedding space, and good clustering algorithms must be employed to capture latent barycenters of user intent clusters to handle both predefined and unseen intents well. in this paper, we postulate there are two salient factors for automatic induction of intents: (1) clustering algorithm for intent labeling @xcite @xcite and (2) user utterance embedding space @xcite @xcite @xcite . we analyze how the two key factors affect to user intent clustering and intent induction. our extensive experiments with existing models demonstrate that pretrained minilm with agglomerative clustering shows significant improvement in nmi, ari, f1, accuracy and example coverage. ## conclusion the conclusion of this paper is threefold: