A Two-Stage Progressive Intent Clustering for Task-Oriented Dialogue (2023)

Abstract: Natural Language Understanding (NLU) is one of the most critical components of task-oriented dialogue, and it is often considered as an intent classification task. To achieve outstanding intent identification performance, system designers often need to hire a large number of domain experts to label the data, which is inefficient and costly. To address this problem, researchers’ attention has gradually shifted to automatic intent clustering methods, which employ low-resource unsupervised approaches to solve classification problems. The classical framework for clustering is deep clustering, which uses deep neural networks (DNNs) to jointly optimize non-clustering loss and clustering loss. However, for new conversational domains or services, utterances required to assign intents are scarce and the performance of DNNs is often dependent on large amounts of data. In addition, although re-clustering with k-means algorithm after training the network usually leads to better results, k-means methods often suffer from poor stability. To address these problems, we propose an effective two-stage progressive approach to refine the clustering. Firstly, we pre-train the network with contrastive loss using all conversations data and then optimize the clustering loss and contrastive loss simultaneously. Secondly, we propose adaptive progressive k-means to alleviate the randomness of vanilla k-means, achieving better performance and smaller deviation. Our method ranks second in DSTC11 Track2 Task 1, a benchmark for intent clustering of task-oriented dialogue, demonstrating the superiority and effectiveness of our method.

## introduction task-oriented dialogue technique is of great popularity and is widely adopted in various applications, such as microsoft's xiaoice @xcite , google assistant etc. natural language understanding (nlu) is one of the most crucial components of task-oriented dialogue, and it is often considered as an intent classification task. supervised deep networks are the most widely used methods for intent classification due to their high performance, such as classification head based on pre-trained language models. however, such kind of methods require a large amount of labeled data that is collected through painstaking analysis of conversation transcripts by domain experts. this is not feasible for many commercial operations, especially in emerging domains or services. therefore, researchers have proposed lowresource unsupervised intent clustering methods to address this problem, as described in speaker role utterances dialog acts gold intent agent how can i help you today? customer i want to pay my auto insurance bill. informintent intent1 agent i will need your account number. customer it's two three one five six four eight seven nine. agent okay. thank you. customer so what is my payment? informintent intent2 agent okay. i'm showing that no payment is due until april. to facilitate better data separation, supporting clustering with contrastive learning (sccl) @xcite is the first method to introduce contrastive learning into clustering by jointly optimizing contrastive loss and clustering loss. however, dnns usually rely on large amounts of data to update model parameters or proper initialization to speed up convergence. in addition, the acquisition of intent utterances is limited and difficult, especially in emerging dialogue domains or services, so we improve the clustering performance from the perspective of model initialization. meanwhile, in order to get better cluster assignments, re-run clustering using k-means is generally performed after training the network. it is known that k-means suffer from poor stability due to the way of obtaining initial centroids. to solve these problems, we propose an effective two-stage progressive intent clustering from conversations. our main contributions are as follows: ## proposed approach to refine the performance of automatic intent clustering for task-oriented dialogues, we propose a two-stage progressive approach. our specific method pipeline is shown in figure 1 . ## conclusion in this paper, to refine the performance of automatic intent clustering for task-oriented dialogue, we propose a two-stage progressive approach. to alleviate the lack of target utterance, we introduce pre-training before joint learning to increase the robustness of the model. in addition, we propose progressive k-means, an enhanced k-means method, which is compatible with deep clustering. also, we use the high-score-center obtained by performing progressive k-means in the first stage to initialize the parameters of the clustering head in the second stage. ultimately, our model results in better performance and lower deviation. our proposed system proved its effectiveness by ranking second in acc metrics and first in f1 metrics in dstc11 track2 task 1. in addition, we conducted an ablation study to verify the performance of each component.