Multi-Stage Coarse-to-Fine Contrastive Learning for Conversation Intent Induction (2023)

Abstract: Intent recognition is critical for task-oriented dialogue systems. However, for emerging domains and new services, it is difficult to accurately identify the key intent of a conversation due to time-consuming data annotation and comparatively poor model transferability. Therefore, the automatic induction of dialogue intention is very important for intelligent dialogue systems. This paper presents our solution to Track 2 of Intent Induction from Conversations for Task-Oriented Dialogue at the Eleventh Dialogue System Technology Challenge (DSTC11). The essence of intention clustering lies in distinguishing the representation of different dialogue utterances. The key to automatic intention induction is that, for any given set of new data, the sentence representation obtained by the model can be well distinguished from different labels. Therefore, we propose a multi-stage coarse-to-fine contrastive learning model training scheme including unsupervised contrastive learning pre-training, supervised contrastive learning pre-training, and fine-tuning with joint contrastive learning and clustering to obtain a better dialogue utterance representation model for the clustering task. In the released DSTC11 Track 2 evaluation results, our proposed system ranked first on both of the two subtasks of this Track.

## introduction the design of dialogue mode is very important for the development of task-oriented dialogue system. it typically consists of a set of intents with corresponding slots for capturing and handling domain-specific dialogue box states. previous work on schema-guided dialogue @xcite @xcite focused on data-efficient joint dialogue state modeling across domains and zero-shot generalization to new apis. however, for new emerging domains and novel services, the identification of key intents of such schema typically requires domain expertise and/or laborious analysis of a large volume of conversation transcripts. as the demand for and adoption of virtual assistants continues to increase, recent work has investigated ways to accelerate pattern design through the automatic induction of intentions @xcite @xcite or the induction of slots and dialogue states @xcite . however, the lack of realistic shared benchmarks with public datasets, metrics, and task definitions has made it difficult to track progress in this area. for this reason, the eleventh dialogue system technology challenge (dstc11) proposed a track of intent induction from conversations for task-oriented dialogue. this track is composed of two subtasks, which are organized as shown in fig. 1 . 1) intent clustering, which needs to cluster the given dialogue statements and evaluate them using standard clustering metrics. 2) open intent induction, in which participants are required to generate a set of intents, each represented by a list of sample utterances. the induced intents and utterances will be evaluated using their performance on a downstream classification task over reference intents. both subtasks aim to discover intents from conversations. this track is very challenging due to the lack of labeled data and the unknown number of conversation intents. this paper presents a system that is evaluated in this track. for these tasks, in order to obtain a better dialogue utterance representation model under the condition that the data is unlabeled, we propose a multi-stage coarse-to-fine contrast learning model training scheme. the backbone of our multi-stage training scheme is the roberta-large @xcite . firstly, pre-training is performed by unsupervised contrastive learning in a large number of consecutive conversations. secondly, the model is fine-tuned using a dataset with labels from the same domain as that of the target data for supervised comparative learning. finally, the model obtained after the above two training steps is further fine-tuned on the target data by joint clustering and contrastive learning to obtain the final model. in addition, for the selection of the number of clustering categories k, we adopted the automatic parameter optimization method based on silhouette coefficients (rousseeuw, 1987) provided in the baseline code. as shown in the released evaluation results, our proposed model ranked first on both subtasks. furthermore, experimental results are analyzed by ablation tests. finally, we draw conclusions and give an overview of our future work. ## related work labeled data for task-oriented dialogue systems is often scarce because of the high cost of data annotation. consequently, learning generic dialogue representations that effectively capture dialogue semantics at various granularities @xcite @xcite lays a good foundation for handling a variety of downstream tasks @xcite . contrastive learning has recently demonstrated promising results in the processing of natural language. among which simcse @xcite and tod-bert @xcite get a very good performance on general texts and dialogues, respectively. dse @xcite set new state-of-the-art results on general dialogues. simcse @xcite uses dropout @xcite to construct positive pairs by passing a sentence through the encoder twice to generate two different embeddings. despite the fact that simcse performs better than ordinary data augmentation that directly manipulates discrete text, it has proven to be a poor performer in the field of dialogue, which is confirmed in the dse @xcite . moreover, tod-bert takes an utterance and the concatenation of all the previous utterances in the dialogue as a positive pair. although showing promise on the same tasks, tod-bert's semantic granularity and data statistics in many other dialogue tasks differ from those evaluated in their paper. dse learns from dialogues by taking consecutive utterances of the same dialogue as positive pairs for contrastive learning, and state-of-the-art results are obtained in several tasks, including intention classification. recently, mtp-clnn @xcite set new state-of-the-art results in the new intent discovery field. however, the supervised pre-training part of it requires the data to have a small number of labels to get better results. according to our experimental verification, the generalization ability of the pretrained model in the first stage is poor for data with no labels at all. in addition, recent research results on short text clustering show that the combined training method based on clustering and contrastive learning, sccl @xcite achieves very good results. however, a better pre-trained model for the clustering task as encoded by sccl can further improve the results of clustering. to this end, the goal of our work in the first two stages is to continue pre-training language models to derive better representations for downstream tasks. ## methodology our proposed multi-stage coarse-to-fine model training scheme consists of three stages: contrastive learning with consecutive utterances (clcu), contrastive learning with the nearest neighbors and the same intent (clnnsi) and contrastive learning with joint clustering (cljc). this is shown in fig. 2 . in the first stage, a pretrained model is obtained by performing unsupervised contrastive learning on a large amount of dialogue data using consecutive discourses of the same dialogue as positive pairs. in the second stage, for labeled data in the same domain, we treat that sample with its neighboring samples or samples with the same intention as a positive pairs, and then fine-tune the model by contrastive learning. the third stage further fine-tunes the model by joint contrastive learning and clustering jointly on the target data. the negative pairs for contrastive learning are collected by small batches of negative sampling in the above three stages of model training. after training, we employ a simple nonparametric clustering algorithm named k-means to obtain clustering results. in this section, we describe our multi-stage coarse-to-fine contrastive learning model training scheme in detail below. ## conclusion in this paper, we present our solution to the challenge of intent induction from conversations for task-oriented dialogue of dstc11. firstly, we chose roberta-large, which was pre-trained on a large number of continuous dialogues. since continuous utterances also contain implicit classification information, they are beneficial for the task of dialogue intention classification. secondly, the use of some labeled datasets in the same domain as the target data makes our model have stronger transfer ability. and we adopt knn to select the positive pair for contrastive learning in order to pull neighboring instances together and push distant ones away in the embedding space to learn compact representations for clustering. lastly, the joint training method of clustering and contrastive learning makes the advantages of clustering and contrastive learning complementary. experimental results demonstrate that our methods can effectively cluster the utterances with intention in the dialogue. our method of competitive performance achieved first place in two subtasks. in the future, we will explore better ways to obtain a better dialogue utterance representation model for the clustering task.