Task-Oriented Clustering for Dialogues (2021)

Abstract: A reliable clustering algorithm for task-oriented dialogues can help developer analysis and define dialogue tasks efficiently. It is challenging to directly apply prior normal text clustering algorithms for task-oriented dialogues, due to the inherent differences between them, such as coreference, omission and diversity expression. In this paper, we propose a Dialogue Task Clustering Network model for task-oriented clustering. The proposed model combines context-aware utterance representations and cross-dialogue utterance cluster representations for task-oriented dialogues clustering. An iterative end-to-end training strategy is utilized for dialogue clustering and representation learning jointly. Experiments on three public datasets show that our model significantly outperform strong baselines in all metrics.

## introduction task-oriented dialogue clustering (todc) aims to group task-oriented dialogues into different clusters according to their underlying tasks. since each cluster includes dialogues for one specific task, it therefore brings convenience for task induction and definition. especially for large unlabeled humanhuman dialogues, todc can be employed to help to induce and define new tasks rapidly which is important for designing of task-oriented dialogue system. most prior studies focus on normal text clustering, and have made significant progress via keywords extracting @xcite , topic model @xcite , deep clustering @xcite @xcite . however, inherent differences between task-oriented dialogues and normal texts make above methods difficult to be applied in clustering of task-oriented 1 https://github.com/ryan-lv/dtcn dialogues directly. the first difficulty is that coreference and information omission occur frequently in dialogues @xcite , which makes it harder to build a good representation for utterances in dialogue than in normal text. the second difficulty is that the task-related slot names and intents are scattered in each utterance implicitly and expressed diversely. in most cases, only slots values are given in dialogues without explicit slot names. only by comparing utterances in different dialogues, we can find task-related implicit information, as shown in fig. 1 . considering these special characteristics in task-oriented dialogues, we emphasize that todc should utilize in-dialogue relations between different utterance to build context-aware representations for each utterance and utilize cross-dialogue similarity between utterances in different dialogues to induce implicit task-related concepts information. to address above problems, we proposed a dialogue task clustering network (dtcn) for todc. the key points of dtcn are two folds. first, we construct in-dialogue utterance adjacency graph for each dialogue, and encode the graph with graph attention networks (gat) to build context-aware representations. second, we cluster all utterances to induce implicit task-related concepts, and then learn utterance cluster representations to utilize this information. further integrating both kinds of representations into dialogue representations. finally, training the model with two stages training strategy, which includes pre-training and joint-training stages, the former pretrains a transformer-based auto-encoder with the proposed gate-based transformer decoder for initial clustering assignments, the latter trains jointly the whole model with a selftraining strategy for optimizing dialogue representations and dialogue cluster assignments iteratively. experimental results on three constructed public dialogue datasets (sgd-s, sgd-m and multiwoz-t) show that our model significantly outperforms the existing strong text clustering algorithms in all metrics on todc. especially, we achieve 19.76% improvement of accuracy on sgd-s dataset compared with the best baseline, which indicates that the proposed dialogue representation method can capture more task-related information. in summary, the contributions of our paper are as follows: clustering data representation and clustering algorithm are the two keys to address clustering problems. previous works @xcite @xcite ) mainly focused on feature transformation or clustering independently. data are usually mapped into a feature space and then directly fed into a clustering algorithm to cluster. in the recent years, owing to the development of deep learning, more and more deep clustering methods @xcite @xcite @xcite were proposed, which can obtain feature representations and cluster assignments simultaneously. graph neural network recently, there has been a surge of interest in graph neural networks (gnns) @xcite approaches for graph representation learning. some gnn variants @xcite are proposed and also applied in dialogue related tasks. chen et al. (2020) proposed graph attention matching network and recurrent graph attention network based on graph attention network to encode utterances, schema graphs and previous dialogue states. ghosal et al. (2019) proposed dialogue graph convolutional network based on graph convolutional network @xcite to model inter and self-party dependency to improve context understanding. ## task formulation given an unlabeled dialogue dataset @xmath0, where n dia denotes the total number of dialogues in dataset and d @xmath1. task-oriented dialogue clustering (todc) aims to group d into k dia clusters according to the underlying tasks. ## the proposed model the proposed dialogue task clustering network (dtcn) is composed of five modules as shown in fig. 2 , and trained with two stages training strategy. in the first stage, we used an autoencoder to learn context-aware utterance representations for initial clustering assignments, in which the utterance encoder (ue) and the structural context encoder (sce) are used as encoder, the utterance decoder (ud) module is used as decoder. in the second stage, introducing two new modules based on the pretrained autoencoder, including the utterance cluster representation learning (ucrl) module for learning utterance cluster representations and the dialogue representation learning (drl) module for learning dialogue representations, and adopting an iterative training strategy for optimizing jointly dialogue clustering assignments and dialogue representations. ## case studies we selected some typical utterance clusters to evaluate the quality of learned context-aware utterance representation, and analysis whether some interpretable task-related concepts can be induced by obtained utterance clusters. one case is shown in the fig. 6 , obviously, these utterances are all about the concept of "user wants to search one-way flight". besides, we also found some special utterance clusters. another case is shown in fig. 7 , there are two segments from two dialogues, to our surprise, all of these utterances are grouped into one cluster. this phenomenon can be explained from two aspects. first, from figure 6 : an example of utterance cluster, these utterances are about the concept of "searchonewayfilght" figure 7 : an example of special utterance cluster, these utterances are all related to slots of "destination, origin and departure_date". top to down, discourse relation information and the involved slots information make the representations of related utterances similar. for example, the both utterances of left segment are involved the same three slots, and associated by requestinform pair relationship, the both representations tend to be similar after incorporating this information. further, from left to right, the cross-dialogue utterances with the similar context information are grouped automatically. these indicate that our model can fully integrate contextual information by constructing adjacency graphs, and can also induce interpretable concepts through clustering utterances. ## conclusion and future work this paper proposes a dialogue task clustering network for dialogue task clustering. the model makes use of both in-dialogue discourse relation information and cross-dialogue utterance similarity relation information to build dialogue representations. and an end-to-end iterative strategy of jointly dialogue representation learning and clustering is used to train the model. experiments on three public datasets show that the proposed model significantly outperforms the existing strong clustering algorithms on dialogue task clustering. in the future, we will further induce more detailed task-related concepts information and explore the inner structure in each dialogue cluster for task induction and definition.