Multi-View Zero-Shot Open Intent Induction from Dialogues: Multi Domain Batch and Proxy Gradient Transfer (2023)

Abstract: In Task Oriented Dialogue (TOD) system, detecting and inducing new intents are two main challenges to apply the system in the real world. In this paper, we suggest the semantic multiview model to resolve these two challenges: (1) SBERT for General Embedding (GE), (2) Multi Domain Batch (MDB) for dialogue domain knowledge, and (3) Proxy Gradient Transfer (PGT) for cluster-specialized semantic. MDB feeds diverse dialogue datasets to the model at once to tackle the multi-domain problem by learning the multiple domain knowledge. We introduce a novel method PGT, which employs the Siamese network to fine-tune the model with a clustering method directly. Our model can learn how to cluster dialogue utterances by using PGT. Experimental results demonstrate that our multi-view model with MDB and PGT significantly improves the Open Intent Induction performance compared to baseline systems.

## introduction in developing task oriented dialogue (tod) systems, it is important to understand the user's intent, such as opening a new account in banking and reporting accidents in finance. in this circumstance, tod system is used to capture the intents automatically within some dialogue turns, generating appropriate responses to users. since pretrained language models (plms) have become mainstream among the recent nlp tasks, various kinds of research to apply the plms to a dialogue system have appeared. @xcite , @xcite al. (2020), majumder et al. (2020), zhao et al. (2020)) in tod systems, one of the core tasks is intent detection, which seeks to discern user intentions from their utterances. however, most of the typical intent detection modules depend heavily on the domain to be applied. due to the domain-lopsided characteristic, it is necessary for typical models to collect the data of the domain to be serviced and go through a cumbersome process to train a model for each domain from scratch. besides, no standard conventions exist for defining and labeling the intents within the same domain. for example, banking77, an intent classification dataset, has 77 intents. in contrast, banking from dstc 11 has 29 intents and some of the intents, "askaboutatm-fees" and "getbranchinfo," are not included in banking77. hence, it is tricky to apply a model trained with banking77 directly to banking dataset. for example, banking77, an intent classification dataset, has 77 intents. in contrast, banking from dstc 11 has 29 intents and some of the intents, "askaboutatmfees" and "getbranchinfo," are not included in banking77. hence, it is impossible to directly apply a model trained with banking77 to banking dataset. in addition, customers may not communicate within only predefined intent sets. summing up, intent detection modules should induce intents from utterances beyond each domain dataset. previous researches utilized bert to produce hidden representations and passed them to clustering algorithms in order to discern intents of utterances @xcite . however, utterances in dialogue data are usually shorter and colloquial compared to generic text where the ellipsis and omission are less likely to occur. furthermore, bert representations are insufficient to capture the intent information for dialogue since it is only trained to focus on learning the context of the generic text. additional methods are required to contain the domain-specific information since the meaning of a sentence can also be interpreted in various ways depending on its domain. therefore, some researchers have trained plms with dialogue data @xcite @xcite . however, they mostly have conducted experiments within a few-shot setting where some portion of a target dataset is available, rather than a zero-shot setting where target domain data is not accessible. in accordance with the study by @xcite , they handled such problems in semisupervised learning schemes. @xcite showed that it is beneficial to train plm with external intent detection datasets. they also proposed a contrastive learning method based on k-nearest neighbor algorithm. nevertheless, the performance gap between few-shot and zero-shot settings has been still large. our whole pipeline is the same as the baseline used in dstc 11 task2 1 . details are presented in section 4 task description. following the baseline pipeline, we only feed some utterances with labeled as informintent without any modification. similar utterances should be clustered in the hidden representation space. in this paper, we aim to improve open intent induction which requires inducing intent sets in various domains without additional training on target domain datasets (zero-shot setting). we suggest the semantic multi-view model to resolve aforementioned problem using three methods : (1) sbert for general embedding (ge), (2) multi domain batch (mdb) for dialogue domain knowledge, and (3) proxy gradient transfer (pgt) for clusterspecialized semantics. by mdb and pgt methods, constructed domain-agnostic model is not necessary to update parameters further when applying to a different target domain. mdb makes a model to learn various domain knowledge by composing a batch with six groups of samples, sampling from each of the six different dataset. pgt is a way of fine-tuning with k-means to improve on the clustering capability, especially in inducing intents properly. although k-means is an integer assignment problem that is impossible to differentiate, pgt allows a differentiable comparison of k-means labels and ground truth labels by adopting siamese network learning process. from our experiment, combining the general embedding (ge) module and the mdb module showed an excellent performance. in addition, when applying the pgt method to these two modules for the clustering, the performance was improved as well. finally, we found that spectral clustering with all three modules (ge, mdb, pgt) works better, compared to k-means clustering. the codes are available in the github repository 2 this paper has three contributions: ## background in this section, we will briefly cover plms (2.1) and explore how plms are applied to tod systems (2.2). finally, we will introduce existing clustering methods (2.3). ## related works studies related to ood detection and intent discovery are very relevant to open intent induction task in terms of the fact that they need to handle unseen intents. the researchers of those studies are interested in how to make a good representation and how to connect it well by using a grouping algorithm. @xcite proposed the method which utilized content view and utterance view to yield the cluster assignments. @xcite introduced margin loss on bi-directional long short term memory (bilstm) @xcite for inter-class discrimination and intra-class compactness. they applied local outlier factor (lof) @xcite to detect unknown intents based on the nearestneighbor's density. chatterjee and sengupta (2020) introduced iter-dbscan which extended db-scan with a label propagation technique. however, the methods mentioned above require domain specific knowledge and laborious feature engineering. also, those methods often cannot fully leverage the prior knowledge and cannot accurately capture fine-grained intents. @xcite leveraged plm and limited labeled data for supervised intent classification. they used the sentence similarity and filtered out low confidence assignment which is for refining clusters. deep aligned clustering @xcite pretrained bert model with known intent samples to inject prior knowledge. as a next step, they produced pseudo-labels by the cluster assignment for self-supervised learning. cpft @xcite ) performed a contrastive learning with mlm to learn discriminative ability on utterance semantics. @xcite suggested the supervised contrastive scheme with dialogue domain labeled dataset on the basis of mp-net. after that, they applied k-means clustering with estimated k based on bayesian optimization. @xcite focused on training fine-grained discriminative semantic features to deal with ood intents among in-domain (ind) distribution. they utilize k-nearest neighbors of ind intents sample. the limitation of those studies is that they mainly focused on a few-shot performance where the premise is that few target-related labels are available. @xcite pretrained transformer with dialogue data and introduced a domain adaptive pretraining method which shows good performance in open intent discovery task. since their domain adaptive pretraining method shows promising results, we adopt it with some variations. @xcite embedded rich information for the clustering by exploiting external and internal data. they show comparable results in the setting where targetrelated labels are unprovided. however, there is a still big gap between a few-shot setting and a zero-shot setting. in this paper, we suggest two learning methods to induce intents from dialogues efficiently: 1. for mdb, how to appropriately inject prior external knowledge in plm model to be robust whenever the domain changes and 2. for pgt, how the capability of model can be boosted in terms of clustering performance. using two methods, the performance of banking and finance significantly increased compared to the model exposed in a fewshot setting. there are three given datasets(insurance, banking, finance) from 1k customer support spoken conversations, insurance for development and others for evaluation. each dataset has two divided data, one for training an evaluation classifier and the other for testing the performance of open intent induction. the challenge assumes that utterances have only speaker roles without labels such as dialog act and intents, the number of which ranges from 5 to 50. the task consists of three subtasks: label construction, classifier construction, and inducing intent. label construction subtask clusters the trainutterances in order to make a set of intents while predicting what the number of intents is. classifier construction subtask is for training a classifier with train-utterances and predicted labels which are from label construction subtask. inducing intent subtask predicts the labels for test-utterances based on a trained classifier. ## methods in this section, we cover our methodology in detail. we adopt the baseline model structure of the dstc 11 challenge as it is, but we investigate how the performance can be boosted by further learning. our goal is for the model to automatically induce and generate meaningful intents from a collection of unlabeled utterances. sbert, the baseline model proposed by dstc 11, is used as the backbone. figure 2 : it shows the procedure of mdb to learn how to make appropriate domain-agnostic hidden representations. multi-domain samples from each dataset construct the total batch (hwu64, atis, ..., mcid). next, sbert converts the total batch to hidden representations. finally, the loss is calculated by applying cosine softmax for samples of each dataset. ## experiment 6.1 dataset we separate our datasets into two groups, a train dataset and a test dataset. for mdb and pgt training, we introduce six datasets. train banking77 @xcite ) is banking dialogue divided into out-of-domain (ood) intent and in-domain (ind) intent. we use fine-grained 77 intents as train data. atis @xcite ) is made of airplane travel transcripts. we preprocess them to extract the 17 intent categories. clinc150 @xcite is for evaluating the intent detection system. hwu64 @xcite is for developing a home robot with various scenarios to serve as personal assistance, comprised of multi-domain(21 domains). mcid @xcite ) is involved in covid-19, a chatbot dataset consisting of user and agent. restaurants-8k (coope et al., 2020) is a set of 8,198 utterances gathered from the actual restaurant book-ing system. the details of the training dataset with specific data statistics are in test we test our model on two different domain datasets without further training. finance and banking are the released test datasets of dstc 11. ## results we show the results on the banking and finance dstc 11 datasets in first, an encoder with spectral clustering outperforms one with k-means clustering in most cases by at least around 5% in acc and at most 16% in acc. spectral clustering additionally constructs an affinity matrix using the similarity based on the nearest neighbors and clusters utterances based on k-means in a laplacian embedding space. therefore, the result implies that spectral clustering is advantageous for finance and banking as it clusters data by considering the connectivity between data points rather than compactness around the cluster center. more details are in appendix a and appendix b next, the mdb module significantly plays a significant role in open intent induction task beyond the domain. for instance, on banking dataset, ftfspectral outperforms baseline by around 13% in acc, 5% in nmi, and 7% in f1 score. on finance dataset, it also shows higher performance, around 12% in acc, 7% in nmi, and 11% in f1 score. these results symbolize that mdb method is significantly profitable for learning how to discriminate similar utterances without a domain dependency. additionally, by observing ttf model's performance improvement upon baseline (ttf-kmeans), we can derive the same result. furthermore, comparing ftf and ftt, the pgt module boosts performance by bridging the gap between clustering and our model. on banking dataset, our ftt-spectral improves ftf-spectral upon around 3.5% in acc, 1.7% in nmi, and 3.3% in f1. and, on finance dataset, 1.5% in acc, 0.3% in nmi, and 1% in f1. comparing ftf and ttf, the general representations induced by ge module contribute to the performance. on banking dataset, our ttf-spectral improves ftf-spectral upon around 6% in acc, 3% in nmi, 6% in f1. and, on finance dataset, in sum, simply changing from k-means to spectral clustering improves the performance significantly in the model we submitted. our tttspectral shows remarkable performance in two different domains (e.g., banking, finance) without additional training process. mainly, the mdb method boosts performance by a large margin, and the pgt module raises the performance to a higher level. we empirically recognize that spectral clustering shows better synergy with our models than k-means clustering. furthermore, mdb and pgt methods work well regardless of the domain without additional fine-tuning processes. ## conclusion in tod systems, it is noteworthy to capture users' intents regardless of domain. we figure out that simply adopting plm does not perform well in dialogue corpus. therefore, plm needs more training. we suggest the multi-view model with ge, mdb, and pgt modules. the mdb method with cosine softmax utilizes an existing dataset to bridge between plm and dialogue domain. the novel pgt method, a methodology for fine-tuning based on k-means, is presented to enhance the clustering capability. as a result, the performance increases in finance and banking datasets. according to our clustering experiments, we found that spectral clustering is best fitted for our model, resulting in more significant and higher performance than simple k-means methods.