DORIC : Domain Robust Fine-Tuning for Open Intent Clustering through Dependency Parsing (2023)

Abstract: We present our work on Track 2 in the Dialog System Technology Challenges 11 (DSTC11). DSTC11-Track2 aims to provide a benchmark for zero-shot, cross-domain, intent-set induction. In the absence of in-domain training dataset, robust utterance representation that can be used across domains is necessary to induce users’ intentions. To achieve this, we leveraged a multi-domain dialogue dataset to fine-tune the language model and proposed extracting Verb-Object pairs to remove the artifacts of unnecessary information. Furthermore, we devised the method that generates each cluster’s name for the explainability of clustered results. Our approach achieved 3rd place in the precision score and showed superior accuracy and normalized mutual information (NMI) score than the baseline model on various domain datasets.

## introduction understanding the user's intent plays an important role in task-oriented dialogue (tod) systems. traditionally, understanding the user's intent requires supervised training using intent-annotated dialogue datasets @xcite @xcite . however, for new emerging domains and services, defining the intent set is challenging and also requires an expert's knowledge. therefore, finding an automatic method that identifies intents from raw conversational data is desirable to reduce costs. the intent clustering task of the 11th dialog system technology challenge (dstc11) aims to provide a realistic benchmark for the intent induction problem. this track evaluates automatic customer intent induction methods from dialogues between human agents and customers, and the dstc11 challenge participants are required to create a set of intent labels based on the conversations. to provide a realistic setting, the number of intents and domain of the test set are not provided until the development phase ends. in this paper, we introduce an automatic intent induction framework that effectively utilizes a public tod dataset. first, we fine-tuned the language model with multi-domain tod datasets so that it has a domain-robust semantic representation. here, we extract verbs and object from utterance to remove the artifacts of unnecessary information. for the training, we applied supervised contrastive learning (scl), which is known to be stable in language model fine-tuning @xcite . second, to infer a new intent set from the unseen domain dataset, we applied a clustering technique that groups the utterances based on the fine-tuned embedding model's representation. furthermore, we generate a label name for each cluster to obtain an interpretable result. the cluster label generation method could reduce the effort of examining each set manually to understand the clustering results. in the experiment with the test dataset (finance and banking domain), we achieved 3rd place in terms of precision and demonstrated superior accuracy (acc) and a higher normalized mutual information (nmi) score than the baseline. furthermore, the generated clustering labels reasonably explain each cluster. finally, we analyzed our model with comparable options and demonstrated the result on development, test, and tod datasets. we named our framework doric, which means domain robust fine-tuning for open intent clustering through dependency parsing. ## dstc11 intent clustering task in this task, participants are required to assign an intent label to each dialogue turn. a set of dialogues are provided as input, and each turn is pre-labeled with both its speaker role (i.e., agent or customer) and dialogue acts (i.e., informintent or not). one development dataset and two test datasets are provided, and each dataset consists of approximately 1k customer support spoken conversations with manual transcriptions and annotations. the development dataset derives from an insurance-related customer support service, and each conversation has an average of 70 turns. in addition, the development dataset contains ground truth intent annotations that allow participants to test and evaluate the model. the number of intent types and the domains of the test dataset are not revealed until the development phase ends. note that no training dataset is given, as this challenge aims to zero-shot intent induction. ## analysis 6.1 verb-object structure in fine-tuning to examine the effect of extracting verb-object structures from the sentence, we compare our proposed method with methods that use the whole sentence during the fine-tuning stage ( ## conclusion in this paper, we describe our solution for the dstc11 intent induction competition. we leveraged the sbert model to embed sentences and fine-tuned the model using dependency parsing results. additionally, we used supervised contrastive loss during fine-tuning to make the model robust in multiple domains. during the analysis, both dependency parsing and scl helped to make the intent induction model more domain robust. furthermore, our intent label generation with hypernym methods allows us to explain the clustering results. according to the results, our approach achieved 3rd place in terms of the precision score and demonstrated better nmi and accuracy compared to the baseline model.