Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11 (2023)

Abstract: With increasing demand for and adoption of virtual assistants, recent work has investigated ways to accelerate bot schema design through the automatic induction of intents or the induction of slots and dialogue states. However, a lack of dedicated benchmarks and standardized evaluation has made progress difficult to track and comparisons between systems difficult to make. This challenge track, held as part of the Eleventh Dialog Systems Technology Challenge, introduces a benchmark that aims to evaluate methods for the automatic induction of customer intents in a realistic setting of customer service interactions between human agents and customers. We propose two subtasks for progressively tackling the automatic induction of intents and corresponding evaluation methodologies. We then present three datasets suitable for evaluating the tasks and propose simple baselines. Finally, we summarize the submissions and results of the challenge track, for which we received submissions from 34 teams.

## introduction task-oriented dialogue systems used for handling high-volume customer service requests have seen growing adoption in recent years. while numerous platforms for building such dialogue systems exist, they still typically require significant time and expert knowledge to achieve good results. in particular, the process of creating non-overlapping intents and corresponding example utterances typically requires domain expertise and/or laborious analysis of a large volume of conversation transcripts. intent induction, automatically deriving intents and corresponding utterances from conversation transcripts or customer queries, has the potential to significantly reduce the time and effort required to build a task-oriented dialogue system from the ground up. recent work in intent mining has often cast intent induction as a problem similar to that of short text clustering, in which unlabeled customer queries and requests are assigned labels and the effectiveness of a system is evaluated by comparing cluster assignments to labels from a ground truth intent schema @xcite @xcite . due to the lack of real world datasets capturing human-human customer support conversations, researchers have repurposed existing intent classification datasets or labeled small subsets of publicly available customer service interactions to evaluate their systems. despite a growing body of work, there is a lack of common evaluative settings and standardized, dedicated benchmarks for intent induction, making progress difficult to track. existing intent classification datasets do not capture the complexity of mining intents from real customer service interactions, which typically have highly skewed distributions of intents that are embedded in noisy conversations. while dedicated, representative benchmarks have been instrumental in driving and measuring progress in natural language processing tasks @xcite , there is a clear gap in this regard for intent induction. to encourage further research and to provide a shared benchmark in the realistic setting of spoken customer service interactions, this challenge track introduces a dataset containing conversations spanning three domains: insurance, personal banking, and finance. the track explores an alternative framing of the intent induction task through the use of two related subtasks: task 1) intent clustering and task 2) open intent induction. for the intent clustering task, we use classic clustering metrics for evaluation. for open intent induction, which seeks to provide a more realistic and noisy setting in which intents are embedded in conversations, we evaluate systems by examining the predictions of an intent classifier trained on induced intent schemas. this setting aims to bring us closer to assessing the impact of automatic induced intents on a final dialogue system. a total of 34 teams participated in the track, with 19 teams also participating in task 2. in this paper, we describe the tasks, evaluation methods, datasets, and baselines. we then describe the submissions from the participating teams and summarize the results and findings from the track. ## related work the majority of existing work in mining intents has focused on clustering-based methods. @xcite proposes a multi-view clustering approach for learning clustering representations by predicting cluster assignments of an alternative view of each input, such as prompts. @xcite investigates variants of db-scan and propose an approach that iteratively breaks down the "noise" cluster from dbscan to address varying densities. others have leveraged intermediate structured prediction tasks (such as dependency parsing or abstract meaning representations) to aid in the induction of intents @xcite @xcite . prior work in intent mining has largely evaluated systems by re-purposing existing datasets commonly used for evaluating intent classification systems. such datasets, including bank77 @xcite , clinc150 @xcite , snips @xcite , atis @xcite , or stackoverflow @xcite , do not include full dialogues. task-oriented dialogue datasets like multiwoz @xcite , multidogo @xcite and sgd @xcite span multiple domains, but individual domains contain few user intents, and conversations are not designed to be representative of real human-to-human customer service interactions. @xcite re-purpose two-turn customer support exchanges from twitter, but only annotate a small subset of dialogues across 14 intents with broad semantics. hakkani-tür et al. (2015) evaluate the classification performance of induced intents after undergoing manual mappings to reference intents. more recent work has used clustering metrics commonly used for evaluation of short text clustering in which the number of clusters is provided, such as clustering acc, nmi, and ari @xcite @xcite . recent work has also investigated the intent discovery problem in which systems must discover novel intents based on a set of pre-existing intents @xcite @xcite . in contrast to this work, this benchmark focuses on the fully unsupervised setting in which no pre-existing intents are defined and the number of reference intents are not provided in advance. ## tasks and evaluation the track consists of two subtasks providing alternative ways of framing and evaluating intent induction. this section describes the motivation for these tasks and metrics used for evaluating them. ## data publicly available dialogue datasets have primarily focused on development and evaluation of taskoriented dialogue systems where conversations are representative of written human-to-bot (h2b) conversations adhering to restricted domains and schemas @xcite . such datasets are not designed to be reflective of the characteristics of human-to-human (h2h) conversations, and thus are unlikely to serve as a realistic test bed for evaluating systems designed to learn from natural conversations. however, realistic live conversations are difficult to simulate due to the training required to convincingly play the role of an expert customer support agent in non-trival domains @xcite and the additional costs associated with collecting and annotating free-form synchronous conversations. to address this gap, we introduce a benchmark dataset designed to emulate natural call center conversations between customers and customer support agents. each conversation emulates a twoparty spoken-form customer support scenario corresponding to a generated scenario based on a combination of intents, slots, and complex conversational phenomena to encourage diversity and naturalness. to naturally collect a wide variety of intents, participants were encouraged to depart from the original intents with additional requests related to each scenario. the process of annotating conversations with reference intents was decoupled from the collection of conversations in order to mimic the manual process of designing an intent schema based on conversations. annotators shared an open intent label set that was periodically reviewed throughout the process to merge duplicate intents. we provide three domains as part of the challenge track: insurance (used as development data), personal banking and finance (used as evaluation data). for evaluating task 2, each domain also includes a h2b-style balanced test set containing utterances labeled with intents from a reference intent schema. these test sets are collected independently from the h2h-style conversations. conversations are also labeled with automatic informintent dialogue act predictions indicating potentially relevant turns for use in task 2, though these are non-exhaustive and include utterances that are not relevant. we present high-level statistics for each of the domains in comparison with pre-existing dialogue datasets in fine-grained counts of intents for conversations and test sets and further analysis of intents are provided in sections a.1 and a.2. task 1 baseline the baseline system for task 1 casts the problem as turn-level unsupervised clustering, adopting k-means @xcite as the clustering algorithm. utterances are encoded using a sentence embedding model from the sentencetransformers library (reimers and gurevych, 2019), all-mpnet-base-v2, which fine-tunes mpnet @xcite ) with a contrastive objective on a dataset consisting of 1 billion sentence pairs derived from a number of datasets. because the number of reference intents is not provided for the task, the baseline system identifies the number of intents automatically by selecting the value for k that results in the highest intrinsic measure of clustering performance. silhouette values @xcite indicate the appropriateness of a cluster assignment for a point based on the similarity to points in its assigned cluster and dissimilarity to points in other clusters. an overall silhouette score can be computed by averaging the silhouettes for all points in a clustering. the k value that leads to the highest silhouette score is selected as the final result. to accelerate the search for optimal clustering based on silhouette scores, we employ sequential model-based global optimization following the tree-structured parzen estimator (tpe) approach, implemented in the hyperopt library @xcite . task 2 baseline the baseline system for task 2 adopts the same clustering approach as task 1. since clustering turns are not provided in this task, the baseline system uses the provided informintent dialogue act classifier predictions to identify turns containing intents. ## submissions we received submissions from 34 teams for task 1 and 19 teams for task 2 encompassing a wide range of techniques. all teams that participated in task 2 also participated in task 1. to preserve anonymity, teams are identified with ids t0 through t37 (several team ids were later removed after confirming duplicate submissions from different members of the same team). this section provides a high-level summary of the submissions using self-reported descriptions and survey results of the participants. a detailed overview of submissions is provided in appendix clustering approach for both tasks, all submissions reported using clustering-based solutions. many submissions used clustering approaches that jointly learn input representations and cluster assignments. for example, at least ten submissions (t00, t02, t05, t07, t16, t17, t24, t30, t34, and t36) reported used sccl @xcite , a clustering approach that incorporates contrastive loss with deep embedded clustering (dec) @xcite . other approaches (e.g. t06 and t37) separated representation learning from clustering, pre-training encoders with contrastive learning followed by k-means clustering. t06 used k-means with representations from dsscc @xcite . several submissions used hdbscan for clustering (t11, t13, t22, and t35), a density-based clustering algorithm that extends dbscan by allowing for clusters of varying density @xcite . like dbscan @xcite , it automatically categorizes outliers in low-density regions as noise, an appealing property for a task in which many noisy, outlying input utterances are expected to be present, though only one of these teams participated in task 2 (t13). two teams (t13 and 35) performed dimensionality reduction on pre-trained embeddings using umap @xcite prior to clustering, following bertopic @xcite . three other teams (t22, t27, and t31) also reported using some form of dimensionality reduction as part of their submissions. pre-trained models almost all submissions reported using specific pre-trained models, many of which were encoders tailored for computing utterance embeddings such as all-mpnet-base-v2 @xcite , dse-bert-base, dse-roberta-large @xcite , and sup-simcse-roberta @xcite . teams t00, t15, t23 and t34 used dsepretrained encoders @xcite , which learn sentence representations tailored for dialogues using contrastive learning with consecutive utterances in dialogues. the majority of submissions also reported further fine-tuning encoders to tailor them to the task of intent induction (such as through sccl, dec, or supervised contrastive learning). t11 reported using two pre-trained nli models to pre-compute a distance matrix between inputs for hdbscan-based clustering, further finetuning one of these models on the task development data. ## results in this section, we summarize the results for task 1 and task 2. to aggregate results across datasets, we use simple averages. as an alternative aggregate score, we compute the average mean reciprocal rank for acc, f 1 , and nmi, averaged over both datasets (mrr avg ). this approach is intended to be less sensitive to scores of the individual datasets or biases of particular metrics. team t23 was the overall winner for task 1, with an honorable mention to team t05 for having the highest mrr avg . ## conclusions we presented task definitions, evaluation methods, datasets, and baselines for the dstc 11 track on intent induction from conversations for task-oriented dialogue. the track saw a variety of submissions from 34 teams, with 19 teams submitting entries for both tasks. we summarized these track submissions and provided analysis of the trends and overall results. the aim of the track is to provide a benchmark facilitating evaluation of methods for automatic induction of customer intents in the realistic setting of customer service interactions. we hope that the benchmark and datasets will encourage new lines of research related to the analysis of human-tohuman conversations.