IDAS : Intent Discovery with Abstractive Summarization (2023)

Abstract: Intent discovery is the task of inferring latent intents from a set of unlabeled utterances, and is a useful step towards the efficient creation of new conversational agents. We show that recent competitive methods in intent discovery can be outperformed by clustering utterances based on abstractive summaries, i.e., “labels”, that retain the core elements while removing non-essential information. We contribute the IDAS approach, which collects a set of descriptive utterance labels by prompting a Large Language Model, starting from a well-chosen seed set of prototypical utterances, to bootstrap an In-Context Learning procedure to generate labels for non-prototypical utterances. The utterances and their resulting noisy labels are then encoded by a frozen pre-trained encoder, and subsequently clustered to recover the latent intents. For the unsupervised task (without any intent labels) IDAS outperforms the state-of-the-art by up to +7.42% in standard cluster metrics for the Banking, StackOverflow, and Transport datasets. For the semi-supervised task (with labels for a subset of intents) IDAS surpasses 2 recent methods on the CLINC benchmark without even using labeled data.

## introduction intent classification is ubiquitous in conversational modelling. to that end, finetuning large language models (llms) on task-specific intent data has been proven very effective @xcite . however, such finetuning requires manually annotated (utterance, intent) pairs as training data, which are time-consuming and thus expensive to acquire. companies often have an abundance of utterances relevant to the application area of their interest, e.g., those exchanged between customers and support agents, but manually annotating them remains costly. consequently, intent discovery aims to recover latent intents without using any such manually annotated utterances, by partitioning a given set of (unlabeled) utterances into clusters, where utterances within a cluster should share the same conversational goal or intent. prior works typically (i) train an unsupervised sentence encoder to map utterances to vectors, after which these are (ii) clustered to infer latent intents. such unsupervised encoder training is achieved largely under the assumption that utterances with similar encodings convey the same intent. for instance, by iteratively clustering and updating the encoder with supervision from the cluster assignments @xcite @xcite , or by retrieving utterances with similar encodings and using them as positive pairs to train the encoder with contrastive learning @xcite . yet, it remains unclear which particular features cause utterance representations to be similar. various noisy features unrelated to the underlying intents, e.g., syntax, n-gram overlap, nouns, etc. may contribute in making utterances similar, leading to sentence encoders whose vector encodings may inadequately represent the underlying intents. different from prior works that train unsupervised encoders, we use a pre-trained encoder without requiring any further finetuning, since we pro-pose making utterances more (dis)similar in the textual space by abstractly summarizing them into concise descriptions, i.e., "labels", that preserve their core elements while removing non-essential information. we hypothesize that these core elements better represent intents and prevent nonintent related information from influencing the vector similarity. we compare our idas approach with the state-ofthe-art in unsupervised intent discovery on banking @xcite , stackoverflow @xcite , and a private dataset from a transport company, to assess idas's effectiveness in practice. we show that idas substantially outperforms the state-of-the-art, with average improvements in cluster metrics of +3.94%, +2.86%, and +3.34% in adjusted rand index, normalized mutual information, and cluster accuracy, respectively. fur-ther, idas surpasses two semi-supervised intent discovery methods on clinc @xcite despite not using any ground truth annotations. neural sentence encoders: rather than relying on external knowledge sources, neural approaches pre-train sentence encoders in a self-supervised way @xcite , or with supervision @xcite @xcite , to produce dense general-purpose vectors that better capture synonymy and semantic relatedness. unsupervised intent discovery: since generalpurpose neural encoders may fail to capture domain-specific intent information, intent discovery solutions have shifted towards unsupervised sentence encoders specifically trained on the domain data at hand. for instance, @xcite train a self-supervised convolutional neural network, and use it to encode and cluster utterances with k-means. @xcite adopt the same 2-step approach, but instead pre-train the encoder with contrastive learning, where utterances with similar vector encodings are retrieved to serve as positive pairs. a more common strategy is to cluster and train the encoder end-to-end, either by (i) iteratively clustering utterances and updating the encoder with supervision from the cluster assignments @xcite @xcite , or (ii) simultaneously clustering utterances and updating the encoder's weights with a joint loss criterion @xcite . as an alternative strategy to make utterances more (dis)similar based on the intents they convey, we employ an llm to summarize utterances into labels that retain both the utterances' core ele- ## conclusions unlike existing methods that train unsupervised sentence encoders, our idas approach employs a frozen pre-trained encoder since it increases the (dis)similarity of (un)related utterances in the textual space by abstractly summarizing utterances into "labels". our experiments demonstrate that idas substantially outperforms the current state-ofthe-art in unsupervised intent discovery across multiple datasets (i.e., banking, stackoverflow, and our private transport), and surpasses two recent semi-supervised methods on clinc, despite not using any labeled intents at all. our findings suggest that our alternative strategy of abstractly summarizing utterances (using a general purpose llm) is more effective than the dominant paradigm of training unsupervised encoders (specifically on dialogue data), and thus may open up new perspectives for novel intent discovery methods. since our generated labels provide a better measure of intentrelatedness, we hypothesize that they could also enhance the performance of existing methods that train unsupervised encoders, e.g., by (i) reducing the number of false positive contrastive pairs for mtp-clnn @xcite , or (ii) improving the purity of clusters induced by methods that iteratively cluster utterances and update the encoder with (self-)supervision from cluster assignments @xcite @xcite . to facilitate such follow-up work, we release our generated labels for the banking, stackoverflow, and clinc datasets. foot_1