I nten DD : A Unified Contrastive Learning Approach for Intent Detection and Discovery (2023)

Abstract: Identifying intents from dialogue utterances forms an integral component of task-oriented dialogue systems. Intent-related tasks are typically formulated either as a classification task, where the utterances are classified into predefined categories or as a clustering task when new and previously unknown intent categories need to be discovered from these utterances. Further, the intent classification may be modeled in a multiclass (MC) or multilabel (ML) setup. While typically these tasks are modeled as separate tasks, we propose IntenDD a unified approach leveraging a shared utterance encoding backbone. IntenDD uses an entirely unsupervised contrastive learning strategy for representation learning, where pseudo-labels for the unlabeled utterances are generated based on their lexical features. Additionally, we introduce a two-step post-processing setup for the classification tasks using modified adsorption. Here, first, the residuals in the training data are propagated followed by smoothing the labels both modeled in a transductive setting. Through extensive evaluations on various benchmark datasets, we find that our approach consistently outperforms competitive baselines across all three tasks. On average, IntenDD reports percentage improvements of 2.32 %, 1.26 %, and 1.52 % in their respective metrics for few-shot MC, few-shot ML, and the intent discovery tasks respectively.

## introduction intents form a core natural language understanding component in task-oriented dialogue (tod) systems. intent detection and discovery not only have immense utility but are also challenging due to numerous factors. intent classes vary vastly from one use case to another, and often arise out of business needs specific to a particular product or organization. further, modeling requirements might necessitate considering fine-grained and semantically-similar concepts as separate intents @xcite . overall, intent-related tasks typically are expected to be scalable and resource efficient, to quickly bootstrap to new tasks and domains; lightweight and modular for maintainability across domains and expressive to handle large, related often overlapping intent scenarios @xcite . intendd proposes a unified framework for intent detection and discovery from dialogue utterances from tod systems. the framework enables the modeling of various intent-related tasks such as intent classification, both multiclass and multilabel, as well as intent discovery, both unsupervised and semi-supervised. in intent detection (classification), we expect every class to have a few labeled instances, say 5 or 10. however, in intent discovery, not all classes are expected to have labeled instances and may even be completely unlabeled. recently, intent-related models focus more on contrastive representation learning, owing to the limited availability of labeled data and the presence of semantically similar and fine-grained label space @xcite . similarly, a common utterance encoder forms the backbone of intendd, irrespective of the task. the utterance encoder is learned by updating the parameters of a general-purpose pre-trained encoder using a two-step contrastive representation learning process. first, we adapt a general-purpose pre-trained encoder by using unlabelled information from various publicly available intent datasets. second, we update the parameters of the encoder using utterances from the target dataset, on which the task needs to be performed, making the encoder specialize on the corpus. here, we use both labeled and unlabelled utterances from the target dataset, where pseudo labels are assigned to the latter. for intent classification, both multiclass and multilabel, intendd consists of a three-step pipeline. it includes training a classifier that uses the rep-resentation from the encoder as its feature representation, followed by two post-processing steps in a transductive setting. specifically, a multilayer perceptron-based classifier is trained by stacking it on top of the utterance representation from our encoder. the post-processing steps consider the target corpus as a graph in a transductive setting. the first postprocessing step involves propagating the residual errors in the training data to the neighbors. the second one further performs label smoothing by propagating the labels obtained from the previous step. both these steps are performed using modified adsorption, an iterative algorithm that enables controlling the propagation of information that passes through a node more tightly @xcite . major contributions: intendd reports performance improvements compared to that of competitive baselines in all the tasks and settings we experimented with, including multiclass and multilabel classification in few-shot and high data settings; unsupervised and semi-supervised intent discovery. our two-step post-processing setup for intent classification leads to statistically significant performance improvements to our base model. while existing intent models focus primarily on better representation learning and data augmentation, we show that classical transductive learning approaches can help improve the performance of intent models even in fully supervised settings. finally, we show that with a careful construction of a graph structure in a transductive learning setting in terms of both edge formation and edge weight formation can further improve our outcomes. ## intendd intendd consists of a two-step representation learning module, a classification module, and an intent detection module. we elaborate on each of these modules in this section. ## experimental setup we perform experiments for the three intent related tasks -intent discovery, multiclass intent detection, and multi-label intent detection. here, we provide training and modeling details that are common to all three tasks and then mention taskspecific details such as the baselines and evaluation metrics at appropriate sections. pretraining datasets. one feature of in-tendd is the unification of these three tasks via a common pretrained transformer backbone. this common pretraining step is performed on clinc-150 @xcite , bank-ing77 @xcite , hwu64 @xcite , nlu++ @xcite , and stackoverflow @xcite . following prior work on contrastive learning for intent detection by @xcite , we additionally include top @xcite , snip @xcite , and atis @xcite . training and modeling details. we choose roberta @xcite with the base configuration as our common encoding backbone and pretrain with aforementioned datasets. for encoding the input utterances, we use a cross-encoder architecture as detailed by @xcite . in this setup, the joint embedding for any pair of utterances (p, q) -needed for contrastive learning for instance-is obtained by embedding it as "[cls] p [sep] q" and the [cls] representation is used as the representation for that pair. @xcite found that a cross-encoder approach works much better than a bi-encoder where any pair of utterances are independently embedded. we perform all of our experiments using the tranformers library @xcite and the pytorch framework @xcite . we train our models using the adamw optimizer with learning rate set to 2e-5, warmup rate of 0.1, and weight decay of 0.01. we pretrain our model for 15 epochs, and thereafter perform task-specific training for another 20 epochs. all experiments are performed on a machine with nvidia a100 80gb and we choose the maximum batch size that fits the gpu memory (= 96). we perform hyperaparameter search for the temperature τ and lambda λ over the ranges τ ∈ {0.1, 0.3, 0.5}, and λ ∈ {0.01, 0.03, 0.05}. ## conclusion in summary, this paper presents a novel approach, intendd, for intent detection and discovery in task-oriented dialogue systems. by leveraging a shared utterance encoding backbone, in-tendd unifies intent classification and novel intent discovery tasks. through unsupervised contrastive learning, the proposed approach learns representations by generating pseudo-labels based on lexical features of unlabeled utterances. additionally, the paper introduces a two-step postprocessing setup using modified adsorption for classification tasks. while intent classification tasks typically focus on contrastive representation learning or data augmentation, we show that a two-step post-processing setup in a transductive setting leads to statistically significant improvements to our base classifier, often rivaling or at par with data augmentation approaches. extensive evaluations on diverse benchmark datasets demonstrate the consistent improvements achieved by our system over competitive baselines.