Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification (2023)

Abstract: Intent classification (IC) plays an important role in task-oriented dialogue systems. However, IC models often generalize poorly when training without sufficient annotated examples for each user intent. We propose a novel pre-training method for text encoders that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks, reducing the need for manual annotations. By applying this pre-training strategy, we also introduce Pre-trained Intent-aware Encoder (PIE), which is designed to align encodings of utterances with their intent names. Specifically, we first train a tagger to identify key phrases within utterances that are crucial for interpreting intents. We then use these extracted phrases to create examples for pre-training a text encoder in a contrastive manner. As a result, our PIE model achieves up to 5.4% and 4.0% higher accuracy than the previous state-of-the-art pre-trained text encoder for the N-way zero- and one-shot settings on four IC datasets.

## introduction identification of user intentions, a problem known as intent classification (ic), plays an important role in task-oriented dialogue (tod) systems. however, it is challenging for tod developers to collect data and re-train models when designing new intent classes. recent studies have aimed to tackle this challenge by applying zero-and few-shot text classification methods and leveraging the semantics of intent label names @xcite @xcite @xcite @xcite . @xcite compare various classification methods on few-shot ic tasks and find that prototypical networks @xcite consistently show strong performance when combined with transformer-based text encoders. prototypical networks use text encoders to construct class representations and retrieve correct classes given queries based on a similarity metric. @xcite also stress that few-shot learning techniques and text encoders can have an orthogonal impact on classification performance. thus, although some studies have focused on improving learning techniques for few-shot ic tasks @xcite , better text encoder selection should also be considered as an impor-tant research direction. @xcite observe that sentence encoders pre-trained on paraphrase or natural language inference datasets serve as strong text encoders for prototypical networks. however, existing sentence encoders are not explicitly designed to produce representations for utterances that are similar to their intent names. therefore, their abilities are limited in zero-and few-shot settings where predictions may heavily rely on the semantics of intent names. pre-training encoders to align user utterances with intent names can mitigate this issue; however, it is typically expensive to obtain annotations for a diverse intent set. in this paper, we propose a novel pre-training method for zero-and few-shot ic tasks (figure 1 ). specifically, we adopt intent role labeling (irl) @xcite , which is an approach for identifying and assigning roles to words or phrases that are relevant to user intents in sentences. once we obtain the irl predictions, we convert them to the pseudo intent names of query utterances and use them to pre-train the encoder in a contrastive learning fashion. this intent-aware contrastive learning aims to not only align utterances with their pseudo intent names in the semantic embedding space, but also to encourage the encoder to pay attention to the intent-relevant spans that are important for distinguishing intents. to the best of our knowledge, this work is the first to extract key information from utterances and use it as pseudo labels for pre-training intent-aware text encoders. the contributions of our work are as follows: ## background: prototypical networks for intent classification prototypical networks @xcite is a metalearning approach that enables classifiers to quickly adapt to unseen classes when only a few labeled examples are available. several studies have demonstrated the effectiveness of prototypical networks when building intent classifiers with a few example utterances @xcite @xcite . they first define a fewshot ic task, also known as an episode in the metalearning context, with k example utterances from n intent classes (i.e., k×n utterances in a single episode). at the training time, the intent classifiers are optimized on a series of these episodes. example utterances for each intent class are called a support set, and are encoded and averaged to produce a class representation, called a prototype. this can be formulated as follows: ## pseudo intent name generation to pre-train an encoder f ϕ that works robustly in zero-or few-shot ic settings, a variety of predefined intent names are required. because annotating them is expensive, we opt to automatically generate pseudo intent names from utterances in our pre-training data. to annotate pseudo intents, we employ a tagging method, intent role labeling (irl). irl can be considered similar to semantic role labeling (srl), which is a task of assigning general semantic roles to words or phrases in sentences @xcite . however, irl focuses on providing an extractive summary of the intent expressed in a user's utterance, annotating important roles with respect to the goal of the user rather than a predicate. specifically, it tags words or phrases that are key to interpret intent. irl was first introduced by zeng et al. (2021) for discovering intents from utterances, but their tagger focuses only on chinese utterances. in this section, we outline the process of building the irl tagger from scratch. we provide a description of how we annotate irl training data on english utterances (section 3.1), the training procedure for the irl tagger (section 3.2), and the utilization of irl predictions for generating pseudo intent names to pre-train our model (section 3.3). ## intent-aware contrastive learning we aim to build an encoder that produces similar representations between utterances and the corresponding intent names. in this section, we introduce the intent-aware contrastive learning approach using triples of an utterance, gold intent, and pseudo intent from various dialogue datasets. our training objective is designed to align the representations of utterances and their intent names in the semantic embedding space. for this purpose, we use the infonce loss (van den oord et al., 2018), which pulls positive pairs close to each other and pushes away negative pairs. the loss for the i-th sample x i is formulated as follows: the second supervised positive pair is between the input utterances and their gold utterances. we define gold utterances as randomly sampled utterances that share the same gold intent names as the input utterances: our final loss is a combination of these three losses as follows: ## conclusions in this work, we propose a pre-training method that leverages pseudo intent names constructed using an irl tagger in a semi-supervised manner, followed by intent-aware pre-training (pie). experiments on four intent classification datasets show that our model achieves state-of-the-art performance on all datasets, outperforming the strongest sentence encoder baseline by up to 5.4% and 4.0% in n-way zero-and one-shot settings, respectively. our analysis shows that pie performs robustly compared to the baselines in challenging and practical settings with a large number of classes and small number of support examples. in future work, we will explore the use of irl and our pie model in multi-label intent classification or out-of-scope detection tasks.