An Explicit-Joint and Supervised-Contrastive Learning Framework for Few-Shot Intent Classification and Slot Filling (2021)

Abstract: Intent classification (IC) and slot filling (SF) are critical building blocks in task-oriented dialogue systems. These two tasks are closely-related and can flourish each other. Since only a few utterances can be utilized for identifying fast-emerging new intents and slots, data scarcity issue often occurs when implementing IC and SF. However, few IC/SF models perform well when the number of training samples per class is quite small. In this paper, we propose a novel explicit-joint and supervised-contrastive learning framework for few-shot intent classification and slot filling. Its highlights are as follows. (i) The model extracts intent and slot representations via bidirectional interactions, and extends prototypical network to achieve explicit-joint learning, which guarantees that IC and SF tasks can mutually reinforce each other. (ii) The model integrates with supervised contrastive learning, which ensures that samples from same class are pulled together and samples from different classes are pushed apart. In addition, the model follows a not common but practical way to construct the episode, which gets rid of the traditional setting with fixed way and shot, and allows for unbalanced datasets. Extensive experiments on three public datasets show that our model can achieve promising performance.

## introduction with the vigorous development of conversational ai, task-oriented dialogue systems have been widely-used in many applications, e.g., virtual personal assistants like apple siri and google assistant, and chatbots deployed in various domains @xcite . intent classification (ic) and slot filling (sf) are key components in task-oriented dialogue systems, and their performance will directly affect the downstream dialogue management and natural language generation tasks * corresponding author. @xcite . traditional ic/sf models have achieved impressive performance @xcite , but they often require large amount of labeled instances per class, which is expensive and unachievable in industry especially in the initial phase of a dialogue system. few-shot learning aims to solve the data scarcity issue, which can recognize novel categories effectively with only a handful of labeled samples by leveraging the prior knowledge learned from previous categories. most few-shot learning studies concentrate on computer vision domain @xcite @xcite . recently, to handle various new or unacquainted intents popped up quickly from different domains, some few-shot ic/sf models are proposed @xcite . nevertheless, these methods usually focus on a single task and do not attempt to address these two tasks simultaneously. intuitively, ic and sf are two complementary tasks and the information of one task can be utilized in the other task to improve the performance. existing joint ic and sf models have achieved impressive performance in supervised learning scenarios @xcite . but only a couple of methods are custom-designed for few-shot joint ic and sf task. @xcite directly apply the popular few-shot learning models maml and prototypical network to explore the few-shot joint ic and sf. during the same period, bhathiya and thayasivam (2020) also attempt to utilize maml to deal with this problem in a similar way. though these models outperform the single task model, they just implicitly model the relationship between ic and sf. the mutual interaction between ic and sf in these methods is still unknowable, which seems to be a black box (not using a concrete formula to characterize the interaction), thus difficult to analyze the internal mechanism. in this paper, we propose to model the relationship between ic and sf precisely and clearly, as meanwhile, intent and slot labels' descriptions are fed into preprocessing module to generate intent embedding matrix and slot embedding matrix. then the two matrices and utterance's embedding are fed into explicit joint learning module, while utterance's embedding is put forward into supervised contrastive learning module. in explicit joint learning module, intent and slot extractors are used to extract intent and slot information, which leverage the attention mechanism. then, we can obtain slot-attentionbased intent representation and intent-attention-based slot representation. next, prototypical network uses intent labels to guide slot embedding learning and vice versa. in supervised contrastive learning module, we construct contrastive samples for each query instance using support set. and the scl loss function can push samples from the same class more close and samples from different classes further apart. in the testing process, prototypical network is used to predict intent and slot labels, while supervised contrastive learning module is disabled. to verify the effectiveness of the proposed model, we conduct extensive experiments on three public datasets. catering to the unbalanced datasets and very limited labeled samples in real application scenarios, we adopt a not common but practical way to construct the episode for few-shot learning, i.e., in each episode, the way and shot are variable. the empirical study validates our proposal and shows promising results of our framework on ic and sf tasks. few-shot classification is an important and challenging task. several methods have been proposed to tackle this problem. in particular, several metricbased methods @xcite @xcite @xcite have been proposed, which first learn an embedding space and then utilize a metric to classify instances of new categories according to prox-imities with the labeled examples. in addition to metric-based methods, some optimization-based approaches @xcite @xcite have also been explored for few-shot classification. contrastive learning contrastive learning applied to self-supervised representation learning has seen a resurgence of interest in recent years, leading to state-of-the-art performance in unsupervised training of deep image models @xcite . @xcite extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. recently, @xcite propose a novel objective function that contains a supervised contrastive learning term for fine-tuning pre-trained language models, which improves the model generalization ability significantly. joint intent classification and slot filling due to the close relationship between ic and sf, @xcite ; @xcite ; @xcite ; @xcite propose joint models to consider the correlation between these two tasks. these models can be classified into two categories. one type of approaches @xcite adopt a multitask framework to solve these two tasks simultaneously. although these models outperform the single-task model, they just model the relationship implicitly by sharing the encoder parameters. the other type of approaches @xcite explicitly adopt the intent information to guide the slot filling task. @xcite further propose a co-interactive transformer which considers the cross-impact between these two tasks. these explicit-joint learning models have achieved very remarkable performance, but they mainly focus on the traditional supervised learning setting. ## problem definition a labeled utterance with t words (tokens) can be represented as (x, t, y), where @xmath0, w 2 , ..., w t ) is an utterance with t words, @xmath1, t 2 , ..., t t ) is composed of slot labels of each word in x, y is the intent label of x. in this paper, few-shot classification is conducted via episode learning strategy. in the training period, we partition the training set into multiple episodes. each episode consists of a support set s and a query set q. in particular, we randomly select n classes from the training classes, and obtain a class set c in each episode. then the support set is formed by randomly selecting k c labeled samples (utterances) from each of the n classes, i.e., @xmath2, where and a fraction of the remainder of these n classes' samples (k q examples per class) serve as the query set, i.e., @xmath3, where in the test period, we also partition the test set into multiple episodes. each episode contains a support set @xmath4, where s @xmath5, t i , y c )|@xmath6, ..., k c )}, and a query set @xmath7, where there is no overlap between the training classes and test classes. ## conclusion in this paper, we propose a new and practicable framework for few-shot intent classification and slot filling. the performance gains of our method come from two aspects: explicit-joint learning and supervised-contrastive learning. by explicit-joint learning, we can effectively utilize the close relationship between ic and sf tasks. by supervisedcontrastive learning, we can obtain more classindicative representations. we thoroughly evaluate our framework on few-shot ic and sf tasks and achieve impressive performance on three public datasets snips, atis and top. in future work, we plan to explore more explicit-joint learning strategies and extend our framework to deal with multiple-intent classification.