Strategies to Improve Few-shot Learning for Intent Classification and Slot-Filling (2022)

Abstract: Intent classification (IC) and slot filling (SF) are two fundamental tasks in modern Natural Language Understanding (NLU) systems. Collecting and annotating large amounts of data to train deep learning models for such systems are not scalable. This problem can be addressed by learning from few examples using fast supervised meta-learning techniques such as prototypical networks. In this work, we systematically investigate how contrastive learning and data augmentation methods can benefit these existing meta-learning pipelines for jointly modelled IC/SF tasks. Through extensive experiments across standard IC/SF benchmarks (SNIPS and ATIS), we show that our proposed approaches outperform standard meta-learning methods: contrastive losses as a regularizer in conjunction with prototypical networks consistently outperform the existing state-of-the-art for both IC and SF tasks, while data augmentation strategies primarily improve few-shot IC by a significant margin

## introduction nlu specific intent classification and slot-filling models often need to learn from only a few contextual examples given by the end user in industrial model deployment scenarios. such models are often trained using meta-learning, a competitive fewshot learning strategy to learn from only a few examples. in this paper, we systematically dissect the existing meta-learning pipelines for jointly modelled few-shot intent classification (ic) and slot filling (sf) and identify practical training strategies to improve their performance by a significant margin. precisely, we investigate how different data augmentation and contrastive learning strategies improve ic/sf performance, and show that our training approach outperforms state-of-the-art models for few-shot ic/sf. given the user utterance: "book me a table for 6 at lebanese taverna", an ic model identifies "restaurant booking" as the intent of interest, and an sf model identifies the slot types and values: party_size:"6", name: "lebanese taverna". these functionalities are typically driven by powerful deep learning models that rely on huge amounts of domain-specific training data. as such labeled data is rarely available, building models that can learn from only a few examples per class is inevitable. few-shot learning techniques @xcite @xcite @xcite have been recently proposed to address the problem of generalizing to unseen classes in ic/sf when only a few training examples per class are available. @xcite utilized meta-learning approaches such as prototypical networks @xcite and maml @xcite to jointly model ic/sf. they showed that prototypical networks outperform other prevalent meta-learning techniques such as maml as well as fine-tuning. moreover, one primary benefit of prototypical networks is that it is computationally cheap during meta-testing, thus making it a good candidate for industrial few-shot learning systems. in this paper, we extend this powerful supervised meta-learning technique with unsupervised contrastive learning and data augmentation. @xcite showed that metalearners can be particular prone to overfitting which can be partially alleviated by data augmentation @xcite . data augmentation strategies in nlp have been shown to boost performance in general text classification settings @xcite @xcite , however, there exists very little work on how data augmentation can be effectively used in the meta-learning pipeline specific to nlu tasks. to address this question, we first use a data augmentation strategy slot-list values for ic/sf tasks which generates synthetic utterances using dictionary-based slot-values. we note that similar dictionary based augmentation has been previously used in @xcite , but in the context of dialogue state tracking, orthogonal to our use-case. additionally, we investigate how state-of-the-art augmentation strategies such as backtranslation @xcite and perturbation-based augmentations such as eda -easy data augmentation @xcite )can be used alongside prototypical networks. we further investigate how contrastive learning @xcite can be used as a regularizer during the meta-training stage to create better generalizable meta-learners. contrastive learning is useful in creating improved prototypes as they pull similar representations together while pushing apart dissimilar ones. through extensive experiments across snips and atis, we show that metatraining with contrastive losses as a regularizer improves ic/sf performance for unseen classes with few examples. our contributions include: ## proposed approaches we follow the few-shot learning setup for ic/sf described in @xcite with a few modifications. instead of using a frozen backbone such as bert or elmo with a bilstm head, we use a more powerful pre-trained roberta encoder. additionally, in contrast to @xcite , we update our encoder during the meta-training stage. for a given utterance x @xmath0, x i 2 , ..., x i n } with n tokens, we first use the roberta model denoted by f ϕ to encode the utterance resulting in h @xmath1, h i 1 , ..., h i n }. we use @xmath2. for slot filling, we use each of the token embeddings {h i j } n @xmath3. given a support set s, assuming s l consists of utterances belonging to the intent class c l and s a consists of tokens from the slot class c a , we first compute the class prototypes for intents (c l ) and slots (c a ): given a query example z and a distance function d, a distribution over the different classes is computed using the softmax of the distances to the different class prototypes. specifically we denote the intent specific log likelihood loss as: the general idea of contrastive learning @xcite is to pull together the representations of similar samples while pushing apart the representations of dissimilar samples in an embedding space. in our work, we specifically incorporate the supervised contrastive loss as an added regularizer with the prototypical loss computation in eq. ( 4 ). in particular we identify places in the meta-training pipeline where the incorporation of the contrastive loss is most beneficial for good generalization to few-shot classes. we devise two types of contrastive losses for the ic/sf tasks: (a) contrastive loss for intents l contrastiveic (ϕ) where @xmath4; (b) contrastive loss for slots l contrastivesf (ϕ) where the individual token embeddings are used in the loss. the regularized prototypical loss is the following: we provide more details about the two contrastive losses in the appendix section. ## conclusion in this work, we systematically dissect metalearning pipelines for few-shot ic/sf tasks and identify stages during meta-learning where contrastive learning and data augmentation can be effective. empirically, we found that contrastive losses are effective regularizers during metatraining and outperform the current state-of-theart few-shot joint ic/sf benchmarks across both snips and atis. impact of data augmentation in general is highly dependent on the stage at which it is applied during meta-learning. notably, a combination of contrastive losses and data augmentation via slot-list values during meta-training leads to the best performances across both snips and atis. these strategies for improving few-shot ic/sf tasks create a strong benchmark and open up possibilities on more stronger modes of metaspecific augmentation and contrastive learning.