Few-shot Intent Classification and Slot Filling with Retrieved Examples (2021)

Abstract: Few-shot learning arises in important practical scenarios, such as when a natural language understanding system needs to learn new semantic labels for an emerging, resource-scarce domain. In this paper, we explore retrieval-based methods for intent classification and slot filling tasks in few-shot settings. Retrieval-based methods make predictions based on labeled examples in the retrieval index that are similar to the input, and thus can adapt to new domains simply by changing the index without having to retrain the model. However, it is non-trivial to apply such methods on tasks with a complex label space like slot filling. To this end, we propose a span-level retrieval method that learns similar contextualized representations for spans with the same label via a novel batch-softmax objective. At inference time, we use the labels of the retrieved spans to construct the final structure with the highest aggregated score. Our method outperforms previous systems in various few-shot settings on the CLINC and SNIPS benchmarks.

## introduction few-shot learning is a crucial problem for practical language understanding applications. in the fewshot setting, the model (typically trained on source domains with abundant data) needs to adapt to a set of unseen labels in the target domain with only a few examples. for instance, when developers introduce a new product feature, a query understanding model has to learn new semantic labels from a small dataset they manage to collect. few-shot learning is challenging due to the imbalance in the amount of data between the source and target domains. traditional classification methods, even with the recent advancement of pretrained language models @xcite , could suffer from over-fitting @xcite work done during internship at google research @xcite or catastrophic forgetting @xcite when incorporating the data-scarce target domain. on the other hand, metric learning methods @xcite @xcite have been shown to work well in few-shot scenarios. these methods are based on modeling similarity between inputs, effectively allowing the model to be decoupled from the semantics of the output space. for example, a model would learn that the utterance "i'd like to book a table at black horse tavern at 7 pm" (from figure 1 ) is similar to "make me a reservation at 8" and thus are likely to have similar semantic representations, even without knowing the semantic schema in use. unlike learning output labels, which is difficult when examples are scarce, learning a similarity model can be done on the abundant source domain data, making such models data-efficient even in few-shot settings. while there are many instantiations of metric learning methods (see section 3), we focus on retrieval-based methods, which maintain an explicit retrieval index of labeled examples. the most basic setting of retrieval-based model for few-shot learning is: after training a similarity model and encoding target domain data into the index, we can retrieve examples most similar to the given input, and then make a prediction based on their labels. compared to methods that do not maintain an index, such as prototypical networks @xcite , retrieval-based methods are less sensitive to outliers with few data points, and are powerful when we have abundant data in the source domain @xcite . however, applying retrieval-based models on tasks with a structured output space is non-trivial. for example, even if we know that the utterance in figure 1 is similar to "make me a reservation at 8", we cannot directly use its slot values (e.g., the time slot has value "8" which is not in the input), and not all slots in the input (e.g., "black in this work, we propose retriever, a retrieval-based framework that tackles both classification and span-level prediction tasks. the core idea is to match token spans in an input to the most similar labeled spans in a retrieval index. for example, for the span "7 pm" in the input utterance, the model retrieves "8" as a similar span (given their surrounding contexts), thus predicting that "7 pm" has the same slot name time as "8". during training, we fine-tune a two-tower model with bert @xcite encoders, along with a novel batch softmax objective, to encourage high similarity between contextualized span representations sharing the same label. at inference time, we retrieve the most similar span from the fewshot examples for every potential input span, and then decode a structured output that has the highest average span similarity score. we show that our proposed method is effective on both few-shot intent classification and slotfilling tasks, when evaluated on clinc @xcite and snips @xcite datasets, respectively. experimental results show that retriever achieves high accuracy on fewshot target domains without retraining on the target data. for example, it outperforms the strongest baseline by 4.45% on snips for the slot-filling task. ## benefits of retriever in addition to being more robust against overfitting and catastrophic forgetting problems, which are essential in few-shot learning settings, our proposed method has multiple advantages overs strong baselines. for instance, if the scheme is changed or some prediction bugs need to be fixed, there is minimum re-training required. more importantly, compared to classification or prototypical networks which require adding arbitrary number of instances to the training data and hope that the model will predict as expected @xcite , retriever can guarantee the prediction when a similar query is encountered. at the same time, retriever is more interpretable where the retrieved examples can serve as explanations. in addition, different from the simplified assumption that one utterance may only have one intent @xcite , retriever can be used to predict multiple labels. lastly, because retriever does not need to model transition probability, the decoding procedure can be paralleled and potentially be modified to be non-autoregressive for speedup. we can also tune threshold (explained in section 5.2) to change precision and recall according to use case requirements. recently, @xcite showed that a simple nearest neighbor model with feature transformations can achieve competitive results with the state-of-the-art methods on image classification. inspired by their work, we train our retrieval-based model with a novel batch softmax objective. ## setup we consider two tasks where the input is an utterance x with tokens x 1 , . . . , x n and the output is some structure y. for the slot filling task, the output y is a set of non-overlapping labeled spans {(r i , ℓ i )} m @xmath0.g., "7 pm") and ℓ i is a slot name (e.g., time). for the intent classification task, the output y is simply an intent label ℓ for the whole utterance x. for notational consistency, we view intent classification as predicting a labeled span (r, ℓ) where @xmath1:n . in the few-shot setup, examples (x, y) are divided into source and target domains. examples in the target domain may contain some labels ℓ that are unseen in the source domain. the model will be given ample training data from the source domain, but only a few training examples from the target domain. for instance, the model receives only @xmath2. the model can be evaluated on test data from both domains. ## model we propose a retrieval-based model, retriever, for intent classification and slot filling in the fewshot setting. figure 1 illustrates our approach. at a high level, from examples (x, y) in the target training data (and optionally the source training data), we construct a retrieval index consisting of labeled spans (r, ℓ) from y. given a test utterance x, for each span of interest in x (all spans x i:j for slot filling; only x 1:n for intent classification), we retrieve the most similar labeled spans (r, ℓ) from the index, and then use them to decode an output y that maximizes the average span similarity score. the use of retrieval provides several benefits. for instance, we empirically show in section 7.1 that the model does not suffer from catastrophic forgetting because both source and target data are present in the retrieval index. class imbalance can also be directly mitigated in the retrieval index. additionally, since the trained model is nonparametric, we could replace the retrieval index to handle different target domains without having to retrain the model. this also means that the model does not need access to target data during training, unlike traditional classification methods. ## experiments and results we evaluate our proposed approach on two datasets: clinc @xcite for intent classification and snips @xcite for slot filling. note that we use max (eq. 4) as the reduction function for both tasks since it empirically yields the best results. the effect of reduction functions will be analyzed later in section 7.1. ## analysis 7.1 intent classification models without re-training the pre-train on src domain section in varying the support set at inference time the construction of the support set is critical to retrievalbased methods. in ## conclusion in this paper, we propose a retrieval-based method, retriever, for few-shot intent classification and slot filling. we conduct extensive experiments to compare different model variants and baselines, and show that our proposed approach is effective in the few-shot learning scenario. we believe that our method can also work on open domain dialog tasks where annotations may be more scarce and other text classification tasks. in the future, we plan to extend our method to predict more complex structures with span-based retrieval.