Learning to Classify Intents and Slot Labels Given a Handful of Examples (2020)

Abstract: Intent classification (IC) and slot filling (SF) are core components in most goal-oriented dialogue systems. Current IC/SF models perform poorly when the number of training examples per class is small. We propose a new few-shot learning task, few-shot IC/SF, to study and improve the performance of IC and SF models on classes not seen at training time in ultra low resource scenarios. We establish a few-shot IC/SF benchmark by defining few-shot splits for three public IC/SF datasets, ATIS, TOP, and Snips. We show that two popular few-shot learning algorithms, model agnostic meta learning (MAML) and prototypical networks, outperform a fine-tuning baseline on this benchmark. Prototypical networks achieves significant gains in IC performance on the ATIS and TOP datasets, while both prototypical networks and MAML outperform the baseline with respect to SF on all three datasets. In addition, we demonstrate that joint training as well as the use of pre-trained language models, ELMo and BERT in our case, are complementary to these few-shot learning methods and yield further gains.

## introduction in the context of goal-oriented dialogue systems, intent classification (ic) is the process of classifying a user's utterance into an intent, such as book-flight or addtoplaylist, referring to the user's goal. while slot filling (sf) is the process of identifying and classifying certain tokens in the utterance into their corresponding labels, in a manner akin to named entity recognition (ner). however, in contrast to ner, typical slots are particular to the domain of the dialogue, such as music or travel. as a reference point, we list intent and slot label annotations for an example utterance from the snips dataset with the addtoplaylist ic in figure 1 . as of late, most state-of-the-art ic/sf models are based on feed-forward, convolutional, or recurrent neural networks @xcite @xcite . these neural models offer substantial gains in performance, but they often require a large number of labeled examples (on the order of hundreds) per intent class and slot-label to achieve these gains. the relative scarcity of large-scale datasets annotated with intents and slots prohibits the use of neural ic/sf models in many promising domains, such as medical consultation, where it is difficult to obtain large quantities of annotated dialogues. accordingly, we propose the task of few-shot ic/sf, catering to domain adaptation in low resource scenarios, where there are only a handful of annotated examples available per intent and slot in the target domain. to the best of our knowledge, this work is the first to apply the few-shot learning framework to a joint sentence classification and sequence labeling task. in the nlp literature, fewshot learning often refers to a low resource, cross lingual setting where there is limited data available in the target language. we emphasize that our definition of few-shot ic/sf is distinct in that we limit the amount of data available per target class rather than target language. few-shot ic/sf builds on a large body of existing few-shot classification work. drawing inspiration from computer vision, we experiment with two prominent few shot image classification approaches, prototypical networks and model agnostic meta learning (maml). both these methods seek to decrease over-fitting and improve generalization on small datasets, albeit via different mechanisms. prototypical networks learns class specific representations, called prototypes, and performs inference by assigning the class label associated with the prototype closest to an input embedding. whereas maml modifies the learning objective to optimize for pre-training representations that transfer well when fine-tuned on a small number of labeled examples. for benchmarking purposes, we establish fewshot splits for three publicly available ic/sf datasets: atis @xcite , snips @xcite , and top @xcite . empirically, prototypical networks yields substantial improvements on this benchmark over the popular "fine-tuning" approach @xcite , where representations are pre-trained on a large, "source" dataset and then fine-tuned on a smaller, "target" dataset. despite performing worse on intent classification, maml also achieves gains over "fine-tuning" on the slot filling task. orthogonally, we experiment with the use of two pre-trained language models, bert and elmo, as well as joint training on multiple datasets. these experiments show that the use of pre-trained, contextual representations is complementary to both methods. while prototypical networks is uniquely able to leverage joint training to consistently boost slot filling performance. in summary, our primary contributions are fourfold: ## few-shot ic/sf benchmark as there is no existing benchmark for few-shot ic/sf, we propose few-shot splits for the air travel information system @xcite ), snips @xcite , and task oriented parsing (top, @xcite ) datasets. a few-shot ic/sf benchmark is beneficial for two reasons. firstly, the benchmark evaluates generalization across multiple domains. secondly, researchers can combine these datasets in the future to experiment with larger settings of n-way during training and evaluation. ## conclusion we show that few-shot learning techniques can substantially improve ic/sf performance in ultra low resource scenarios. specifically, our extension of prototypical networks for joint ic and sf consis-tently outperforms a fine-tuning baseline with respect to both ic accuracy and slot f1 score. moreover, we establish a benchmark for few-shot ic/sf to support future work on this important topic. our contribution is a step toward the creation of more sample efficient ic/sf models. yet there is still considerable work to be done in pursuit of this goal. in particular, we encourage the creation of larger few-shot ic/sf benchmarks to test how few-shot learning algorithms scale with larger episode sizes. split ic name # utt # sl sl names (without ic prefix) train atis flight 4298 71 or, mod, meal, flight, economy, connect, day name, city name, round trip, class type, flight mod, compartment, flight stop, flight days, flight time, fare amount, airline name, airline code, airport name, airport code, cost relative, aircraft code, flight number, period of day, toloc.city name, arrive time.time, meal description, toloc.state name, depart time.time, toloc.state code, depart date.year, fare basis code, fromloc.city name, stoploc.city name, fromloc.state name, toloc.airport name, fromloc.state code, toloc.country name, toloc.airport code, stoploc.state code, fromloc.airport name, fromloc.airport code, stoploc.airport name, stoploc.airport code, depart date.day name, arrive date.day name, depart time.end time, arrive time.end time, return date.day name, arrive date.month name, arrive date.day number, depart date.month name, depart date.day number, depart time.start time, arrive time.start time, depart time.period mod, arrive time.period mod, return time.period mod, return date.month name, return date.day number, arrive time.time relative, depart time.time relative, depart date.date relative, return date.date relative, arrive date.date relative, depart time.period of day, depart date.today relative, arrive date.today relative, arrive time.period of day, return time.period of day, return date.today relative train atis capacity 37 5 mod, airline name, aircraft code, toloc.city name, fromloc.city name train atis flight no 20 22 or, flight mod, class type, flight time, airline name, cost relative, flight number, toloc.city name, arrive time.time, toloc.state name, depart time.time, toloc.state code, fromloc.city name, stoploc.city name, fromloc.state name, depart date.day name, depart date.month name, depart date.day number, arrive time.time relative, depart time.time relative, depart time.period of day, depart date.today relative train atis meal 12 12 meal, airline name, airline code, flight number, toloc.city name, arrive time.time, toloc.state code, meal description, fromloc.city name, toloc.airport code, depart date.day name, depart time.period of day train atis restriction 6 6 round trip, fare amount, cost relative, toloc.city name, restriction code, fromloc.city name dev atis airfare 471 45 or, meal, economy, connect, round trip, class type, flight mod, fare amount, flight stop, flight time, flight days, airline name, airline code, cost relative, flight number, aircraft code, toloc.city name, depart time.time, toloc.state code, toloc.state name, depart date.year, arrive time.time, fromloc.city name, stoploc.city name, toloc.airport name, fromloc.state name, toloc.airport code, fromloc.state code, fromloc.airport code, fromloc.airport name, depart date.day name, arrive date.day name, arrive date.month name, arrive date.day number, depart date.month name, depart date.day number, return date.month name, return date.day number, depart time.period mod, depart time.time relative, depart date.date relative, arrive time.time relative, arrive date.date relative, depart time.period of day, depart date.today relative dev atis flight time 55 20 flight mod, class type, flight time, airline name, airline code, airport name, flight number, aircraft code, toloc.city name, depart time.time, meal description, fromloc.city name, toloc.airport code, fromloc.airport name, depart date.day name, depart date.month name, depart date.day number, depart date.date relative, depart time.time relative, depart time.period of day dev atis quantity 54 25 economy, city name, class type, round trip, flight stop, flight days, airline code, airline name, flight number, aircraft code, toloc.city name, arrive time.time, fare basis code, depart time.time, fromloc.city name, stoploc.city name, toloc.airport name, arrive date.month name, arrive date.day number, depart date.month name, depart date.day number, arrive time.time relative, depart time.time relative, depart date.today relative, depart time.period of day dev atis distance 30 8 city name, airport name, toloc.city name, depart time.time, fromloc.city name, fromloc.airport name, depart date.month name, depart date.day number dev atis city 25 11 city name, class type, airline name, airport code, airport name, toloc.city name, depart time.time, fromloc.city name, fromloc.airport code, depart time.time relative, depart time.period of day dev atis ground fare 25 6 city name, airport name, transport type, toloc.city name, fromloc.city name, fromloc.airport name dev atis airline;atis flight no 2 7 toloc.city name, arrive time.time, fromloc.city name, depart date.month name, depart date.day number, depart date.date relative, arrive time.time relative test atis ground service 291 23 or, time, day name, city name, state code, state name, month name, day number, flight time, airport name, airport code, time relative, transport type, today relative, period of day, toloc.city name, fromloc.city name, toloc.airport name, fromloc.airport name, depart date.day name, depart date.month name, depart date.day number, depart date.date relative test atis airline 195 36 mod, connect, city name, class type, round trip, flight stop, flight days, airline code, airport name, airline name, aircraft code, flight number, cost relative, toloc.city name, toloc.state code, depart time.time, arrive time.time, toloc.state name, fromloc.city name, stoploc.city name, toloc.airport name, fromloc.state code, fromloc.airport code, fromloc.airport name, depart date.day name, depart time.end time, arrive date.month name, arrive date.day number, depart date.month name, depart date.day number, depart time.start time, depart time.time relative, depart date.date relative, depart date.today relative, depart time.period of day, arrive time.period of day test atis abbreviation 180 14 mod, meal, meal code, days code, class type, airport code, airline code, airline name, aircraft code, booking class, toloc.city name, fare basis code, restriction code, fromloc.city name test atis aircraft 90 23 mod, city name, class type, flight mod, airline name, airline code, flight number, aircraft code, toloc.city name, depart time.time, toloc.state code, arrive time.time, fromloc.city name, stoploc.city name, depart date.day name, arrive date.day name, depart date.month name, depart date.day number, arrive date.month name, arrive date.day number, depart time.time relative, arrive time.time relative, depart time.period of day test atis airport 38 9 mod, city name, state code, state name, flight stop, airport name, airline name, toloc.city name, fromloc.city name test atis flight;atis airfare 33 21 flight mod, round trip, fare amount, flight stop, airline name, flight number, cost relative, toloc.city name, depart time.time, arrive time.time, toloc.state code, fromloc.city name, depart date.day name, return date.day name, depart date.month name, depart date.day number, depart time.time relative, depart date.date relative, arrive time.time relative, return date.date relative, depart time.period of day test atis day name 2 2 toloc.city name, fromloc.city name