Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey (2020)

Abstract: In recent years, fostered by deep learning technologies and by the high demand for conversational AI, various approaches have been proposed that address the capacity to elicit and understand user’s needs in task-oriented dialogue systems. We focus on two core tasks, slot filling (SF) and intent classification (IC), and survey how neural based models have rapidly evolved to address natural language understanding in dialogue systems. We introduce three neural architectures: independent models, which model SF and IC separately, joint models, which exploit the mutual benefit of the two tasks simultaneously, and transfer learning models, that scale the model to new domains. We discuss the current state of the research in SF and IC, and highlight challenges that still require attention.

## introduction the ability to understand user's requests is essential to develop effective task-oriented dialogue systems. for example, in the utterance "i want to listen to hey jude by the beatles", a dialogue system should correctly identify that the user's intention is to give a command to play a song, and that hey jude and the beatles are, respectively, the song's title and the artist name that the user would like to listen. in a dialogue system this information is typically represented through a semantic-frame structure @xcite , and extracting such representation involves two tasks: identifying the correct frame (i.e. intent classification (ic)) and filling the correct value for the slots of the frame (i.e. slot filling (sf)). in recent years, neural-network based models have achieved the state of the art for a wide range of natural language processing tasks, including sf and ic. various neural architectures have been experimented on sf and ic, including rnn-based @xcite and attention-based @xcite approaches, till the more recent transformers models @xcite . input representations have also evolved from static word embeddings @xcite @xcite to contextualized word embeddings @xcite . such progress allows to better address dialogue phenomena involving sf and ic, including context modeling, handling out-of-vocabulary words, long-distance dependency between words, and to better exploit the synergy between sf and ic through joint models. in addition to rapid progresses in the research community, the demand for commercial conversational ai is also growing fast, shown by a variety of available solutions, such as microsoft luis, google dialogflow, rasa, and amazon alexa. these solutions also use various kinds of semantic frame representations as part of their framework. motivated by the rapid explosion of scientific progress, and by unprecedented market attention, we think that a guided map of the approaches on sf and ic can be useful for a large spectrum of researchers and practitioners interested in dialogue systems. the primary goal of the survey is to give a broad overview of recent neural models applied to sf and ic, and to compare their performance in the context of task-oriented dialogue systems. we also highlight and discuss open issues that still need to be addressed in the future. the paper is structured as follows: section 2 describes the sf and ic tasks, commonly used datasets and evaluation metrics. section 3, 4, and 5 elaborate on the progress and state of the art of independent, joint, and transfer learning models for both tasks. section 6 discusses the performance of existing models and open challenges. utterance i want to listen to hey jude by the beatles slot o o o o o b-song i-song o b-artist i-artist intent play song ## slot filling and intent classification this section provides some background relevant for sf and ic, sets the scope of the survey with respect to the context in dialogue systems, defines sf and ic as tasks, and introduces the datasets and the metrics that will be used in the rest of the paper. ## state of the art and beyond based on the results in portable and data efficient models. instead of evaluating models with the typical leaderboard setup with fixed (train/dev/test) splits on a specific target domain, it would be also important to test models in different scenarios, so that different aspects of the model can be captured. for example, as neural models are data hungry, more work is still needed on transfer learning scenarios, where evaluation is carried out with less or without labeled data (zero-shot) for a particular target domain. in addition, most models for sf and ic are evaluated on english, which means that more effort is still needed to make models that work well for other languages. some recent works have started exploring zero-shot cross lingual methods @xcite @xcite and also few-shot scenarios @xcite and the room for improvement for these scenarios is still large. in short, designing a data efficient model that is portable across domains and languages is still a challenging problem for the coming future. leveraging unlabeled data from live traffic. in real situations, personal digital assistants such as google home, apple siri and amazon alexa, receive live traffic data from real users. this large amount of unlabeled data from live traffic is a potential data source for model training, in addition to in-house annotated data. unlabeled live data are likely different from in-house data, as they can contain more diverse utterances and also noisy and irrelevant utterances. in this situation, existing methods to tap on unlabeled data, such as semi-supervised learning, still face unique challenges to handle live data. it is worth to note that a bottleneck in this direction is that working on live data in academic settings is not trivial. some recent works explore this line of research by applying semi-supervised learning @xcite and also data selection (do and gaspers, 2019) mechanism. generative models. most of the proposed models are discriminative, among the few works carried out for generative models for sf and ic, @xcite have shown that a generative model is relatively better than a discriminative model in a situation where data is scarce. one possible direction for generative models is to apply data augmentation to automatically create additional training data @xcite @xcite @xcite . the main challenge for data augmentation is to generate diverse and fluent synthetic utterance, which preserve the semantics of the original utterance. evaluation of sf and ic on more complex dataset. existing neural approaches typically evaluated on single-intent utterance, however in a real-world scenario users may indicate multiple-intent in an utterance e.g. "show me all flights from atlanta to london and get the cost" @xcite or even expressing multiple sentences in one single turn. while most datasets for slot filling and intent classification are single-turn utterance, there are some recent multi-turn datasets that provide slot annotation on the token-level, namely the restaurant-8k, taskmaster-1 and 2 @xcite , and frame @xcite datasets. the subset of schema guided dialogue (sgd) dataset @xcite used in dtsc-8 is also annotated with slots in the token-level and covers 16 domains. in addition to that, the top dataset @xcite introduces datasets annotated with hierarchical representation and mtop dataset @xcite provides both flat and hierarchical representation on 6 languages across 11 domains. ## conclusion we have surveyed recent neural-based models applied to sf and ic in the context of task-oriented dialogue systems. we examined three approaches, i.e. independent, joint, and transfer learning based models. joint models exploiting the relation between sf and ic simultaneously shown relatively better performance than independent models. empirical results have shown that most joint models nearly "solve" widely used datasets, atis and snips, given sufficient in-domain training data. nevertheless, there are still several challenges related to sf and ic, especially improving the scalability of the model to new domains and languages when limited labeled data are available.