Enriched Pre-trained Transformers for Joint Slot Filling and Intent Detection (2023)

Abstract: Detecting the user’s intent and finding the corresponding slots among the utterance’s words are important tasks in natural language understanding. Their interconnected nature makes their joint modeling a standard part of training such models. Moreover, data scarceness and specialized vocabularies pose additional challenges. Recently, the advances in pre-trained language models, namely contextualized models such as ELMo and BERT have revolutionized the field by tapping the potential of training very large models with just a few steps of fine-tuning on a task-specific dataset. Here, we leverage such models, and we design a novel architecture on top of them. Moreover, we propose an intent pooling attention mechanism, and we reinforce the slot filling task by fusing intent distributions, word features, and token representations. The experimental results on standard datasets show that our model outperforms both the current non-BERT state of the art as well as stronger BERT-based baselines.

## introduction with the proliferation of portable devices, smart speakers, and the evolution of personal assistants, such as amazon alexa, apple siri, google assistant, a need for better natural language understanding (nlu) has emerged. moreover, many web platforms and applications that interact with the users depend on the abilities of an internal nlu component, e.g., customer service with social media @xcite , in dialogue systems in general @xcite , for web queries understanding @xcite , and general understanding of natural language interaction @xcite . the major challenges such systems face are (i) finding the intention behind the user's request, and (ii) gathering the necessary information to complete it via slot filling, while (iii) engaging in a dialogue with the user. the slot filling task naturally arises as a sequence tagging task. conventional neural network architectures, such as rnns or cnns are appealing approaches to tackle this problem. various extensions thereof can be found in previous work @xcite @xcite @xcite . moreover, sequence tagging approaches such as maximum entropy markov model (memm) @xcite and conditional random fields (crf) @xcite @xcite have been added on top to enforce better modeling of the dependencies between the posteriors for the slot filling task. recent work has introduced other methods such as hierarchical structured capsule networks @xcite , and graph interactive networks @xcite . in this work, we investigate the usefulness of pre-trained models for the natural language understanding (nlu). our approach is based on bert @xcite and its successor roberta @xcite . that model offer two main advantages over previous ones @xcite @xcite @xcite : (i) they are we further adapt the pre-trained models for the nlu tasks. for the intent, we introduce a pooling attention layer, which uses a weighted sum of the token representations from the last language modelling layer. moreover, we reinforce the slot representation with the predicted intent distribution, and word features such as predicted word casing, and named entities. to demonstrate its effectiveness, we evaluate it on two publicly available datasets: atis @xcite and snips @xcite . our contributions can be summarized as follows: ## transformer-nlu we propose a joint approach for intent classification and slot filling built on top of a pre-trained language model. we further improve the base model in three ways: (i) for intent detection, we obtain a pooled representation from the last hidden states for all tokens (section 2.1), (ii) we obtain predictions for the word case and named entities for each token (word features), and (iii) we feed the predicted intent distribution vector, bert's last hidden representations, and word features into a slot filling layer (see section 2.2). the complete architecture of the model is shown in figure 1b . ## experimental setup dataset in our experiments, we use two publicly available datasets, the airline travel information system (atis) @xcite , and snips @xcite . the atis dataset contains transcripts from audio recordings of flight information requests, while the snips dataset is gathered by a custom intent engine for personal voice assistants. albeit both are widely used in nlu benchmarks, atis is substantially smaller -almost three times in terms of examples, and it contains fifteen times less words. however, it has a richer set of labels, 21 intents and 120 slot categories, as opposed to the 7 intents and 72 slots in snips. another key difference is the diversity of domains -atis has only utterances from the flight domain, while snips covers various subjects, including entertainment, restaurant reservations, weather forecasts, etc. (see a comparison between the two datasets is shown in ## experiments and analysis evaluation results while models become more accurate, the absolute difference between two experiments becomes smaller and smaller, thus a better measurement is needed. hereby, we introduce a fine-grained measure, i.e., relative error reduction (rer) percentage, which is defined as the proportion of absolute error reduced by a model a compared to model b . in absolute terms, we have 0.76 for intent, 4.84 for sentence, and 1.77 for slots (see ## conclusion we studied the two main challenges in natural language understanding, i.e., intent detection and slot filling. addressing these tasks is important in a number of scenarios arising on web platforms and web-based applications such as customer service in social media, dialogue systems, web queries understanding, and general understanding of natural language interaction with the user. in particular, we proposed an enriched pretrained language model to jointly model the two tasks (i.e., intent detection and slot filling), i.e., transformer-nlu. we designed a pooling attention layer in order to obtain intent representation beyond just the pooled one from the special start token. further, we reinforced the slot filling with word-specific features, and the predicted intent distribution. our experiments on two standard datasets showed that transformer-nlu outperforms other alternatives for all standard measures used to evaluate nlu tasks. we found that the use of roberta and adding a crf layer on top of the slot filling network did not help.