I n F o BERT : Zero-Shot Approach to Natural Language Understanding Using Contextualized Word Embedding (2021)

Abstract: Natural language understanding is an important task in modern dialogue systems. It becomes more important with the rapid extension of the dialogue systems’ functionality. In this work, we present an approach to zero-shot transfer learning for the tasks of intent classification and slot-filling based on pre-trained language models. We use deep contextualized models feeding them with utterances and natural language descriptions of user intents to get text embeddings. These embeddings then used by a small neural network to produce predictions for intent and slot probabilities. This architecture achieves new state-of-the-art results in two zero-shot scenarios. One is a single language new skill adaptation and another one is a cross-lingual adaptation.

## introduction dialogue systems become more and more popular in everyday life. a dialogue system has two main language understanding tasks which are of key importance for any skill: user intent recognition and information extraction from user input. the former task traditionally is interpreted as a classification problem, since the possible intents list assumed to be given. the latter task is formulated as slot filling, i.e. extraction of a text span considered to be a slot value. with dialogue systems becoming more popular their functionality is also growing in terms of both skills available to use and languages their functionality is accessible in. the rapid growth entails lack of usage data for the new skills and new languages and causes the lists of intents and slots to be dynamic thus make classical approaches inapplicable. we consider these newly emerged scenarios and demonstrate that our approach is well suited for both. in the first scenario, there are several known domains and a few unknown domains. the second scenario contains skills with training data for one language (english) and only test data for two other languages @xcite . in this paper, we present a new architecture which is based on pre-trained language models, like bert @xcite , and uses a small "head" (i.e. a neural network using the embeddings from the language models as input) evaluated in two scenarios: zero-shot single language natural language understanding on schema-guided dialog dataset @xcite and zero-shot cross-lingual natural language understanding on multi-lingual task-oriented dialog dataset @xcite . for the schema-guided dialog dataset, there is a baseline described alongside the dataset which is evaluated in the zero-shot transfer learning setup. it is using deep contextualized embedding model (bert) to embed the utterances and natural language descriptions for the slots and intents. another approach was presented in @xcite , where the authors present a system based on the baseline with an addition of handcrafted context features. our model is also close to this baseline with some important differences (described in section devoted to our approach) allowing it to outperform both previously presented approaches. for the multi-language task-oriented dialog dataset there are also published baselines in zero-shot cross-lingual transfer setup described alongside the dataset. the authors train a whole machine translation system to make embeddings of utterances and use them for prediction in both tasks. another approach proposed in @xcite , where the authors use combination of recurrent neural network and latent variable model. also an approach was presented in @xcite . the authors use bert model to embed utterances and use special context gate for prediction in both tasks. we also use a pre-trained language-model to embed natural language descriptions of slots and intents and to achieve state of the art results in the zero-shot scenario. formal contribution of this paper is four-fold: we present a new model architecture for (1) zero-shot intent classification and (2) zero-shot slot filling which with usage of appropriate embedding model achieves a new state of the art results on (3) crossskill and cross-domain transfer and on (4) crosslanguage transfer. ## related work there are several baseline systems for mtd which we describe here, while baseline systems for sgd are described alongside our model in the specific section below. one baseline system (multi. cove) presented in @xcite is based on deep contextualized embedding model cove @xcite . this system is using bidirectional lstm @xcite over cove embeddings with crf for slot prediction and an attention mechanism for intent recognition. this system has a variant with usage of an autoencoder for produced embedding refinement (multi. cove w/ auto). @xcite are using the same design as multi. cove with other embedding models, namely multilingual bert and xlm (lample and conneau, 2019), this approach is called transformer-mlt. in this work is also presented adaptation of rcsls model @xcite used as an embedding one, which aligns word embeddings with help of retrival loss (rcsls-mlt). the authors of @xcite are using slightly different approach. their system is using crosslingually aligned word embeddings, they are fed into bidirectional lstm, which output is used by attention mechanism for intent recognition and by a latent variable model for joint slot and intent prediction. in the work @xcite also presented another baseline called translate train. this system is using direct alignment of word tokens to fed this alignment into crf for prediction. there are models which use contextualized representations for natural language understanding in different setups. an approach to slot filling called zero-shot adaptive transfer described in @xcite . a model takes an utterance and natural language description of a slot and produces bioencoding of the utterance, i.e. points to the span, containing the value. to contextualize the word representations authors use bidirectional recurrent neural network. another approach is presented in @xcite , where the authors use multilin-gual bert as embedding model and predict jointly slots and intents with additional refinement task of data source prediction. deep contextualized embeddings, especially bert, have been used for intent classification and slot filling tasks in other setups, e.g. @xcite @xcite , where authors consider classic setup with shared slots and intents for train and test. there were also other works focused on zeroshot modeling @xcite @xcite , domain adaptation and transfer learning techniques @xcite in recent years. deep learning based approaches have achieved state of the art performance on dialogue state tracking tasks. popular approaches on smallscale datasets estimate the dialogue state as a distribution over all possible slot-values @xcite or individually score all slot-value combinations @xcite . such approaches are not practical for deployment in virtual assistants operating over real-world services having a very large and dynamic set of possible values. addressing these concerns, approaches utilizing a dynamic vocabulary of slot values have been proposed @xcite @xcite . ## task description and datasets we consider two independent scenarios in this work, although they share an important feature: zero-shot transfer learning. the first scenario is a cross-skill and cross-domain transfer which is evaluated on schema-guided dialog (sgd) dataset. it is formulated as follows: in each domain, there are one or more skills. each skill has its intents and slots described. also a skill could share an intent and/or a slot with another skill, but in general case, they are not sharing anything. to complicate things, in the dataset there is a domain (alarm) which is presented only in the development set, but not in the train set. the second scenario is a cross-lingual transfer and is evaluated on multi-language task-oriented dialog dataset (mtd) dataset. each skill presented in the dataset is described (to a reasonable extent) identically in all three languages. so a system could be trained on one language and evaluated on the other two languages. ## instruction following bert we think of the natural language descriptions of intents and slots as instructions for a model to follow in order to achieve a result, which is either intent classification or slot filling. we base our approach on bert-like models, especially their contextualization ability, and we show that these models could solve both parts with a small help of an additional simple module. we add to the pre-trained language-models a recurrence module for intent classification task and a feed-forward module for slot filling one. we call our approach instruction following bert (infobert). ## experiments we conduct experiments on two datasets, described in the section devoted to the datasets. for both sgd and mtd datasets we measure quality for tasks of intent classification and non-categorical slots extraction in the zero-shot scenario. to measure the intent classification quality, we use accuracy metric, due to a model has a dynamic list of intents to choose from. to measure the slot extraction quality we use f1 metric, which allows us to measure both single-dom. ## conclusion in this work we presented a model architecture which allows us to use different embedding models as a building block. this architecture is demonstrated to be effective in two zero-shot transfer tasks, namely cross-domain and cross-lingual. our model using an appropriate embedding model (tod-bert for cross-domain task and several multi-lingual models for cross-lingual one) shows state of the art performance on the intent recognition and slot filling tasks. it is interesting to mention that in our experiments we found that the best results for spanish and thai languages in the cross-lingual transfer task are not achievable at the same time. we suppose that this fact could be explained by the errors in the tokenization model used for most of the multi-lingual models. thus usage of language-agnostic bert sentence embedding model as an embedding one allows our system to outperform all the other systems on spanish language data, but broken tokenization in this model does not allow to show any improvement for thai language. as future work authors consider the study of other bert descendant models, which are plenty nowadays. another direction of the work is research in low resource scenario, when some data is available for a model to tune onto. in closing, we hope that this work will facilitate the research in the direction of transfer learning in dialogue systems.