A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings (2020)

Abstract: Learning-based slot filling - a key component of spoken language understanding systems - typically requires a large amount of in-domain hand-labeled data for training. In this paper, we propose a novel two-stage model architecture that can be trained with only a few in-domain hand-labeled examples. The first step is designed to remove non-slot tokens (i.e., O labeled tokens), as they introduce noise in the input of slot filling models. This step is domain-agnostic and therefore, can be trained by exploiting out-of-domain data. The second step identifies slot names only for slot tokens by using state-of-the-art pretrained contextual embeddings such as ELMO and BERT. We show that our approach outperforms other state-of-art systems on the SNIPS benchmark dataset.

## introduction slot filling models, which predict task-specific names (e.g. artist, time) for these slots from user utterances, are a key component of spoken language understanding (slu) systems. deep learning approaches @xcite @xcite @xcite @xcite for slu involve training on a large amount of annotated training data. likewise, multi-domain studies @xcite that rely on deep learning methods require a large amount of data for each domain. however, slot filling is a very challenging task if only a few labeled samples are available. therefore, this paper proposes methods to address the low-resource domain issue of slot filling. we aim at improving performance of the slot filling task in different low-resource scenarios by exploring the effective usage of a few in-domain samples with two different scenarios: (1) if data from other domains is not possible but a few samples are available in the current domain (2) if data from other domains are available and a few samples are accessible in the current domain. we exploit domain-agnostic syntactic similarities (e.g., the main verb of a sentence cannot be a slot) to learn the conceptual differences between slot and non-slot tokens in order to dismiss non-slot tokens from the input space. therefore, using labeled data (slot and o labels) across domains can improve the non-slot token reduction step in the target domain and thereby the slot name prediction step. therefore, we propose a novel two-stage model that first reduces this noise by adding a non-slot detection step and then predicts slot names. the identified non-slots are then removed from the input space of the name prediction step. our modeling approach is inspired by @xcite @xcite . we suggest using a few annotated samples as training input instead of slot descriptions and slot names as in zero-shot learning studies @xcite @xcite . this is for two reasons: (1) the creation of slot descriptions needs qualified linguistic expertise and is thus expensive. (2) the relationship between slot names and the corresponding tokens is not constant. to give an example, the relationship between the 'genre' slot name and 'drama' token is hypernymic whereas the relationship between the 'artist' slot name and 'tarkan' token is instance based. hence, it may not be valid to learn only one function to represent the different relationships between names and tokens. as a classification algorithm, we employ rocchio classification method @xcite for labeling the tokens with their domain specific name labels after reducing the non-slot tokens from the input. rocchio classifier is a very simple classification method that separates the inputs into centroids computed as the center of mass of all vectors in the class, i.e., builds a prototype vector for each class. decision process is simply made based on distance metrics. because of the availability of only a small amount of data in the current domain and the semantically rich and robust presentation in contextual pretrained embeddings, we argue that rocchio classifier is sufficient for our task. furthermore by using this simple classification method, we show the effectiveness of the non-slot noise reduction step from the input. ## model architecture we define our consecutive model structure as follows: given an utterance with t tokens, first we employ slot labeling model in order to identify slot tokens while eliminating the non-slot tokens of input utterance. consecutively, we predict the slot name of the slot tokens which are received from the slot labeling model. the figure 3 illustrates the overview of the consecutive model architecture with its inputs and outputs while showing the usage of contextualized word embeddings in order to represent input tokens. ## qualitative analysis we analyzed the results on individual slots by comparing them according to contextualized embeddings and proposed models. we observed that bert shows consistent lower results for the tokens like city, state from bookresteurant, and location name, object location type from search-screeninge whereas it outperforms elmo for proper name detection like object name from rate-book and searchcreativew. domains. the wrong predictions of name labeling, e.g., false-positive rates of names (e.g., object select, cuisine, spatial relation) for o label, draw the attention. an extreme difference between low precision and relatively high recall is observed. however, the precision results are drastically improved when rocchioslot+ and neuralslot+name models are employed. for example, ratebook domain's slot object select has 0.41 precision with name model whereas the precision of it is 0.69 and 0.93 with rocchioslot+ and neuralslot+name models respectively. on the other hand, when the timerange label of getweather is reviewed, rocchioslot+name as well as name model failed. due to leak of nonslot tokens, timerange values labeled as 'o'. roc-chioslot fails for labeling the values (e.g., eleven months from now) of timerange with s, because it is a clustering-based method and is not able to capture the sequential dependencies. neural-slot+name models, however, shows significant increases. the comparison of the results from both models indicates that the wrong predictions of the 'o' label drastically reduced with neural-slot+name model. similar proper nouns, e.g., album and track, in the same domain denote the weakness of the proposed systems. neuralslot+name model is not able to distinguish similar proper nouns. for example, the highest false-negative rate for album is track while it is album for track. ## conclusion and future work we propose a novel two-stage model for slot filling in low-resource domains. our results demonstrate the importance of non-slot token reduction on slot filling with resource constraints by using a simple classification method. furthermore, the benefit of employing slot filling data from other different domains for non-slot reduction is demonstrated. in addition, increasing sample sizes for the prototypes shows significant improvements. base on our findings, future usage of multi-domain or limited data could be effective in improving slot filling methods from a non-slot reduction perspective. additionally, the outcomes of the multi-domain data usage in our study contributes a new perspective in supervised domain adaptation and generalization studies.