Improving Slot Filling by Utilizing Contextual Information (2020)

Abstract: Slot Filling (SF) is one of the sub-tasks of Spoken Language Understanding (SLU) which aims to extract semantic constituents from a given natural language utterance. It is formulated as a sequence labeling task. Recently, it has been shown that contextual information is vital for this task. However, existing models employ contextual information in a restricted manner, e.g., using self-attention. Such methods fail to distinguish the effects of the context on the word representation and the word label. To address this issue, in this paper, we propose a novel method to incorporate the contextual information in two different levels, i.e., representation level and task-specific (i.e., label) level. Our extensive experiments on three benchmark datasets on SF show the effectiveness of our model leading to new state-of-the-art results on all three benchmark datasets for the task of SF.

## introduction slot filling (sf) is the task of identifying the semantic constituents expressed in a natural language utterance. it is one of the sub-tasks of spoken language understanding (slu) and plays a vital role in personal assistant tools such as siri, alexa, and google assistant. this task is formulated as a sequence labeling problem. for instance, in the given sentence "play signe anderson chant music that is newest.", the goal is to identify "signe anderson" as "artist", "chant music" as "music-item" and "newest" as "sort". early work on sf has employed feature engineering methods to train statistical models, e.g., conditional random field @xcite . later, deep learning emerged as a promising approach for sf @xcite @xcite . the success of deep models could be attributed to pre-trained word embeddings to generalize words and deep learning architectures to compose the word embeddings to induce effective representations. in addition to improving word representation using deep models, @xcite showed that incorporating the context of each word into its representation could improve the results. concretely, the effect of using context in word representation is two-fold: (1) representation level: as the meaning of the word is dependent on its context, incorporating the contextual information is vital to represent the true meaning of the word in the sentence (2) task level: for sf, the label of the word is related to the other words in the sentence and providing information about the other words, in prediction layer, could improve the performance. unfortunately, the existing work employs the context in a restricted manner, e.g., via attention mechanism, which obfuscates the model about the two aforementioned effects of the contextual information. in order to address the limitations of the prior work to exploit the context for sf, in this paper, we propose a multi-task setting to train the model. more specifically, our model is encouraged to explicitly ensure the two aforementioned effects of the contextual information for the task of sf. in particular, in addition to the main sequence labeling task, we introduce new sub-tasks to ensure each effect. firstly, in the representation level, we enforce the consistency between the word representations and their context. this enforcement is achieved via increasing the mutual information (mi) between these two representations. secondly, in the task level, we propose two new sub-tasks: (1) to predict the label of the word solely from its context and (2) to predict which labels exist in the given sentence in a multi-label classification setting. by doing so, we encourage the model to encode task-specific features in the context of each word. our extensive experiments on three benchmark datasets, empirically prove the effectiveness of the proposed model leading to new the state-of-the-art results on all three datasets. ## related work in the literature, slot filling (sf), is categorized as one of the sub-tasks of spoken language understanding (slu). early work employed feature engineering for statistical models, e.g., conditional random field @xcite . due to the lack of generalisation ability of feature based models, deep learning based models superseded them @xcite @xcite . also, joint models to simultaneously predict the intent of the utterance and to extract the semantic slots has also gained a lot of attention @xcite @xcite @xcite @xcite . in addition to the supervised settings, recently other setting such as progressive learning @xcite or zero-shot learning has also been studied @xcite . to the best of our knowledge, none of the existing work introduces a multi-task learning solely for the sf to incorporate the contextual information in both representation and task levels. ## model our model is trained in a multi-task setting in which the main task is slot filling to identify the best possible sequence of labels for the given sentence. in the first auxiliary task we aim to increase consistency between the word representation and its context. the second auxiliary task is to enhance task specific information in contextual information. in this section, we explain each of these tasks in more details. ## conclusion in this work, we introduced a new deep model for the task of slot filling (sf). in a multi-task setting, our model increases the mutual information between the word representation and its context, improves label information in the context and predicts which concepts are expressed in the given sentence. our experiments on three benchmark datasets show the effectiveness of our model by achieving the state-of-the-art results on all datasets for the sf task.