Learning to Tag OOV Tokens by Integrating Contextual Representation and Background Knowledge (2020)

Abstract: Neural-based context-aware models for slot tagging have achieved state-of-the-art performance. However, the presence of OOV(out-of-vocab) words significantly degrades the performance of neural-based models, especially in a few-shot scenario. In this paper, we propose a novel knowledge-enhanced slot tagging model to integrate contextual representation of input text and the large-scale lexical background knowledge. Besides, we use multi-level graph attention to explicitly model lexical relations. The experiments show that our proposed knowledge integration mechanism achieves consistent improvements across settings with different sizes of training data on two public benchmark datasets.

## introduction slot tagging is a critical component of spoken language understanding(slu) in dialogue systems. it aims at parsing semantic concepts from user utterances. for instance, given the utterance "i'd also like to have lunch during my flight" from the atis dataset, a slot tagging model might identify lunch as a meal description type. given sufficient training data, recent neural-based models @xcite @xcite @xcite have achieved remarkably good results. however, these works often suffer from poor slot tagging accuracy when rare words or oov( out-of-vocab) words exist. @xcite has verified the presence of oov words further degrades the performance of neural-based models, especially in a few-shot scenario where training data can not provide adequate contextual semantics. previous context-aware models merely focus on how to capture deep contextual semantics to aid * weiran xu is the corresponding author. in recognizing slot entities, while neglecting ontology behind the words or large-scale background knowledge. explicit lexical relations are vital to recognizing unseen words when there is not adequate training data, that is, few-shot scenarios. fig 1 gives a motivating example of slot tagging to explain the phenomenon. this example suggests slot tagging requires not only understanding the complex linguistic context constraints but also reasoning explicit lexical relations via large-scale background knowledge graphs. previous state-of-the-art context-aware models @xcite only learn contextual information based on a multi-layer bil-stm encoder and self-attention layer. @xcite @xcite use handcrafted lexicons (also known as gazettes or dictionaries), which are typically collections of phrases semantically related, to improve slot tagging. one major limitation is that lexicons collected by domain experts are relatively small on the scale and fail to model complicated relations between words, such as relation hierarchy. in this paper, we propose a novel knowledgeenhanced method for slot tagging by integrating contextual representation of input text and the largescale lexical background knowledge, enabling the model to reason explicit lexical relations. we aim to leverage both linguistic regularities covered by deep lms and high-quality knowledge derived from curated kbs. consequently, our model could infer rare and unseen words in the test dataset by incorporating contextual semantics learned from the training dataset and lexical relations from ontology. as depicted in fig 2 , given an input sequence, we first retrieve potentially relevant kb entities and encode them into distributed representations that describe global graph-structured information. then we employ a bert @xcite encoder layer to capture context-aware representations of the sequence and attend to the kb embeddings using multi-level graph attention. finally, we integrate bert embeddings and the desired kb embeddings to predict the slot type. our main contributions are three-fold: (1) we investigate and demonstrate the feasibility of applying lexical ontology to facilitate recognizing oov words in the few-shot scenario. to the best of our knowledge, this is the first to consider the large-scale background knowledge for enhancing context-aware slot tagging models. (2) we propose a knowledge integration mechanism and use multi-level graph attention to model explicit lexical relations. (3) plenty of experiments on two benchmark datasets show that our proposed method achieves consistently better performance than various state-of-theart context-aware methods. ## our approach in this work, we consider the slot tagging task in the few-shot scenario, especially for oov tokens. given a sequence with n tokens @xmath0, our goal is to predict a corresponding tagging sequence @xmath1. this section first explains our bert-based model and then introduces the proposed knowledge integration mechanism for inducing background commonsense. the overall model architecture is illustrated in fig 2 . ## experiments 3.1 setup datasets to evaluate our approach, we conduct experiments on two public benchmark datasets, atis @xcite and snips @xcite . atis contains 4,478 utterances in the training set and 893 utterances in the test set, while snips contains 13,084 and 700 utterances, respectively. the percentage of oov words between the training and test datasets is 0.77%(atis) and 5.95%(snips). samples in snips are from different topics, such as getting weather and booking a restaurant, resulting in a larger vocabulary. by contrast, samples in atis are all about flight information with similar vocabularies across them. therefore, snips is much more complicated, mainly due to data diversity and the large vocabulary. the full statistics are shown in the to simulate the few-shot scenarios, we downsample the original training sets of atis and snips to different extents while keeping valid and test sets fixed. we aim to evaluate the effectiveness of integrating external kb under the settings of varied sizes of training data available. evaluation we evaluate the performance of slot tagging using the f1 score metric. in the experiments, we use the english uncased bert-base model, which has 12 layers, 768 hidden states, and 12 heads. the hidden size for the bilstm layer is set to 128. adam (kingma and ba, 2014) is used for optimization with an initial learning rate of 1e-5. the dropout probability is 0.1, and the batch size is 64. we finetune all hyperparameters on the valid set. ## conclusion we present a novel knowledge integration mechanism of incorporating background kb and deep contextual representations to facilitate the few-shot slot tagging task. experiments confirm the effectiveness of modeling explicit lexical relations, which has not yet been explored by previous works. moreover, we find that our method delivers more benefits to data scarcity scenarios. we hope to provide new guidance for the future slot tagging work.