Semantic Slot Prediction on low corpus data using finite user defined list (2020)

Abstract: Semantic slot prediction is one of the important task for natural language understanding (NLU). They depend on the quality and quantity of the human crafted training data, which affects model generalization. With the advent of voice assistants exposing AI platforms to third party developers, training data quality and quantity matters for any machine learning algorithm to learn and generalize properly.AI platforms provides provision to add custom external plist defined by the developers for the training data. Hence we are exploring dataset, called LowCorpusSlotData, containing low corpus training data with larger number of slots and significant test data. We also use external plist for the above dataset to aid in slot identification. We experimented using state of the art architectures like Bi-directional Encoder Representations from Transformers (BERT) with variants and Bi-directional Encoder with Custom Decoder. To address the low corpus problem, we propose a pipeline approach where we extract candidate slot information using the external plist extractor module and feed as input along with utterance.

## introduction in recent market, many voice assistants like samsung bixby, amazon alexa are providing ai platforms for developers outside to develop their custom applications. in these platforms, developers add the data for their custom made applications which is being used by the voice assistant for the training its nlu components. the main responsibility of nlu is to understand the user utterance and to determine the domain, intent and slots. domain prediction extracts the domain in which the utterance belongs to i.e., the application that will execute the user intention. intent detector is to extract the user intention or the action that we will execute inside the application. slot filler is to ex-tract the semantic slots or objects of interest on which we execute the action. example is name entity finding like name of person, location and date time. the data, developed by the developers, might be small for training slot-detection task when compared to intent detection or domain prediction task. therefore, the current scope of this paper is limited solving the slot filler task. to explain slot filler task with an example, consider the utterance "is the barbeque nation closed at this time?". slot filler task is to identify "barbeque nation" as business-name and "closed" as openhourdescriptor from the utterance. lots of work is going on the field of slot prediction as independent task. @xcite proposes the architecture based on encoder-decoder attention model with aligned input where bi-gru as encoder and gru decoder which learns jointly both slot filling and delexicalized sentence generation. @xcite proposes the variants of lstm and gru integrated with the crf layer at end for the task of slot filling. @xcite recommends the usage of rnn for the slot prediction. @xcite performs an experiment using a multi-task model with open vocabulary embeddings increases the generalizability by which the data required for the training the slot-filling is minimalised. @xcite proposes a various lstm-crf models for sentence tagging. @xcite proposes the encoder-labeler lstm which performs slot filling conditioned on the encoded sentence-level information which was generated by lstm. @xcite recommends a recurrent support vector machine, which is a combination of recurrent neural network, and a structured support vector machine for the slot tagging. @xcite proposes a cross-domain slot filling using bi-lstm for handling the limitation of data and unseen slot types. @xcite proposes a capsule based neural network model, which accomplishes slot filling and intent detection via a dynamic routing-by-agreement schema. firdaus et al. (2019 ) recommends a multi-task hierarchical approach using the cnn, rnn to get the contextual information and uses crf for model label dependency. all the above approaches suffers from the lack of intent information to enhance slot task. to circumvent the above limitation, exploration on slot prediction as joint task with intent gained momentum. @xcite proposes a new architecture for joint intent detection and slot filling based on pre-trained bert @xcite , added the self-attention and slot gate with crf which improvement in slot filling. gangadharaiah and narayanaswamy (2019) proposes attention information (calculating the attention of current encoder with respect to previous encoders), in addition to encoder during each decoder step for predicting joint intent and slot. inspired by performance of the above state of the art architectures in slot prediction work, we are exploring architectures namely bert @xcite and encoder with custom decoder (gangadharaiah and narayanaswamy, 2019) on open source dataset with external list information. we organize rest of the paper as follows. section 2 describes the proposed approach. section 3 describes the experimental setup including dataset, metrics used followed by results. finally, we conclude and suggest future work and extensions. ## results and analysis ## conclusion this work demonstrated the use of external slot information along with sentence. we showed the performance of dnn models on low corpus data with external plist and showed there is an improvement of 13.49%, by bi-directional encoder with custom decoder when compared to state of the art bert model. we believe the pipeline approach to such user-developed dataset will aid in better model generalization for semantic slot prediction. future scope of the paper includes exploration of non-sequential models for sequence labelling task. also we are planning to extend the work to predict domain, intent along with slot prediction.