Neighbor Contextual Information Learners for Joint Intent and Slot Prediction (2020)

Abstract: Intent Identification and Slot Identification aretwo important task for Natural Language Understanding(NLU). Exploration in this areahave gained significance using networks likeRNN, LSTM and GRU. However, modelscontaining the above modules are sequentialin nature, which consumes lot of resourceslike memory to train the model in cloud itself. With the advent of many voice assistantsdelivering offline solutions for manyapplications, there is a need for finding replacementfor such sequential networks. Explorationin self-attention, CNN modules hasgained pace in the recent times. Here we exploreCNN based models like Trellis and modifiedthe architecture to make it bi-directionalwith fusion techniques. In addition, we proposeCNN with Self Attention network calledNeighbor Contextual Information Projector usingMulti Head Attention (NCIPMA) architecture. These architectures beat state of the art inopen source datasets like ATIS, SNIPS.

## introduction intelligent voice assistant like samsung bixby, google assistant, amazon alexa and microsoft cortana are increasingly becoming popular. these assistants cater to the user request by extracting the user intention from the spoken utterance. nlu express the user intention in terms of intent label and slot tags. there is one intent label for entire utterance, which signifies unique action to execute. whereas slots are tags given to tokens in utterance, which signifies extra information required to execute the unique action. slot tags are denoted in ibo format. for example, consider the utterance "what flights are available from denver to san francisco". we denote intent label as "atis flight". we denote slot information as "o o o o o b-fromloc.city name o b-toloc.city name i-toloc.city name". there is one slot tag for every token in the utterance, where "o" represents that token is not a slot and b-fromloc.city name represents that token is "beginning of slot fromloc.city name" and "btoloc.city name i-toloc.city name" represents that corresponding tokens are beginning and continuation of slot toloc.city name respectively. we consider intent identification as classification task and slot tagging as sequence labelling task where we predict slot for each word. lots of work has happened in this area. initially, research community explored both intent and slot tasks as independent task. different techniques were explored for intent classification. @xcite created ensemble model by combining learnings of cnn, lstm and gru using multilayer perceptron to enhance intent classification. @xcite proposed enriched word embedding for making similar words together and dissimilar words farther, which aided intent detection better. @xcite proposed use of self-attention for enhancing intent classification by capturing long-range and multi-scale dependency in data. similarly for slot tagging, @xcite proposed use of label dependencies along with input sentence for enhancing slot learnings using lstm. @xcite suggested use of recurrent neural network for slot tagging task. ngoc thang vu (2016) proposed novel cnn architecture for slot tagging task, which also used past and future words information. these individual models approach resulted in pipelined design of nlu. where intent model will predict intent and they use intent output in slot model to predict slot. this resulted in error propagation i.e. error in intent output resulted in slot tagging errors. in addition, the intent and slot learnings are not available for each other to enhance each task learning. to circumvent the above limitation, research community started exploring unified model where they identify both intent classification and slot tagging. @xcite proposed attention as rnn encoder as input to decoder for predicting slot and intent by the decoder. @xcite proposed exploration of attention with bi-lstm for joint intent and slot prediction. @xcite suggested usage of cnn and rnn for contextual understanding of the utterance along with crf for label dependency to predict intent and slot. @xcite proposed intent pooling attention along with word features on top of bert for predicting intent and slot. the above approaches addresses both the task using single unified network where both the learnings are propagated to single network. further, exploration of parallel learning networks, using common base module, with fusing of intent and slot learnings have gained momentum. @xcite suggested slot gate mechanism to fuse the intent attention learning with slot attention learning to predict slot along with intent. @xcite proposed a novel iteration mechanism to fuse the meanings of intent and slot subnet for predicting intent and slot. inspired by the work on parallel learning networks, where we address the intent and slot tasks learnings by each parallel network, we are proposing a novel architecture, neighbor contextual importance projector (ncip) for learning the word importance in its immediate vicinity to boost its importance learning in the overall utterance. we use parallel multi head attention on top of ncip to project importance phrases for each task, to predict intent and slot. we are also exploring trellis network, which learns immediate neighbors in a layer and entire utterance in multiple such layers. in addition, the weights of the network is shared in both temporal direction as well as across layers. hence, we are proposing a bi-directional trellis network with different fusion techniques, like linear fusion, concatenation, to predict intent and slot to mimic bi-directional lstm or gru. we organize rest of the section as follows. section 2 describes the proposed approaches. section 3 describes the experimental setup including dataset, metrics used followed by results. finally, we conclude and suggest future work and extensions. as trellis network has structural and algorithmic elements from both lstm & cnn, it achieved state of art in various lm tasks. therefore, we are exploring extension of the same in intent determination and slot tagging. original trellis network process the input sequence in forward direction only. we propose bidirectional network with two parallel trellis network encoders, one for forward pass and one for backward pass, for intent determination and slot tagging. ## conclusion we are able to find a replacement for sequential learning models like lstm, gru and rnn by using cnn with self-attention. we are able to see that ncip module is able to project the importance of uni-gram, bi-gram and tri-gram well. trellis network based models worked well but further research is required with them to improve on intent classification and slot tagging tasks. future scopes are exploration of unified model to predict domain, intent and slot for the said task. exploration of impact of shared weights across layers for cnn with self-attention is a needed task to reduce size without impact in accuracy.