Unified Multi Intent Order and Slot Prediction using Selective Learning Propagation (2020)

Abstract: Natural Language Understanding (NLU) involves two important task namely Intent Determination(ID) and Slot Filling (SF). With recent advancements in Intent Determination and Slot Filling tasks, explorations on handling of multiple intent information in a single utterance is increasing to make the NLU more conversation-based rather than command execution-based. Many have proven this task with huge multi-intent training data. In addition, lots of research have addressed multi intent problem only. The problem of multi intent also poses the challenge of addressing the order of execution of intents found. Hence, we are proposing a unified architecture to address multi-intent detection, associated slotsdetection and order of execution of found intents using low proportion multi-intent corpusin the training data. This architecture consists of Multi Word Importance relation propagator using Multi-Head GRU and Importance learner propagator module using self-attention. This architecture has beaten state-of-the-art by 2.58% on the MultiIntentData dataset.

## introduction many voice assistants like samsung bixby, amazon alexa, microsoft cortana, google assistant has provided voice solution to ease the phone usage for the users. to make user experience more conversational rather than command oriented, exploration on handling multi intent by nlu is increasing. nlu currently handles three important task for identification. domain detector (dd) is the task of identifying which domain or application should execute the utterance. id is the task of identifying what is the intent of the user from the utterance. sf is the task of identifying the objects of interest (named entities) on which we execute the intent operation. for multi-intent, id task involves identification of one or more intents in the utterance told by the user. hence, id must be able to identify the dynamic number of intents present in the utterance along with identification of the boundaries or segments for each intents. in addition, the order of execution of the identified intents matters as one intent execution might be dependent on the other intent execution. lastly, we would like to have low proportion of training data for multi-intent so that we reduce the dependency on data generation and maintenance. hence, we propose a unified architecture that address the following problems: multi intent identification, slot identification, multi-intent boundary detection and execution order of intents. lots of work has happened in the area of single intent and slot. @xcite proposes the exploration of bert architecture for nlu task where pre-trained bi-directional representation on unlabeled corpus, with simple fine tuning, aided in the task of combined single intent prediction and slot @xcite offer bi-directional interrelated information sharing between intent learnings and slot learnings. in addition, they use new iteration mechanism to enhance the sharing of the learnings. @xcite project the usage of word attention, calculated using word embedding, in addition to semantic level attention at each decoding step of bi-lstm. they also use fusion gate for fusing the intent and slot learnings for enhancing the relationships between intent and slot. @xcite recommend the use of multi modal bi-linear pooling technique for fusing the learnings of intent and slot. @xcite outlines the usage of bi-lstm along with attention for jointly learning the learnings of intent and slot. @xcite recommends the usage of convolutional neural network @xcite along with tri-crf for the joint task of intent detection and slot learning. @xcite recommends the usage of attention information along with recur-rent neural networks in encoder-decoder stage that enhances the learnings of intent and slot. @xcite recommend the usage of encoder for encoding the sentence representation using cnn , for local feature and high level phrase representation, and bi-lstm for capturing contextual semantic information and decoding using attention information, calculated from encoder, in each decoding step of long short term memory [lstm] decoder. @xcite offers the usage of recurrent neural network @xcite for solving the problem of joint intent and slot by updating the intent detection as and when words are coming from the utterance. wang et al. (2018a) suggest the usage of bi-model network where two parallel bi-lstm are used and they use the hidden information of one bi-lstm to another in each network. then they use the learnings to predict intent and slot from each network. @xcite offer to use cross-attentive information propagation for enhancing the meaning of the word at word level as well as tagging level to aid the task of joint intent and slot prediction. the above networks has explored many ways of capturing important word level information and fusing the learnings for predicting single intent and slot. inspired on the information captured, explorations on multi intent detection and slots have gathered steam recently to make the task finding generic. @xcite suggest bi-lstm encoder for encoding the sentence information and uses sentence level attention information as well as word-level attention information for each time step in both the decoders. one decoder predicts word-level intent detection, another decoder predicts word-level slot detection and one feed forward neural network predicts sentencelevel intent prediction. this network suffers from contiguous boundary utterance detection. @xcite suggests the usage of share information between multiple intents to identify segments belonging to each intent by using hidden layer to map the learning of word importance to intent prediction. the network is very shallow and is unable to capture the long distance word relationship. @xcite suggest usage of twostage system to detect multiple intents in a single utterance when the model is trained with single utterance by first breaking the utterance into two chunks in the first stage and processing each chunk sequentially by the model. this method suffers from pipeline approach where error in first stage results in propagative error in model stage. the exploration, in this area, is very less comapred to single intent and slot, dur to the absence of proper open source dataset for multi intent data.hence, we are proposing a novel architecture, which will address the following: multiple intent detection, intent segmentation or boundaries and execution order of the found multi intents and slot prediction, where execution order determines the relationship between intents to derive sequence of execution on the voice assistant system.all the experiments are run on multiintentdata dataset, a newly deveoped dataset. we first address the problem statement in detail in dataset section, and then followed by dataset pre-processing for extracting the required information, architecture explanations, results discussion and finally conclusion. ## dataset we solve four types of problem in this paper namely word-level intent prediction, sentence-level intent prediction, word-level order prediction, and wordlevel slot prediction. we use word-level intent prediction for finding the boundaries or segments of multiple intents present in the sentence as shown in figure we use sentence-level intent prediction for identifying all the intents present in the sentence as shown in figure we use word-level order prediction for finding out the order in which the intent segments or intent boundaries must execute as shown in figure 3 we use word-level slot prediction for finding all the slots in the sentence as shown in figure 4 . figure 4: word level slot prediction for handling very less multi-intent training data, we have created new training and test dataset with the help of linguist namely multiintentdata. multi-intentdata dataset was created using single intent information from two domains namely gallery and camera. gallery domain contains the information shown in open gallery with or without using picture name, album name and folder name g-102 share the pictures found using picture name, album name and folder name g-219 add the pictures found using picture name, album name and folder name to wallpaper camera domain contains the information shown in ## proposed method this section explains data preprocessing followed by model 1, model 2 and model 3. ## result and discussion we ran all the three models using multiintentdata dataset. we modified rashmi and narayanaswamy (2019) architecture to predict order as well, similar to prediction of word-level intent by adding new decoder and providing same attention information for each decoder step to predict word-level order, and ran on multiintentdata dataset to compare with state of the art. all the result are captured in from the table we are able to beat the state of the art architecture by 2.58%. this is attributed to the fact that parallel network along with importance learner module is able to enhance the learning of intent, slot and order when compared to unified architecture proposed in @xcite . in addition, the word differentiation between part of the catchall and part of the separator is handled well by model 3. from the result in ## conclusion this work showed the importance of multi intent detection with associated slots and its order of execution over single intent and slot. this also showed the importance of multi intent learning using low corpus data. we are able to derive that multi head gru aide in better contextual understanding of the input embedding representation. in addition, the presence of parallel network for intent and slot learning, along with two-importance learner module has shown better understanding on the differentiation between boundaries of the slot and start of the next intent. in addition, the word level intent aided in influencing the overall sentence level intent. the word level intent as well as the separator also influence the intent ordering execution. this has resulted in improvement to the tune of 3.72%. multi head self-attention learning on context aided in improving from state of the art model by 2.58%. future scope is to resolve anaphora resolution of slots within intent and across intent.