Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling (2022)

Abstract: Joint intent detection and slot filling is a key research topic in natural language understanding (NLU). Existing joint intent and slot filling systems analyze and compute features collectively for all slot types, and importantly, have no way to explain the slot filling model decisions. In this work, we propose a novel approach that: (i) learns to generate additional slot type specific features in order to improve accuracy and (ii) provides explanations for slot filling decisions for the first time in a joint NLU model. We perform an additional constrained supervision using a set of binary classifiers for the slot type specific feature learning, thus ensuring appropriate attention weights are learned in the process to explain slot filling decisions for utterances. Our model is inherently explainable and does not need any post-hoc processing. We evaluate our approach on two widely used datasets and show accuracy improvements. Moreover, a detailed analysis is also provided for the exclusive slot explainability.

## introduction natural language understanding (nlu) is a critical component in building intelligent interactive agents such as amazon alexa, google assistant, microsoft's cortana, and samsung's bixby. it helps to understand the intricate details of user utterances, including: (i) understanding the intents of the utterances that help determine the agent's actions and (ii) detecting slots that signify important entities and phrases required to complete the actions. figure 1 shows an example utterance with an intent and two slots (a slot is a type and value pair). typically, slot classification (i.e., slot filling) is viewed as a sequence labeling problem which is handled through bio sequence labeling notation as shown in figure 1 . state-of-the-art techniques in nlu jointly optimize both the intent detection and slot filling tasks @xcite , including various attention, gating, and transformer-based architectures @xcite . however, existing techniques treat all slot types collectively. for example, they may compute a general attention weights vector to represent all the slot types. therefore, they do not learn slot type specific patterns, and features based on those patterns that can potentially increase the model performance. in this work, we show that both learning to capture slot type specific patterns through attention weights and computing new features from them 'explicitly' is beneficial in at least two ways: (i) boosting joint nlu model accuracy, and (ii) making the model explain its slot filling decisions at a granular level. to properly train our slot type specific attention weights learning, we introduce an auxiliary network of binary classifiers to the joint model that has an additional optimization objective. auxiliary networks and respective optimization objectives in nlp are often used to enforce additional constraints @xcite . we use an auxiliary network to enforce slot type specific attention learning. on the other hand, explainable ai has gained attention in nlp @xcite as it provides 'explanations' for 'black box' deep learning model decisions. the terms 'interpretable ai' and 'explainable ai' have been used interchangeably in the literature @xcite . to be consistent and avoid confusion 1 , we use the term 'explainable' in this work to refer to a model's abil-ity to provide explanations to the outside world of its inner working, similar to @xcite . according to @xcite , explainable nlp models fall into two main categories: (i) models that require external posthoc processing for explainability, and (ii) models that are inherently explainable due to the capability built into the model. the latter is argued to be preferred in the literature @xcite . explainable models are transparent @xcite and hence, can gain trust. however, existing joint nlu approaches disregard this important explainability aspect. our proposed joint nlu model in this work is inherently explainable and explains slot filling decisions for each slot type, which is at a granular level. to the best of our knowledge, our approach is the first to inherently support finegrained explainability for slot filling in a joint nlu model. our contributions in this work are two-fold: ## related work intent detection and slot filling tasks are two typical sub-tasks of nlu. slot filling is generally challenging as decisions have to be made for each word/token and can be seen applied in interesting use cases @xcite . the two tasks were performed independently in the past @xcite . however, jointly optimizing them has shown better accuracy recently (e.g., @xcite @xcite @xcite ). moreover, recent advances in contextual language models have improved the language encoding capabilities that are directly reflected in joint nlu models @xcite compared to traditional static word embedding approaches @xcite . existing joint nlu models leverage rnn @xcite , cnn @xcite , intent influenced attention @xcite , gated attention @xcite , self attention @xcite @xcite , stack propagation @xcite , hierarchical information flow @xcite @xcite , multi task learning @xcite , memory networks @xcite , syntactic information integration @xcite , and custom transformer architectures @xcite . the models mentioned above can learn correlations and hierarchy that exist between intents and slots, but compute features collectively over all slot types. therefore, these correlations or attention mechanisms are not useful to explain slot filling decisions because (i) they are abstract and do not correspond to each individual slot filling decision, and (ii) may focus on general language semantics or special tokens @xcite . also, most deep neural models are 'black boxes' and hence, not transparent and explainable @xcite , making them hard to earn trust and fix errors. hence, explainable systems have gained attention for nlp problems such as sentiment classification and question answering @xcite and outside nlp @xcite . importantly, rudin argues that developing inherently explainable models is important because use of post-hoc techniques to explain existing systems leads to many problems @xcite . ours is the first inherently explainable method for slot filling in a joint nlu model. in our proposed model, we compute slot type specific attention weights to explain slot filling decisions. there has been a debate on whether attentions are explainable @xcite . however, it has been shown that attentions can be used for explanations, if they contribute towards the model outcomes/behaviors @xcite . we show that this is true in our model and provide evidence for a strong correlation between the slot type attentions and slot predictions (section 4.2). orthogonal to ours, having multiple final slot classifiers is possible to model separate features but it introduces the big challenge of resolving overlapping spans (i.e., nested ner) @xcite . ## approach our approach consists of three network components: (i) intent classifier, (ii) slot classifier, and (iii) slot type weight and feature generator. existing joint nlu models consist of the first two network components arranged in various configurations whereas ours is distinguishably different from them because of the third auxiliary network component. this third component learns to compute: (i) separate per slot type features (i.e., logits) and (ii) fine-grained per slot type attention weights for each utterance token. further, it is supervised through automatically generated training data (see section 3.1.2) and constrained to exactly the number of slot types. our per slot type attention weights can provide insights on the focus points of the input utterance that are relevant to labeling a particular slot type, which is absent in other approaches. our approach is illustrated in detail in figure 2 . ## evaluation and discussion we use two most widely used datasets for joint nlu: atis @xcite and snips @xcite . atis has 4478, 500, and 893 utterances for train, dev, and test splits, respectively. snips has 13084, 700, and 700 utterances for train, dev, and test splits, respectively. we measure slot accuracy on the exact span match and maximum utterance length is 50. hyperparameters we experimented with batch size, and number of epochs and set learning rate and dropout as in jointbert @xcite (appendix a.3). we experimented with 20, 30, and 40 epochs, batch sizes of 32, 64, and 128, and trained the model for the entirety of the epochs 2 . using 40 and 20 epochs with batch size 32 gave the best performance on atis and snips datasets, respectively. for all the experiments, we used ## conclusion we presented a novel joint nlu approach where we jointly model intent detection and slot filling with per slot type attention learning. slot type specific attention weights enabled the model to provide fine-grained slot explanations and additional slot type features computed through the auxiliary network (i.e., logits) improved model accuracy. to the best of our knowledge, our proposed approach is the first to inherently provide explanations for slot filling while jointly detecting both intents and slots in utterances, without any post-hoc processing. the added transparency from our explainable model is beneficial for further debugging and improvement of models, and for increasing end user trust in personal voice assistants and task oriented dialog systems. in future, we plan to investigate the use of this type of inherently explainable model through auxiliary network constraint enforcement for problems such as text classification and ner.