Towards Explainable Dialogue System: Explaining Intent Classification using Saliency Techniques (2021)

Abstract: Deep learning based methods have shown tremendous success in several Natural Language Processing (NLP) tasks. The recent trends in the usage of Deep Learning based models for natural language tasks have definitely produced incredible performance for several application areas. However, one major problem that most of these models face is the lack of transparency, i.e. the actual decision process of the underlying model is not explainable. In this paper, at first we solve a very fundamental problem of Natural Language Understanding (NLU), i.e. intent detection using a Bi-directional Long Short Term Memory (BiLSTM). In order to determine the defining features that lead to a specific intent class, we use the Layerwise Relevance Propagation (LRP) algorithm to find the defining feature(s). In the process, we conclude that saliency method of eLRP (epsilon Layerwise Relevance Propagation) is a prominent process for highlighting the important features of the input responsible for the current classification which results in significant insights to the inner workings, such as the reasons for misclassification by the black box model.

## introduction chatbots or conversational agents have been gaining immense popularity in recent years. this is one of the most widely used artificial intelligence (ai) applications that has a market value of usd 190.8 millions, and is expected to grow upto usd 1,250.1 million by the year 2025 foot_0 . these chatbots are being used in almost every vertical of our society, such as travel, healthcare, judiciary etc. with the rapid adaptation of chatbots as digital assistants, it is important that these chatbots should be very robust, as many of these domains (e.g., health, judiciary etc.) are very sensitive, and minor inaccuracies in information can lead to significant damage. the model as a whole can be made robust if all its individual components are also accurate. the very first step of most modular dialogue systems is the natural language understanding (nlu) phase, that comprises of dialogue act classification, intent detection and slot filling. this part of the dialogue system plays an important role of deciphering the syntax and semantics of the user input, to aptly produce the bot's reply. while the intent classification focuses on the semantic meaning of the input, slot filling focuses on extracting the relevant information like named entities etc. these systems are not perfect, and even the stateof-the-art models very often fail to classify the intent correctly. in order to understand what went wrong in these misclassifications, the features of the text that led to the incorrect classification can provide some helpful information. due to the rise of deep neural network based architectures the transparency of such models is low. this leads to the requirement of explainable artificial intelligence (xai) methods that determine the important features of the input text. there are 2 major methods that highlight the feature importance, namely saliency based methods and attention based methods. saliency based methods (section 2.2) are adhoc techniques that explain individual inference done by the model. this is done after the model training process, hence the cost depends on the number of explanations required. the additional cost of these xai models tend to make the architectural framework more expensive. since most of the xai methods explain each prediction individually, the processing cost keeps on increasing during the model deployment. attention, on the other hand, calculates the feature importance over the entire training data, and seems like a cost effective alternative to saliency in figuring out the relevant features of the input text. whether attention is actually a good alternative for explanation is still a matter to be explored @xcite . when compared to the saliency based techniques, due to the attention's focus on gradient descent as the weight updating criteria, attention has shown high correlation to the gradient based saliency techniques @xcite . in this paper, we investigate into an explainable deep learning based intent classifier. we compute the features responsible for misclassification of the utterances, in order to get a better idea of why actually the trained model incorrectly predicted these test inputs. this leads to a much better understanding of the limitations of long short term memory (lstm) based models. one such limitation is when we go over the atis dataset, where we find out that the model misclassifies an intent class(as shown in figure 2 ) 'meal' even though it learns to identify the 'meal' token as the most important feature as cumulative weight of tokens pointing to the 'flight' intent is higher. another such instance can be the 'day name' intent being misclassified as 'flight'. the model does not even learn to pay attention to token(s) like 'day' or 'day of the week' as the total instances of 'day name' in the entire dataset is less than 0.1% of the dataset. ## related work in this section, we present a very brief literature survey that starts with a intent classification followed by saliency based explainable models. ## methodology we implement lrp (epsilon layerwise relevance propagation) model over a bilstm trained model to find the important features for a particular prediction. this is done with the aim of finding the reason behind the misclassification in incorrectly predicted utterances, as that can potentially help us deduce the reason for misclassification and improve our understanding of the model. ## conclusion and future work in this paper, we have attempted to build an explainable intent detection model with the saliency based methods. the model is able to identify the appropriate and relevant features used for intent classification. we also discussed some issues with these approaches, most of which deal with the fact that the saliency techniques calculate the feature importance (which constitute an explanation) as an adhoc measure. saliency does have quite a few benefits of itself. the modular nature of the implementations provides a degree of model-agnostic behaviour which allows us to treat the black boxed nature of the deep learning models as an afterthought and focus entirely on the performance. after the model is trained and tuned, we applied the saliency techniques for determining the feature relevance. this also ensures that we can apply different saliency techniques for the same base model and the same saliency technique to different models allowing for a high degree of robustness. however, even for saliency it is not necessary that the importance assigned to a feature is, in fact, due to the relevance of the features but could simply be a result of the underlying issues with the technique used. for example, in occlusion based methods, if we remove a feature, it is possible that the change in the prediction is just the result of the new input not being in the format the model expects @xcite . for future work, we plan to use the feature importance information and use it to retrain the model in such a way to reduce misclassification. one such method could be to use the important features of misclassifications to help identify which kind of data to add, to improve the performance of the model further. however, such a method needs to be done in such a way that the incorrect predictions do not get corrected at the cost of misclassification of originally correctly predicted utterances. another such method could be to identify the nodes which are more relevant to a highly misclassified intent and boost those neurons to improve model performance. however, this also needs to make sure the nodes we are associating with a particular intent do not have high influence on other intents as well, as that might lower the accuracy of some other intent.