Explain by Evidence: An Explainable Memory-based Neural Network for Question Answering (2020)

Abstract: Interpretability and explainability of deep neural net models are always challenging due to their size and complexity. Many previous works focused on visualizing internal components of neural networks to represent them through human-friendly concepts. On the other hand, in real life, when making a decision, human tends to rely on similar situations in the past. Thus, we argue that one potential approach to make the model interpretable and explainable is to design it in a way such that the model explicitly connects the current sample with the seen samples, and bases its decision on these samples. In this work, we design one such model: an explainable, evidence-based memory network architecture, which learns to summarize the dataset and extract supporting evidences to make its decision. The model achieves state-of-the-art performance on two popular question answering datasets, the TrecQA dataset and the WikiQA dataset. Via further analysis, we showed that this model can reliably trace the errors it has made in the validation step to the training instances that might have caused this error. We believe that this error-tracing capability might be beneficial in improving dataset quality in many applications.

## introduction interpretability of neural networks is an active research field in machine learning. deep neural networks might have tens if not hundreds of millions of parameters @xcite organized into intricate architectures. the sheer amount of parameters and the complexity of the architectures largely prevent human to directly make sense of which concepts and how the network truly learns. the comparative lack of explainable intuition behind deep neural networks might hamper the development and adoption of those models. in certain scenarios, prediction accuracy alone is not sufficient @xcite . for example, as discussed in @xcite , it is difficult to trust a deep model even if it has high test set performance given the inherent biases in the dataset. thus, we argue that interpretability is perhaps one of the keys to accelerate both the development and adoption of deep neural networks. there have been many successful attempts from the research community to make sense of deep models' prediction. these attempts can be broadly categorized into several classes. one of the major classes concerns with the network visualization techniques, for example, visual saliency representations in convolutional models @xcite . for recurrent neural networks (rnn), @xcite focused on analyzing and visualizing the rnn to explain its ability to keep track of long-range information. the visualization-based methods, although achieving great successes, still operate on a very high level of abstractions. it requires a great deal of machine learning knowledge to make use of those visualizations. thus, these techniques are not always useful for a broader audience, who might not have the machine learning expertise. looking back at classic machine learning models, one class of models stands out as being very intuitive and easy to understand: the instance-based learning algorithms. the k-nearest neighbors algorithm, a prime example, operates on a very human-like assumption. to elaborate, if the current circumstances are similar to that of a known situation in the past, we may very well make this decision based on the outcome of the past decision. we argue that this assumption puts the interpretability on a much lower level of abstraction compared to the visualization methods. if somehow our model can learn how to link the evidences from the training data to the prediction phase, we will have a direct source of interpretability that can be appreciated by a broader audience. the k-nearest neighbors algorithm, as an instance-based method, might not be a deep neural network technique; however, there have been many papers in the deep model literature inspired by or related to this method. a notable example is the neural nearest neighbors network @xcite . moreover, there is a class of problems with strong links to k-nearest neighbors: few-shot learning. it is from the two major papers in the few-shot learning literature, the prototypical network @xcite and the matching network @xcite , we find a potential realization for our ideas. in few-shot learning, it is possible to learn the support from each of the instances from the support set to the current prediction; however, such approach is infeasible when the training data get larger. inspired by the techniques discussed in @xcite , we apply a training-level data summarizer based on the neural turing machine (ntm) @xcite ) that reads the dataset and summarizes (or writes) it into a few meta-evidence nodes. these meta-evidence nodes, in turn, lend support to each of the prediction similar to a few-shot learning model. the parameters of the ntm are jointly trained with other parameters of the network. our final model not only has great predictive power and achieves state-of-the-art results on two popular answer selection datasets, but also shows a strong "error-tracing" capability, in which the errors in the validation set can be traced to the sources in the training set. to summarize, our contributions in this work are twofold. first, we propose a novel neural network model that achieves state-of-the-art performance on two answer selection datasets. second, we show the utility of the error-tracing capability of our model to find the noisy instances in the training data that degrades the performance on the validation data. this capability might be very useful in real-life machine learning scenarios where the training labels are noisy or the inter-annotator agreement is low. ## proposed framework question answering (or answer selection) is the task of identifying the correct answer to a question from a pool of candidate answers. it is an active research problem with applications in many areas @xcite @xcite @xcite . similar to most recent papers on this topic @xcite @xcite , we cast the question answering problem as a binary classification problem by concatenating the question with each of the candidate answers and assigning positive label to the concatenation containing the correct answer. in most supervised learning scenarios, performing a full distance calculation between the current data point and every training data point would be computationally intractable. to overcome this burden, we propose a memory controller based on ntm to summarize the dataset into meta-evidence nodes. similar to ntm, the controller is characterized by reading and writing mechanisms. assume that we provide the controller with k cells e 1 , . . . , e k in a memory bank (i.e. to store k support/evidence vectors), and let us denote the t-th data point as x t (obtained by using a pretrained embedding model to embed the concatenation of a question and a candidate answer), the memory controller works as follows. writing mechanism. the writing mechanism characterizes how the controller updates its memory given a new data point. to update the memory, however, we first need an indexing mechanism for writing. instead of using the original indexing of the ntm, we adopt the simpler indexing procedure from the memory network, which has been proven to be useful in this task @xcite . at time step t, for each incoming data point x t , we compute the attention weight w e t i for the support vector e t i : from these attention weights, we find the writing index for an input x t by maximizing the cosine similarity between x t and the evidence vectors: with the writing index found, we compute the memory update weight via a gating mechanism: reading mechanism. the reading mechanism characterizes how the controller uses its memory and the current input to produce an output. instead of reading one memory cell, we aim to learn the support of all meta-evidence nodes. thus, the weighted sum is used to create a support vector s t : we then incorporate the original input with the support vector s t to produce the negative/positive class probabilities p (x t ) as follows: the overall information flow of our model is visualized in figure 1 . our formulation draws inspiration from the ntm and the memory network. our indexing algorithms in writing and reading mechanisms are similar to the memory network, which is simpler than the ntm. however, the memory network only stores intermediate computation steps in the memory, and these memories can be considered as internal layers of the network. our memory, on the contrary, is external and not trained, only updated by the writing mechanism. in this regard, the memory bank of our model is more similar to the ntm.