D o LFI n: Distributions over Latent Features for Interpretability (2020)

Abstract: Interpreting the inner workings of neural models is a key step in ensuring the robustness and trustworthiness of the models, but work on neural network interpretability typically faces a trade-off: either the models are too constrained to be very useful, or the solutions found by the models are too complex to interpret. We propose a novel strategy for achieving interpretability that – in our experiments – avoids this trade-off. Our approach builds on the success of using probability as the central quantity, such as for instance within the attention mechanism. In our architecture, DoLFIn (Distributions over Latent Features for Interpretability), we do no determine beforehand what each feature represents, and features go altogether into an unordered set. Each feature has an associated probability ranging from 0 to 1, weighing its importance for further processing. We show that, unlike attention and saliency map approaches, this set-up makes it straight-forward to compute the probability with which an input component supports the decision the neural model makes. To demonstrate the usefulness of the approach, we apply DoLFIn to text classification, and show that DoLFIn not only provides interpretable solutions, but even slightly outperforms the classical CNN and BiLSTM text classifiers on the SST2 and AG-news datasets.

## introduction having insights into how a trained neural network solves a given task is important, in order to build more robust, trustworthy, and accurate models @xcite @xcite . however gaining insights is often challenging because of the large number of parameters and the nonlinear dependencies between components of the model. one approach to 'opening the blackbox' is to use saliency maps @xcite , to examine which input components (e.g., words) are taken into account. nevertheless salience alone does not tell us how exactly salient components contribute to the decision of a model. for instance, a sentiment-analysis model might mark both "boring" and "wonderful" in "this film would be wonderful, if the beginning wasn't so boring" as salient, but saliency scores do not reveal the crucial interaction between all the words in the sentence that ultimately let it express a negative sentiment. alternatively, one can analyse the flow of information through the network, by tracking (relevance) gradients backwards @xcite or decomposing (forward) contributions to all intermediate quantities @xcite . these methods address some of the problems of saliency maps, but produce results that themselves require further interpretation. for instance, such a method might quantify the relative contribution of the 3rd word when processing the 7th word in the 2nd layer of a multilayer lstm. that quantity, however, does not easily translate to an explanation for the final prediction that the model generates. a third approach looks at the attention mechanism @xcite , which regulates which components a model attends to. visualising attended components is straight-forward @xcite as an attention weight is the probability that the corresponding component is taken into account for further computation. at a higher level, as in @xcite , one can compute rationales which are attended pieces of text. however, these approaches face some of the same problems as saliency maps; e.g., attention alone does not indicate how much an input component supports a category. in this paper we focus, like the attention-based approach, on probability. we start from the observation that probability is a well studied mathematical tool, that has already been much used in building explainable and robust neural networks (although we also note that simply turning interpretation quantities from other approaches into probabilities by normalizing is not necessarily helpful, as these probabilities are not faithful to the true behaviour of the model). we propose dolfin (distributions over latent features for interpretability) based on probability; our architecture maps an input (e.g., a text) to a bag of latent features (bolf, see figure 1a and §3 ) so that interpretation quantities are probabilistic. in other words, the representation of an input is a vector whose elements range from 0 to 1, indicating to what extent a latent feature is in the bag. to do so, we employ linear-softmax layers (i.e., neural layers with softmax activation, lsl for short) to map input components to distributions over latent features, and a truncated sum to aggregate the resulting distributions. with this new type of representations, we can easily compute q(c|w, s), the probability that input component w in context s supports category c, by decomposing the term to p(f |w, s), the distribution over latent features, and q(c|f ), the probability that feature f supports category c. the former is given by an lsl and the latter can be estimated by the input-output statistics. to illustrate the feasibility and benefits of this idea, we employ dolfin for text classification that, going beyond saliency maps and attention, can tell us the probability a word supports a category. we demonstrate that dolfin does not trade off interpretability against classification accuracy. we build dolfin-conv and dolfin-bilstm which are variants of the classical cnn proposed by @xcite and bilstm text classifier (figure 1b , c ). carrying out experiments on three popular datasets, trec question, sst2, and ag-news, we find that dolfin achieves slightly higher accuracy than cnn and bilstm on sst2 and ag-news. it is worth noting that, although used for text classification in this paper, dolfin is applicable to a wide range of classification tasks such as natural language inference and relation prediction. ## text classification baselines given a text of n words @xmath0, ..., w n-1 ) and a set of m categories @xmath1, ..., c m-1 }, a probabilistic text classifier is p(c|s), which assigns a probability to the prediction that c is the category of s. a traditional text classification architecture adopts the diagram shown in figure 1b , a classical cnn text classifier @xcite utilises a convolutional layer to map each word w i (with its context) to a vector w @xmath2. it then uses max pooling over {w i } n-1 @xmath3. a bilstm text classifier (figure 1c ) uses an bilstm to compute w i , and an aggregator concatenating the backward part of w 0 and the forward part of w n-1 to form s. ## dolfin with bags of latent features (bolf) for simplicity, we introduce dolfin as a neural text classifier, depicted in figure 1a , but dolfin should be applicable to several classification tasks. the key part of dolfin, bolf, is an aggregator consisting of linear-softmax layers (lsl) and a truncated sum. this aggregator firstly maps each w i to a distribution u i over d latent features using lsls, i.e., u @xmath4, so that u i,@xmath5, s). it then uses the truncated sum to compute @xmath6, we apply the element-wise min operator so that the j-th entry of r, i.e., r @xmath7, 1], can be considered as a soft indicator of the extent to which the j-th latent feature f j contributes to the final output. intuitively, we represent s by a bag of latent features r. if r j is close to 1, feature f j likely appears in the bag. to represent s, where each feature f j is represented by a vector f @xmath8. the sum inside the brackets can be seen as a bag of the latent feature representations. ## experiments in our experimental evaluation we investigate whether dolfin can indeed produce interpretable solutions, without sacrificing accuracy. our implementation is in python with pytorch @xcite . the source code and data are available at https://github.com/lephong/dolfin . extra information is provided in the appendix. figure 2 : top -a trec question. each word is highlighted according to q(c|w, s). for example, "how" is highlighted more than "bails" in the second question because q(desc|how, s) > q(desc|bails, s). the left-most token on each line is the label of category c (e.g. the second question is assigned to category desc). bottom -a text in ag-news. ## conclusion we have proposed a new architecture dolfin based on probability for building explainable models. dolfin represents input by a bag of latent features using linear-softmax layers to map input components to distributions over latent features, and a truncated sum to aggregate these resulting distributions. we showed that, different from attention and saliency maps, it is straight-forward to compute how much an input component supports a category. demonstrating our idea, we applied dolfin to text classification. compared with the classical cnn and bilstm text classifiers, dolfin achieved comparable accuracies, but much better interpretability. bolei zhou, aditya khosla, agata lapedriza, aude oliva, and antonio torralba. 2016. learning deep features for discriminative localization. in proceedings of the ieee conference on computer vision and pattern recognition, pages 2921-2929.