C AN I TRUST YOU MORE ? M ODEL -A GNOSTIC H IERARCHICAL E XPLANATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models.
We propose Mah´e , a novel approach to provide M odel- a gnostic h ierarchical ´e xplanations of how powerful machine learning models, such as deep neural networks, capture these dependencies, either context-dependent or contextfree.
Speciﬁcally, Mah´e provides context-dependent explanations by a novel local interpretation algorithm that effectively captures any-order interactions and obtains context-free explanations through generalizing context-dependent interactions to explain global behaviors.
Experimental results show that Mah´e obtains improved local interaction interpretations over state-of-the-art methods and successfully provides explanations of interactions that are context-free.
I NTRODUCTION
State-of-the-art machine learning models, such as deep neural networks, are exceptional at modeling complex dependencies in structured data, such as text (Vaswani et al., 2017; Tai et al., 2015), images (He et al., 2016;
Huang et al., 2017), and DNA sequences (Alipanahi et al., 2015; Zeng et al., 2016).
However, there has been no clear explanation on what type of dependencies are captured in the black-box models that perform so well (Ribeiro et al., 2018; Murdoch et al., 2018).
In this paper, we make one of the ﬁrst attempts at solving this important problem through interpreting two forms of structures, i.e., context-dependent representations and context-free representations.
A context-dependent representation is the one in which a model’s prediction depends speciﬁcally on a data instance level (such as a sentence or an image).
In order to illustrate the concept, we consider an example in image analysis.
A yellow round-shape object can be identiﬁed as the sun or the moon given its context, either bright blue sky or dark night.
A context-free representation is one where the representation behaves similarly independent of instances (i.e., global behaviors).
In a hypothetical task of classifying sentiment in sentences, each sentence carries very different meaning, but when “not” and “bad” depend on each other, their sentiment contribution is almost always positive - i.e., the structure is context-free.
To investigate context-dependent and context-free structure, we lend to existing deﬁnitions in interpretable machine learning (Ribeiro et al., 2016;
Kim et al., 2018).
A context-dependent interpretation is a local interpretation of the dependencies at or within the vicinity of a single data instance.
Conversely, a context-free interpretation is a global interpretation of those dependencies and applies to entire classes of data.
We formalize the notion of a dependency within context-dependent and -free representations as an interaction relationship between the prediction and input features.
Interactions are appropriate because they can describe arbitrarily complex relationships between prediction and features, which are commonly captured by state-of-the-art models like deep neural networks (Tsang et al., 2017; Murdoch et al., 2018).
In this work, we propose Mah ´ e , a framework for explaining the context-dependent and contextfree structures of any complex prediction model, with a focus on explaining neural networks.
The context-dependent explanations are built based on recent work on local intepretations (such as (Ribeiro et al., 2016; Murdoch et al., 2018; Singh et al., 2018)).
Speciﬁcally, Mah´e takes as input a model to explain and a data instance, and returns a hierarchical explanation, a format proposed by Singh et al.
(2018) to show local group-variable relationships used in predictions (Figure 1).
To provide context-free explanations, Mah´e generalizes those context-dependent interactions with con-
[Figure]
sistent behavior in a model and determines whether a local representation in the model is responsible for the global behavior.
In this case, Mah ´ e takes as input a model and representative data corresponding to an interaction of interest and returns whether or not that interaction is context-free.
We conduct experiments on both synthetic datasets and real-world application datasets, which shows that Mah´e ’s context-dependent explanations can signiﬁcantly outperform state-of-the-art methods for local interaction interpretation, and Mah´e is capable of successfully ﬁnding context-free explanations of interactions.
In addition, we identify promising cases where the methodology for context-free explanations can successfully edit models.
Our contributions are as follows: 1) Mah´e achieves the task of improved context-dependent explanations based on performance and generality, compared to state-of-the-art methods for local interaction interpretation, 2) Mah´e is the ﬁrst to provide context-free explanations of interactions in deep learning models, and 3) Mah´e provides a promising direction for modifying context-free interactions in deep learning models without significant performance degradation.
R ELATED W ORKS
Attribution Interpretability : A common form of interpretation is feature attribution, which is concerned with how features of a data instance contribute to a model output.
Within this category, there are two distinct approaches: additive and sensitivity attribution.
Additive attribution interprets how much each feature contributes to the model output when these contributions are summed.
In contrast, sensitivity attribution interprets how sensitive a model output is to changes in features.
Examples of additive attribution techniques include LIME (Ribeiro et al., 2016) and CD (Murdoch et al., 2018).
Examples of sensitivity attribution methods include Integrated Gradients (Sundararajan et al., 2017), DeepLIFT (Shrikumar et al., 2017), and SmoothGrad (Smilkov et al., 2017).
Unlike previous approaches, Mah´e provides additive attribution interpretations that consist of non-additive groups of variables (interactions) in addition to the normal additive contributions of each variable.
Interaction Interpretability : An interaction in its generic form is a non-additive effect between features on an outcome variable.
Only until recently has there been development in interpreting nonadditive interactions despite often being learned in complex machine learning models.
The difﬁculty interpreting non-additive interactions stems from their lack of exact functional identity compared to, for example, a multiplicative interaction.
Methods that exist to interpret non-additive interactions are NID (Tsang et al., 2017) and Additive Groves (Sorokina et al., 2008).
In contrast, many more methods exist to interpret speciﬁc interactions, namely multiplicative ones.
Notable methods include
CD (Murdoch et al., 2018), Tree-Shap (Lundberg & Lee, 2017), and GLMs with multiplicative interactions (Purushotham et al., 2014).
Unlike previous methods, our approach provides local interpretations of the more challenging non-additive interaction .
Locally Interpretable Model-Agnostic Explanations (LIME) :
LIME (Ribeiro et al., 2016) is a very popular type of model interpretation.
Its popularity comes from additive attribution interpretations to explain the output of any prediction model.
The original and most popular version of LIME uses a linear model to approximate model predictions in the local vicinity of a data instance.
Since its introduction, variants of LIME have been proposed, for example Anchors (Ribeiro et al., 2018) and LIME-SUP (Hu et al., 2018).
While Anchors generates a form of context-free explanation, its method of selecting fully representative features for a prediction does not consider interactions.
For example, Anchors assumes that (not, bad) “virtually guarentees” a sentiment prediction to be positive, whereas in Mah´e this is not necessarily true; only their interaction is positive (See Table 5 for an example).
LIME-SUP touches upon interactions but does not study their interpretation.
I NTERACTION E XPLANATIONS 3. I NTERACTIONS IN A DDITIVE A TTRIBUTION E XPLANATIONS
Traditionally, additive attribution methods are based on linear functions (Lundberg & Lee, 2017).
For an data instance x ∈ R p , weights w ∈ R p and bias b , attribution scores are given by w i
x i in
[Equation]
Given a model output to explain f ( · ) and a set of n data points { x ( i ) } ni =1 that are inﬁnitesimally close or local to x , a linear approximation of D = { ( x (1) , f ( x (1) )) , . . .
, ( x ( n ) , f ( x ( n ) )) } will accurately ﬁt to the functional surface of f at the data instance, such that φ ( x ) =
f
( x ) .
Because it is possible in such scenarios that φ ( x ) = f ( x ) ≈ b , there must be some nonzero distances between x and x ( i ) to obtain informative attribution scores.
LIME, as it was originally proposed, uses a linear approximation as above where samples are generated in a nonzero local vicinity of x (Ribeiro et al., 2016).
The drawback of linear LIME is that there is often an error (cid:15) =
| f
( x ) − φ ( x ) | > 0 .
For complex models f , the functional surface at x can be nonlinear.
Because D consists of x ( i ) with distance d > 0 from x , a closer ﬁt to f ( x ) in its nonlinear vicinity, i.e. { f ( x ( i ) ) } ni =1 , can be achieved with the following generalization of Eq. 1:
[Equation]
where g i ( · ) can be any function, for example one that is arbitrarily nonlinear.
This function is called a generalized additive model (GAM) (Hastie & Tibshirani, 1990), and now attribution scores can be given by g i ( x i ) for each feature i .
For the purposes of interpreting individual feature attribution, the GAM may be enough.
However, if we would like broader explanations, we can also obtain non-additive attributions or interactions between variables (Lou et al., 2013), which provide an even better ﬁt to the complex local vicinity.
Augmenting Eq.
2 with non-additive interactions yields:
[Equation]
where g (cid:48) i ( · ) is again any function, x I ∈ R |I| are interacting variables corresponding to the variable indices I , and {I i } Ki =1 is a set of K interactions.
Attribution scores can now be obtained from each g
i and g (cid:48) i .
In this paper, we learn g
i and g (cid:48)
i using Multilayer Perceptrons (MLPs).
Link functions can be used to convert φ or φ K to classiﬁcation.
Mah´e F RAMEWORK
In this section, we introduce our Mah´e framework, which can provide context-dependent and context-free explanations of interactions.
To provide context-dependent explanations, we propose to use a two-step procedure that ﬁrst identiﬁes what variables interact locally, then learns a model of interactions (as Eq. 3) to provide a local interaction score at the data instance in question.
The procedure of ﬁrst detecting interactions then building non-additive models for them has been studied previously (Lou et al., 2013; Tsang et al., 2017); however, previous works have not focused on using the same non-additive models to provide local interaction attribution scores, which enable us to visualize interactions of any size as demonstrated later in §5.2.3.
4. C ONTEXT -D EPENDENT E XPLANATIONS
Local Interaction Detection : To perform interaction detection on samples local to data instance x , we ﬁrst sample n points with mean x ∈ R p to generate x (1) , . . .
, x ( n ) , x ( i ) ∼ N ( x , σ 2 I ) for all i = 1 , . . . , n .
In cases where data features are binary, sampled points consist of randomly perturbed features from x , and distances from x are computed by a distance metric (e.g. cosine distance for text/images, (cid:96) 2 for continuous features) as done in previous perturbation explanation methods (Ribeiro et al., 2016; Lundberg & Lee, 2017; Ribeiro et al., 2018).
Sample weights (e.g. the frequency each sample appears in the sampled dataset) are given by a kernel function of sample distances.
We use a Gaussian kernel with stddev σ , truncating all samples beyond a 1 σ local vicinity.
For a model output f ( · ) to explain, our framework is ﬂexible to any interaction detection method that applies to the dataset D
= { ( x (1) , f ( x (1) )) , . . .
, ( x ( n ) , f ( x ( n ) )) } .
Since we seek to detect non-additive interactions, we use the neural interaction detection (NID) framework (Tsang et al., 2017), which interprets learned neural network weights to obtain interactions.
To the best of our knowledge, this detection method is the only polynomial-time algorithm that accurately ranks anyorder non-additive interactions after training one model, compared to alternative methods that must train an exponential number O (2 p ) of models.
The basic idea of NID is to interpret an MLP's accurate representation of data to accurately identify the statistical interactions present in this data.
Because MLPs learn interactions at nonlinear activation functions, NID performs feature interaction detection by tracing high-strength (cid:96) 1 -regularized weights from features to common hidden units.
In particular, NID efﬁciently detects any-order interactions by ﬁrst assuming each ﬁrst layer hidden unit in a trained MLP captures at most one interaction, then NID greedily identiﬁes these interactions and their strengths through a 2D traversal over the MLP's input weight matrix.
In addition to its efﬁciency, applying NID to our framework Mah´e carries intuition because an MLP can approximate any nonlinear interacting function (Hornik, 1991), and the local vicinity of f ( x ) (e.g. a deep net) can be an arbitrary form of such function.
Hierarchical Interaction Attributions : Upon obtaining an interaction ranking from NID, GAMs with interactions (Eq. 3) can be learned for different top- K interactions (Tsang et al., 2017).
In the
Mah´e framework, there are L + 1 different levels of a hierarchical explanation which constitutes our context-dependent explanation.
When presenting the hierarchy, the ﬁrst level shows the additive attributions of individual features, such as the attributions provided by a trained φ ( · ) in Eqs.
1 or 2.
Subsequently, the parameters w of φ ( · ; w , b ) are frozen before interaction models are added to construct φ K ( · ) in Eq.
The next levels of the hierarchy can be presented either as the interaction attributions of { g (cid:48) i ( · ) } Ki =1 or g (cid:48) K ( · ) alone (Eq. 3), where at each level K is increased and { g (cid:48)
i ( · ) } Ki =1 , bias b are retrained.
The practice of training interaction models g (cid:48)
i on the residual of φ has been used previously to construct interpretable models (Lou et al., 2013).
Since φ K is trained at each hierarchical level on D , the ﬁt of each φ K can also be explained via predictive performance.
The stopping criteria for the number of hierarchical levels can depend on the predictive performance or user preference.
4. C ONTEXT -F REE E XPLANATIONS
In order to provide context-free explanations, we propose determining whether the local interactions assumed to be context-dependent in §4.1 can generalize to explain global behavior in f .
To this end, we ﬁrst deﬁne ideal conditions for which a generic local explanation can generalize.
Deﬁnition 1 (Generalizing Local Explanations) .
Let f ( · ) be the model output we wish to explain, and X f be the data domain of f .
Let a local explanation of f at x ∈ X f be some explanation E that is true for f ( x ) and depends on samples x (cid:96) ∈
X f that are only in the local vicinity of x , i.e. m ( x , x (cid:96) )
≤ d provided a distance metric m and distance d ≥ 0 .
The local explanation E is a global explanation if the following two conditions are met: 1) Explanation E is true for f at all data samples in X f , including samples outside the local vicinity of x , i.e. all samples x
g ∈ X
f satisfying m ( x , x g ) > d . 2)
There exists a sample x (cid:48)
∈
X f and a local modiﬁcation to f ( x (cid:48) )
(modifying f ( x (cid:96) ) in the vicinity m ( x (cid:48) , x (cid:96) ) ≤ d ) that changes E for all samples in X f while still meeting condition 1).
For example, consider a simple linear regression model we wish to explain, f ( x ) =
w 1 x 1 + w 2 x 2 .
Let its local explanation be the feature attributions w 1 x 1 and w 2 x 2 .
This local explanation is a global explanation because 1) for all values of x 1 and x 2 , the feature attributions are still w 1 x 1 and w 2 x 2 , and 2) if any of the weights are changed, e.g. w 1 → w (cid:48) 1 , the attribution explanation will change, but the feature attributions are still w (cid:48)
1 x 1 and w 2 x 2 for all values of x 1 and x 2 .
Our context-free explanation of interaction I is: whenever local interaction I exists, its attribution will in general have the same polarity (or sign).
Since it is impossible to empirically prove that a local explanation is true for all data instances globally (via Deﬁnition 1), this work is focused on providing evidence of a context-free interaction representation.
This evidence can be obtained by checking whether our explanation is consistent with the two conditions from Deﬁnition 1 for the interaction of interest I : 1) For representative data instances in the domain of f , if local interaction
I exists, does it always have the same attribution polarity?
The representative data instances be separated from each other at an average distance beyond d . 2)
Can local interaction I at a single data instance ˜ x be used to negate I ’s attribution polarity for all representative data instances where
I exists?
We propose to modify an interaction attribution of the model’s output f ( x ) at data instance x by utilizing a trained model g (cid:48) k
( x I ) of interaction
I k , where 1 ≤ k ≤ K (Eq. 3).
Let ˜ g (cid:48) k
( · ) be a modiﬁed version of g (cid:48) k
We can then deﬁne a modiﬁed form of Eq. 3:
[Equation]
Without retraining ˜ φ k ( · ) , we use ˜ φ k and the same local vicinity { x ( i ) } ni =1 in D to generate a new dataset ˜ D
= { ( x (1) , ˜ φ k ( x (1) ) , . . .
, ( x ( n ) , ˜ φ k ( x ( n ) )) } .
Finally, we can modify the interaction attribution of f ( x ) by ﬁne-tuning f ( · ) on dataset ˜ D .
In this paper, we modify interactions by negating them: ˜ g (cid:48)
k ( · ) = − cg (cid:48) k
( · ) , where − c negates the interaction attribution with a speciﬁed magnitude c .
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
How can modifying a local interaction affect interactions outside its local vicinity?
This would suggest that the manifold hypothesis is true for f ( · ) ’s representations of these interactions (Figure 2).
The manifold hypothesis states that similar data lie near a low-dimensional manifold in a highdimensional space (Turk & Pentland, 1991;
Lee et al., 2003;
Cayton, 2005; Narayanan & Mitter, 2010).
Various studies have suggested that the hypothesis applies to the data representations learned by neural networks (Rifai et al., 2011; Basri & Jacobs, 2016).
The hypothesis is frequently used to visualize how deep networks represent data clusters (Maaten & Hinton, 2008; LeCun et al., 2015), and it was previously applied to representations of interactions (Reed et al., 2014), but not for neural networks.
Part of our objective is to generalize our explanation as much as possible.
In the case of language-related tasks, we additionally generalize based on our meaning of a local interaction and the distance metric we use, m .
In this paper, local interactions for language tasks do not have word interactions ﬁxed to speciﬁc positions; instead, these interactions are only deﬁned by the words themselves (the interaction values) and their positional order.
For example, the (“not”, “bad”) interaction would match in the sentences: “this is not bad” and “this does not seemed to be that bad”.
For comparing texts and measuring the size of local vicinities, we use Levenshtein (edit) distance (Levenshtein, 1966), which allows us to compare sentences with different word counts.
E XPERIMENTS 5. E XPERIMENTAL S ETUP
We evaluate the effectiveness of Mah´e ﬁrst on synthetic data and then on four real-world datasets.
To evaluate context-dependent explanations of Mah´e , we ﬁrst evaluate the accuracy of Mah´e at local interaction detection and modeling on the outputs of complex base models trained on synthetic ground truth interactions.
We compare Mah´e to Shap-Tree (Lundberg et al., 2018), ACDMLP (Singh et al., 2018), and ACD-LSTM (Murdoch et al., 2018;
Singh et al., 2018), which are local interaction modeling baselines for the respective models they explain: XGBoost (Chen & Guestrin, 2016), multilayer perceptrons (MLP), and long short-term memory networks (LSTM)
(Hochreiter & Schmidhuber, 1997).
[Figure]
tree
[Figure]
lstm lstm
[Figure]
s t a nd a r d
i ze d M S E ∗
Shap
[Figure]
ACD
[Figure]
ACD
[Figure]
Mah´e
[Figure]
In all other experiments, we study Mah´e ’s explanations of state-of-the-art level models trained on real-world datasets.
The state-of-the-art models are: 1) DNA-CNN, a 2-layer 1D convolutional neural network (CNN) trained on MYC-DNA binding data 2 (Mordelet et al., 2013;
Yang et al., 2013; Alipanahi et al., 2015; Zeng et al., 2016; Wang et al., 2018), 2) Sentiment-LSTM, a 2-layer bi-directional LSTM trained on the Stanford Sentiment Treebank (SST)
(Socher et al., 2013; Tai et al., 2015), 3) ResNet152, an image classiﬁer pretrained on ImageNet (Russakovsky et al., 2015;
He et al., 2016), and 4) Transformer, a machine translation model pretrained on WMT-14 En →
Fr (Vaswani et al., 2017; Ott et al., 2018).
The following hyperparameters are used in our experiments.
We use n = 5 k local-vicinity samples for D ( n = 1 k for synthetic experiments), with 80% - 10% - 10% train-validation-test splits to train and evaluate Mah´e .
The distance metrics for vicinity size are: (cid:96) 2 distance for synthetic experiments, cosine distance for DNA-CNN and ResNet152, and Levenshtein distance for Sentiment-LSTM and
Transformer.
We use on-off superpixel and word approaches to binary feature representation for explaining ResNet152 and Sentiment-LSTM respectively (Ribeiro et al., 2016; Lundberg & Lee, 2017), and the other experiments for real-world datasets use perturbation distributions that randomly perturbs features to belong to the same categories of original features, as in (Ribeiro et al., 2018).
[Table]
For the hyperparameters of the neural networks in Mah´e , we use MLPs with 50 - 30 - 10 ﬁrst-to-last hidden layer sizes to perform interaction detection in the NID framework (Tsang et al., 2017).
These MLPs are trained with (cid:96) 1 regularization λ 1 = 5e − 4 .
The learning rate used is always 5e − 3 except for Transformer experiments, whose learning rate of 5e − 4 helped with interaction detection under highly unbalanced output classes.
The MLP-based interaction models in the GAM (Eq. 3) always have architectures of 30 - 10 .
They are trained with (cid:96) 2 regularization of λ 2 = 1e − 5 and learning rate of 1e − 3 .
Because learning GAMs can be slow, we make a linear approximation of the univariate functions in Eq.
3, such that g i ( x i ) =
x i .
This approximation also allows us to make direct comparisons between Mah´e and linear LIME, since x i is exactly the linear part (Eq. 1).
All neural networks train with early stopping, and Level L is decided where validation performance does not improve more than 10% .
c ranges from 3 to 4 in our experiments.
5. C ONTEXT -D EPENDENT E XPLANATIONS 5.2. S YNTHETIC E XPERIMENTS
In order to evaluate Mah´e ’s context-dependent explanations, we ﬁrst compare them to state-of-themethods for local interaction interpretation.
A standard way to evaluate the accuracy of interaction detection and modeling methods has been to experiment on synthetic data because ground truth interactions are generally unknown in real-world data (Hooker, 2004;
Sorokina et al., 2008; Lou et al., 2013; Tsang et al., 2017).
Similar to Hooker (2007), we evaluate interactions in a subset region of a synthetic function domain.
We generate synthetic data using functions F 1 − F 4 (Table 1) with continuous features uniformly distributed between − 1 to 1 , train complex base models (as
[Table]
5.2. E VALUATING ON R EAL -W ORLD D ATA
In this section we demonstrate our approaches to evaluating Mah´e ’s context-dependent explanations on real-world data.
We ﬁrst evaluate the prediction performance of Mah´e on the test set of D as interactions are added in Eq.
3, i.e. K increases.
We run Mah´e 10 times on each of 40 randomly selected data instances from the test sets associated with DNA-CNN ( σ = 0 . 4 ), Sentiment-LSTM ( σ = 6 ), and ResNet152 ( σ = 0 . 4 ).
For Transformer ( σ = 6 ), stability is examined on a speciﬁc grammar ( cet ) translation, to be detailed in §5.3.
The local vicinity samples and model initializations in Mah´e are randomized in every trial.
Table 2 shows the average prediction error as K starts from 0 , which is linear LIME, and is increased to the last hierarchical level L .
As seen, including interactions results in signiﬁcant performance improvements.
An alternative approach to evaluating Mah´e is to ask human evaluaters to select between the better explanation linear LIME and Mah´e .
We recruit
Amazon Mechanical Turk users to select between explanations of Sentiment-LSTM.
To minimize the difference between the two explanations, Mah´e only shows the K = 1 interaction from the interaction ranking and aggregates its attribution (see
Figure 4) with that of subsumed features in the linear part of Mah´e (linear LIME, §5.1).
Only sentences for which there is a signiﬁcant difference between linear LIME and Mah´e are shown, i.e. there is any polarity difference between the attributions of linear and interaction explanations.
To reduce ambiguities of uninterpretable explanations arising from a misbehaving model - an issue also faced by Sundararajan et al.
(2017) in interpretation evaluation - we only show explanations of sentences that the model classiﬁed correctly.
Further details about this experiment can be found in
Supplementary B.
The result of this experiment is that the majority of preferred explanations ( 65% , p = 0 . 029 ) is with interactions, supporting the inclusion of interactions in hierarchical explanations.
5.2. H IERARCHICAL E XPLANATIONS
Examples of context-dependent explanations in hierarchical format for ResNet152, SentimentLSTM, and Transformer are shown in Figure 6, Table 5, and Table 6 respectively after page 8.
5. C ONTEXT -F REE E XPLANATIONS
In this section, we show examples of context-free explanations of interactions found by Mah´e .
We ﬁrst study English-to-French translations made with Transformer on a speciﬁc type of known inter-
[Table]
[Table] [Table]
[Table]
action, the presence of a French word for “this” or “that”, cet , which only appears when the noun it modiﬁes begins with a vowel.
The presence of cet is used as the prediction variable in local interaction extraction.
To minimize the sources of cet , we limit original sentence lengths to 15 words.
We modify Transformer (§4.2) through differentiating the max value of the cet output neurons over all translated words, and to have enough sentences, we perform translations on WikiText-103 (Merity et al., 2016) in addition to the WMT-14 test set.
The results of context-free experiments on cet interactions of adjacent English words are shown in Table 3.
The interactions always have positive polarities towards cet , and after modifying Transformer at a single data instance for a given interaction, its polarity almost always become negative.
Examples of new translations from the modiﬁed Transformer are shown in the “after” rows in Table 4, where cet now disappears from the translations.
Transformer BLEU score only decreases by an average percent difference of − 2 .
7% after modiﬁcation, indicating that the original learned representation stays largely intact.
Results for σ = 6 are shown with the avg.
pairwise edit distance between sentences being σ (cid:48)
= 10 .
[Figure]
In experiments on Sentiment-LSTM, we also observe that the polarities of certain interactions are almost always the same (see Figure 5) throughout all sentences in IMDB movie reviews data (Maas et al., 2011) and the test set of SST.
In this case, the words of matching interactions can be separated by any number of words in-between.
As before, whenever the model’s local interaction attribution at a single data instance is negated, the attribution is almost always the opposite sign for the rest of the sentences.
A notable insight about SentimentLSTM is that it appears to represent (too, bad) and (only, worse) as globally positive sentiments, and Mah´e ’s modiﬁcation in large part rectiﬁes this misbehavior (Figure 5).
The modiﬁcations to Sentiment-LSTM only cause an average reduction of 1 . 5% test accuracy.
Results for σ = 16 , σ (cid:48)
= 24 .are shown, and words in detected interactions are separated by 1 .
3 words on average.
Experiments on DNA-CNN and ResNet152 show similar results at ﬁxed interaction positions (§4.2).
For DNA-CNN, out of the 94 times a 6-way interaction of the CACGTG motif (Sharon et al., 2008) was detected in the test set, every time yielded a positive attribution polarity towards DNA-protein afﬁnity, and the same was true after modifying the model in the opposite polarity (cosine distance σ = 0 .
35 , σ (cid:48)
= 0 . 408 ).
For ResNet152, context-free interactions are also found (cosine distance σ = 0 .
4 , σ (cid:48)
= 0 . 663 ).
However, because superpixels are used, the interactions found may contain artifacts caused by superpixel segmenters, yielding less intuitive interactions (see Supplementary A).
5. L IMITATIONS
Although Mah´e obtains accurate local interactions on synthetic data using NID, there is no guarantee that NID ﬁnds correct interactions.
Mah´e faces common issues of model-agnostic perturbation methods in interpreting high-dimensional feature spaces, choice of perturbation distribution, and speed (Ribeiro et al., 2016; 2018).
Finally, an exhaustive search is used for context-free explanations.
C ONCLUSION
In this work, we proposed Mah´e , a model-agnostic framework of providing context-dependent and context-free explanations of local interactions.
Mah´e has demonstrated the capability of outperforming existing approaches to local interaction interpretation and has shown that local interactions can be context-free.
In future work, we wish to make the process of ﬁnding context-free interactions more efﬁcient, and study to what extent model behavior can be changed by editing its interactions or univariate effects.
Finally, we would like to study the interpretations provided by Mah´e more closely to ﬁnd new insights into structured data.
[Table]
R 2 =0.
linear LIME (level 1)
R 2 =0.
Mahé (level 2)
R 2 =0.
Mahé (level 3)
R 2 =0.
Mahé (level 4) -2.
(a) prediction: stretcher
[Table]
R 2 =0.
linear LIME (level 1)
R 2 =0.
Mahé (level 2)
R 2 =0.
Mahé (level 3)
R 2 =0.
Mahé (level 4) -0.
(b
[Figure]
prediction: window screen
[Table]
R 2 =0.
linear LIME (level 1)
R 2 =0.
Mahé (level 2)
R 2 =0.
Mahé (level 3)
R 2 =0.
Mahé (level 4) -2.
(c) prediction: Pomeranian
[Table] R 2 =0. linear LIME (level 1) R 2 =0.
Mahé (level 2)
R 2 =0.
Mahé (level 3)
R 2 =0. Mahé (level 4) -4. [Figure] [Table]
[Table]
S UPPLEMENTARY M ATERIALS
A
C ONTEXT -F REE EXPLANATIONS IN R ES N ET
[Figure]
B D ETAILS OF M ECHANICAL T URK EXPERIMENT
To determine whether human evaluators agree with the explanations provided by Mah´e , we recruit 60 Amazon Mechanical Turk users to label the more accurate explanation between linear LIME explanations and Mah´e explanations of Sentiment-LSTM (Figure 8).
To minimize the difference between the two explanations, Mah´e only shows the top- 1 interaction from the interaction ranking ( K = 1 , §4.1) and aggregates its attribution (see Figure 4) with that of subsumed features in the linear part of Mah´e (linear LIME, §5.1).
For the comparison, we show users randomly selected sentences of lengths 5 to 12 from the test set of SST with the following conditions.
We exclude sentences that do not have valid interactions, i.e. interactions are not detected.
Only sentences for which there is a signiﬁcant difference between linear LIME and Mah´e are shown, i.e. there is any polarity difference between the attributions of linear and interaction explanations.
To reduce ambiguities of uninterpretable explanations arising from a misbehaving model, we only show explanations of sentences that the model classiﬁed correctly.
Besides conditions on sentences, we only allow each user to select up to four explanations.
For a single sentence, a majority vote of the selections of 5 different users decides which explanation is more accurate.
The ordering of comparisons is randomized.
C M ORE EXAMPLES OF INTERACTIONS WITH CONSISTENT POLARITIES IN S ENTIMENT -LSTM
[Table]