W HAT ’ S IN THE B OX ? E XPLORING THE I NNER L IFE OF N EURAL N ETWORKS WITH R OBUST R ULES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel method for exploring how neurons within a neural network interact.
In particular, we consider activation values of a network for given data, and propose to mine noise-robust rules of the form X
→ Y , where X and Y are sets of neurons in different layers.
To ensure we obtain a small and non-redundant set of high quality rules, we formalize the problem in terms of the Minimum Description Length principle, by which we identify the best set of rules as the one that best compresses the activation data.
To discover good rule sets, we propose the unsupervised E XPLAI NN algorithm.
Extensive evaluation shows that our rules give clear insight in how networks perceive the world: they identify shared, resp.
class-speciﬁc traits, compositionality within the network, as well as locality in convolutional layers.
Our rules are easily interpretable, but also super-charge prototyping as they identify which groups of neurons to consider in unison.
I NTRODUCTION
Neural networks provide state of the art performance in many settings.
How they perform their tasks, how they perceive the world, and especially, how the neurons within the network operate in concert, however, remains largely elusive.
While there exists a plethora of methods for explaining neural networks, most focus on the mapping between input and output (e.g. model distillation) or only characterize individual neurons within the network (e.g. prototyping).
In this paper, we introduce a new method that explains how the neurons in a neural network interact.
In particular, we consider the activations of neurons in the network over a given dataset, and propose to characterize these in terms of rules X
→ Y , where X and Y are sets of neurons in different layers of the network.
A rule hence represents that neurons Y are typically active when neurons X are.
For robustness we explicitly allow for noise, and to ensure that we discover succinct and non-redundant rules we formalize the problem in terms of the Minimum Description Length principle.
To discover good rule sets, we propose the unsupervised E XPLAI NN method.
The rules that we discover give clear insight in how the network performs its task.
As we will see, they identify what the network deems similar and different between classes, show how information ﬂows within the network, and show which convolutional ﬁlters it expects to be active where.
Our rules are easily interpretable, give insight in the differences between datasets, show the effects of ﬁne-tuning, as well as super-charge prototyping as they tell which neurons to consider in unison.
Explaining neural nets is of widespread interest, and especially important with the emergence of applications in healthcare and autonomous driving.
In the interest of space we here shortly introduce the work most relevant to ours, while for more information we refer to surveys (Adadi & Berrada, 2018; Ras et al., 2018; Xie et al., 2020; Gilpin et al., 2018).
There exist several proposals for investigating how networks arrive at a decision for a given sample, with saliency mapping techniques for CNNs among the most prominent (Bach et al., 2015; Zhou et al., 2016; Sundararajan et al., 2017; Shrikumar et al., 2017).
Although these provide insight on what parts of the image are used, they are inherently limited to single samples, and do not reveal structure across multiple samples or classes.
For explaining the inner working of a CNN, research mostly focuses on feature visualization techniques (Olah et al., 2017) that produce visual representations of the information captured by neurons (Mordvintsev et al., 2015; Gatys et al., 2015).
Although these visualizations provide insight on how CNNs perceive the world (M. Øygard, 2016;
Olah et al., 2018)
it has been shown that concepts are often encoded over multiple neurons, and that inspecting individual neurons does not provide meaningful information about their role (Szegedy et al., 2013;
Bau et al., 2017).
How to ﬁnd such groups of neurons and how the information is routed between layers remains unsolved.
An orthogonal approach is that of model distillation, where we train easy to interpret white box models that mimic the decisions of a neural network (Ribeiro et al., 2016; Frosst & Hinton, 2017; Bastani et al., 2017; Tan et al., 2018).
Rules of the form (if–then) are easily interpretable, and hence a popular technique for model distillation (Taha & Ghosh, 1999; Lakkaraju et al., 2017).
Although some consider individual neurons to determine what rules to return (Robnik-ˇSikonja & Kononenko, 2008; ¨Ozbakır et al., 2010; Barakat & Diederich, 2005) all these techniques only yield rules that directly map input to output, and hence do not provide insight into how information ﬂows through the network.
Tran & dAvila Garcez (2018) mine association rules from Deep Belief Networks.
Their approach, however, suffers from the pattern explosion in typical in frequency based rule mining, and is not applicable to state of the art networks. Chu et al. (2018) aim to explain piecewise linear NNs utilizing polytope theory to derive decision features of the network. Providing strong guarantees, this approach is however limited to PLNNs of extremely small size ( < 20 neurons in hidden layers).
We instead propose to mine sets of rules to discover groups of neurons that act together across different layers in feed forward networks, and so reveal how information is composed and routed through the network to arrive at the output.
To discover rules over neuron activations, we need an unsupervised approach.
While a plethora of rule mining methods exists, either based on frequency (Agrawal & Srikant, 1994; Bayardo, 1998;
Moerchen et al., 2011) or statistical testing (H¨am¨al¨ainen, 2012;
Webb, 2010), these typically return millions of rules even for small datasets, thus undermining the goal of interpretability.
We therefore take a pattern set mining approach similar to GRAB (Fischer & Vreeken, 2019), where we are after that set of rules that maximizes a global criterion, rather than treating each rule independently.
Providing succinct and accurate sets of rules, GRAB is however limited to conjunctive expressions.
This traditional pattern language is too restrictive for our setting, as we are also after rules that explain shared patterns between clases, and are robust to the inherently noisy activation data, which both require a more expressive pattern language of conjunctions, approximate conjunctions, and disjunctions. We hence present E XPLAI NN, a non-parametric and unsupervised method that discovers sets of rules that can model these types of rules.
T HEORY
We ﬁrst informally discuss how to discover association rules between neurons.
We then formally introduce the concept of robust rules, and how to ﬁnd them for arbitrary binary datasets, last, we show how to combine these ideas to reveal how neurons are orchestrated within feedforward networks.
2. P ATTERNS OF NEURON CO - ACTIVATION
Similar to neurons in the brain, when they are active, artiﬁcial neurons also send information along their outgoing edges.
To understand ﬂow of information through the network, it is hence essential to understand the activation patterns of neurons between layers.
Our key idea is to use recent advances in pattern mining to discover a succinct and non-redundant set of rules that together describe the activation patterns found for a given dataset.
For two layers I i , I j , these rules X → Y, X
⊂
I
i , Y ⊂
I j express that the set of neurons Y are usually co-activated when neurons X are co-activated.
That is, such a rule provides us local information about co-activations within, as well as the dependence of neurons between layers.
Starting from the output layer, we discover rules between consecutive layers I j , I j − 1 .
Discovering overlapping rules between layers X → Y and Y
→ Z, X ⊂ I j , Y ⊂
I j − 1 , Z ⊂ I j − 2 , allows us to trace how information ﬂows through the entire network.
Before we can mine rules between two sets of neurons – e.g. layers – I i and I j of a network, we have to obtain its binarized activations for a given data set D = { d k =
( s k , o k ) } .
In particular, for each sample s k and neuron set I
i , we take the tensor of activations φ i and binarize it to φ bi . For networks with ReLU activations, which binarize naturally at threshold 0 , we might lose some information about activation strength that is eventually used by subsequent layers, but it allows us to derive crisp symbolic, and directly interpretable statements on how neurons interact.
Furthermore, binarization reﬂects the natural on/off state of biological neurons, also captured by smooth step functions such as sigmoid or tanh used in artiﬁcal neural networks.
We gather the binarized activations into a dataset where corresponds to the is a binary vector of length | I i | + | I j | .
See Fig.
1 for a toy example
Next, given D , our goal is to ﬁnd that set of rules that together succinctly describe the observed activations.
The Minimum Description Length (MDL) principle lends itself as an objective to ﬁnd such sets.
MDL is a statistically well-founded and computable approximation of Kolmogorov Complexity (Li & Vit ´ anyi, 1993).
First introduced by Rissanen (1978), the essential idea is that the model
M ∗ ∈ M that best describes data D is the model that losslessly encodes D using the fewest bits
M ∗ = arg min M ∈M L ( D, M ) .
Here, our model class M is the superset of all possible rules over
D , and by MDL we identify the best model M ∗ as the one that compresses the data best.
Traditionally, rule mining is restricted to conjunctions over items, which is not sufﬁcient for our application; neuron activations tend to be noisy, labels are inherently mutually exclusive, and hence we consider an extended language that allows for partial disjunctions of items (neurons, labels) and introduce a codelength function
L ( D, M ) to instantiate MDL for our model class of rule sets.
2. MDL FOR R OBUST R ULES
Our goal is to ﬁnd a set of rules M that, in terms of description
length
L ( D, M ) , best describes a binary database D
= { t | t ⊂ I} that consists of transactions t that are subsets of items I .
Each rule is of the form X
→ Y, X, Y ⊂ I , and indicates that Y is strongly associated with, i.e. occurs mostly in transactions where X is present.
We say a rule X
→ Y applies to a transaction t iff X ⊂ t and say a rule holds for t if additionally Y ⊂ t .
We indicate these transactions sets as
T X
= { i | t i ∈ D. X
⊂ t i } , respectively T Y | X = { i | t i ∈ T X .
Y ⊂
t i } .
Based on these deﬁnitions of rule transaction sets, we can now formally introduce our codelength function
L ( D, M ) .
Baseline model Our base model M ind = {∅ → I | ∀ I ∈ I} consists of singleton rules only, i.e. it models that all items I are generated independently.
To send the n transactions of D using
M ind , we simply send for each item I in which out of all transactions in the database it appears.
We can do so optimally using a log binomial code, which is given by log (cid:0) | T ∅ | | T I |∅ | (cid:1) = log (cid:0) n | T I | (cid:1) .
To unambiguously decode, the recipient needs to know each | T I | , which we can optimally encode via the parametric complexities of the binomials, which are deﬁned as
[Equation]
and can be computed in linear time (Kontkanen & Myllym¨aki, 2007).
We thus have L ( D, M ind ) =
(cid:80)
I ∈I (cid:0) log (cid:0) n | T I | (cid:1) + L pc ( n ) (cid:1) .
M ind serves as our baseline model, and its singleton rules are a required part of any more complex model as they ensure we can always send any data over I .
Non-trivial models A non-trivial model M contains rules of the form X
→ Y, X, Y ⊂
I that are not part of M ind .
The idea is that we ﬁrst transmit the data for where these non-trivial rules hold, and then send the remaining data using M ind .
To determine where such a rule applies, the receiver needs to know where X holds, and hence the data over X needs to be transmitted ﬁrst.
To ensure that we can decode the data, we only consider models M for which the directed graph G
=
( I , E ) is acyclic, in which there exists an edge between two items i 1 , i
iff they occur in the head and tail of a rule, that is E
= { ( i 1 , i 2 ) | ∃ X → Y ∈ M.
i 1 ∈ X ∧ i 2 ∈ Y } .
We thus get a codelength
[Equation] [Equation]
(cid:18) (cid:88)
[Equation] log [Equation]
|
[Equation]
| (cid:19)(cid:19) + (cid:88)
[Equation]
I ∈ M ind log (cid:18) n | T (cid:48)
I | (cid:19) , where T (cid:48)
I = { t ∈ D | ( I ∈ t ) ∧ (
∀ X
→ Y ∈ M. I ∈ Y =
⇒ t (cid:54)∈ T Y | X ) } is a modiﬁed transaction set containing all transactions with item I not covered by any non-trivial rule.
In addition to the parametric complexities of the binomial codes, the model cost of a non-trivial model also includes the cost of transmitting the non-trivial rules.
To transmit a rule X → Y , we ﬁrst send the cardinalities of X resp.
Y using the universal code for integers L N (Rissanen, 1983).
For n ≥ 1 , this is deﬁned as L N ( z ) =
log ∗ z
+ log c 0 with log ∗ ( z ) = log z
+ log log z + . .
. , summing only over the positive components (Rissanen, 1983).
To satisfy the Kraft equality we set c 0 = 2 . 865064 .
Knowing the cardinalities, we can then send the items of X resp.
Y one by one using an optimal preﬁx code given by L ( X ) = − (cid:80) x ∈ X log | [Equation] | [Equation]
I
[Equation]
| T I | .
For a particular rule
X → Y ∈ M , the model costs for a rule, respectively the full model thus amount to
[Equation]
We provide an example in App. A.1.
With these deﬁnitions, we have an MDL score that identiﬁes the best rule set M ∗ for data D as
[Equation]
where M is the space of all possible rule sets over the items in D .
Robust Rules In real world applications, which are inherently noisy, we need a score that is robust and can approximate where rules hold and apply.
In addition, as output neurons are only active mutually exclusively, rules need to be able to model disjunctions.
The key problem with noisy data is that a single missing item in a transaction can cause a whole rule not to hold or apply.
To discover rules that generalize well, we need to explicitly account for noise.
The idea is to let rules apply, and hold, approximately.
Specifying how many items l , and k , out of all items in the rule head, respectively tail, need to be part of a transaction, we relax the original rule deﬁnition to account for random noise (see App.
Figs.
Setting l = 1 and k
=
1 coincidentally corresponds to a disjunction of items in the head and tail of the rule X → Y , thus allowing to model output neurons correctly, and l = | X | and k
= | Y | correspond to the original stringent rule deﬁnition.
Varying between the two extremes accounts for varying levels of noise.
The optimal l and k are those that minimize the MDL score.
To ensure a lossless encoding, we need to make sure that the receiver can reconstruct the original data.
Thus, for the relaxed head deﬁnition, for each rule we send the number of items l for the rule to apply using L N ( l ) bits.
Knowing each l , the receiver can reconstruct where each rule applies.
Sending where a rule holds now leaves the receiver with an approximation of the data.
To be able to reconstruct the actual data, Fischer & Vreeken (2019) introduced error matrices that when XORed with the approximation yield the original data.
These two matrices X + X → Y , and X − X
→ Y correct for the errors made in the part where the rule applies and holds, respectively applies but does not hold (for an example see App.
Fig. 7).
These error matrices are part of the model M and have to be transmitted with an adapted L ( D, M ) , which we review in App.
A.2.
Complexity of the search To discover rules over the activations of two layers
I i , I j , we have to explore all rules formed by subsets of neurons in I i for the head, combined with any subset of neurons of I j for the tail.
There exist 2 |
I i | × 2 |
I j | such rules, and hence 2 2 | Ii | + | Ij | distinct models would need to be explored.
Fischer & Vreeken (2019) showed that the rule set search space does not lend itself to efﬁcient search as it is neither monotone nor submodular, the counterexamples also holding for our model deﬁnition.
In fact, for robust rules, we additionally have to consider where rules should apply respectively hold – optimizing k and l – which results in approximately 2 | I i |×| I j |× 2 | Ii | + | Ij | models (detailed derivation in App.
A.4).
Exhaustive search is therewith infeasible, which is why we present E XPLAI NN, a heuristic algorithm to efﬁciently discover good sets of rules.
2. D ISCOVERING GOOD RULE SETS WITH E XPLAI NN
E XPLAI NN is based on the idea of iteratively reﬁning the current model by merging and reﬁning already selected rules.
The key insight of the algorithm is that for a rule X → Y to summarize the data well, also rules X → Y (cid:48) with only part of the tail, Y (cid:48)
⊂ Y , should summarize well, as all tail items should be similarly co-occuring with head X .
Starting from the baseline model M ind we iteratively and greedily search for better models until we can no longer improve the MDL score.
As search steps we consider either introducing a new rule to M , by taking a good set of items
X ⊂ I
i for the head and a single item A ∈
I j for the tail and reﬁne the model to M (cid:48) =
M ⊕{ X → A } , seeing if it decreases the overall MDL costs (Eq. 2.2).
Or, we merge two existing rules r
= X → Y 1 ∈ M and r 2 =
X → Y 2 ∈ M , to form a new rule r (cid:48)
=
X → Y 1 ∪ Y 2 and reﬁne the model to
M (cid:48)
=
M ⊕ { r (cid:48) }
=
( M \ { r 1 , r 2 } ) ∪ { r (cid:48) } .
For a rule r (cid:48) , the reﬁnement operator ⊕ is adding the rule r (cid:48)
=
X → Y to M , removing the merged rules that led to r (cid:48) , if any, and updating the singleton transaction lists T A for all items A ∈ Y , removing all transactions where r (cid:48) holds.
To make this search scheme applicable to the scale of a typical neural net, we have to discuss how to efﬁciently search for candidate rules, with heads that can express anything from conjunctions to disjunctions.
Immediately after, we present the full algorithm E XPLAI NN for mining high quality rule sets for two arbitrary sets of neurons (e.g. layers) of a network.
Searching for candidates A key component of E XPLAI NN is the candidate generation process, which implements the two possible steps of generating new and merging existing rules.
Given two layers I i , I j , to efﬁciently discover rules that are both robust to noise, and may include disjunctively active neurons in the head, we can not enumerate all possible rule heads for each individual tail neuron, as this would result in | I j | × 2 |
I i | many rules.
Instead, we keep a list H y for each item y ∈
I j , storing all head neurons x ∈ I i for which y is frequently active when x is active, that is σ x,y = | T x ∩ T y | | T x | > θ , where θ is a conﬁdence threshold. We consider a rule X → Y to be good, if when neurons X are active, the neurons Y are also likely to be active, which is directly represented by the conﬁdence θ .
With parameter µ we account for early decisions on rule merging that later hinder us to see a more general trend.
The lists are sorted decreasing on σ .
We search in each H y for the rule with highest gain over all unions of ﬁrst t = 1 . . .
| H y | neurons in the list.
We add that rule
X → y with highest gain to the candidate list (see App.
Alg. 1).
To compute the gain, we consider all possible values k = 1 . . .
| X | to determine for which transactions T kX
= { t ∈ D | | X ∩ t | ≥ k } the rule should robustly apply, where k = 1 corresponds to disjunction and k = | X | to conjunction.
For an individual neuron y , such a rule would be optimal, but, our goal is to discover groups of neurons that act in concert.
To this end we hence iteratively merge rules with similar heads – similar, rather than same, as this gives robustness both against noise in the data, as well as earlier merging decisions of the algorithm.
For two rules X 1 → Y 1 , X 2 → Y 2 with symmetric difference
X 1 (cid:9) X 2 =
( X 1 \ X 2 ) ∪ ( X 2 \ X 1 ) , we consider possible candidate rules X 1 ∪ X 2 → Y 1 ∪ Y 2 and
X 1 ∩ X 2 → Y 1 ∪ Y 2 , iff | X 1 (cid:9) X 2 | ≤ µ for some threshold µ ∈ N .
For example, µ = 1 corresponds to the case that one head has one label more than the other, all other labels are the same.
Both parameters θ and µ are simple, yet effective runtime optimizations, the best results with respect to MDL will always be obtained with the largest search space, i.e. with θ and µ set to 0 , respectively | X 1 | + | X 2 | .
Besides impacting run-time, many of those rules may be uninteresting from a userperspective, µ and θ allow to directly instruct E XPLAI NN to ignore such rules.
E XPLAI NN
Assembling the above pieces, we arrive at E XPLAI NN, which given two sets of neurons I i , I j and a database of activations of these neurons, yields a heuristic approximation to the
MDL optimal model M ∗ .
By ﬁrst introducing all relevant single neuron rules, it then proceeds by iteratively merging existing rules using the approach described above, until it can achieve no more gain.
For efﬁciency, we separate the generation of the new rules from the merging of existing rules.
In practice, this does not harm performance, as we allow merging of similar heads and can thus revert too greedy decisions introduced by new rules with too large or small heads.
Furthermore, by observing that independent rules X 1 → Y 1 , X 2 → Y 2 , Y 1 ∪ Y 2 = ∅ do not inﬂuence each others impact on the codelength, we can add all independent rules with the highest respective gain at once.
We give the pseudocode as App.
Alg.
Complexity of E XPLAI NN
The generation of new rules results in time O ( n × | I j | × | I i | 3 ) , by iterating over each neuron in I j , and considering each subset of the most overlapping neurons in I i , and considering each threshold k = 1 . . .
| I i | for when the rule should apply, and the factor n from intersecting transaction lists T to compute the gain.
We can have at most | I j | generated rules before considering rule merges, and in every iteration of merging we combine two rules, reducing the rule set size by 1.
In each such step, we consider | I j | 2 merges, for each of which we compute the gain considering noisy head and tail.
We thus have a worst case runtime of O ( n × | I j | 4 ×
| I i | ).
As MDL ensures we consider succinct models that capture only the relevant structure in the data, E XPLAI NN is in practice much faster and easily scales to several thousands of neurons.
E XPERIMENTS
In this section we empirically evaluate E XPLAI NN on synthetic data with known ground truth and real world data to explore how CNNs perceive the world.
We focus on CNNs as they count towards the most widespread use of feedforward networks and naturally lend themselves for visualization, which helps us to interpret the discovered rules.
We consider CNNs of different complexity, including GoogLeNet and VGG-S. We compare to traditional prototyping and activation map approaches on MNIST (LeCun & Cortes, 2010), and examine which information is used how to arrive at classiﬁcation for ImageNet (Russakovsky et al., 2014).
Finally, we investigate the effect of ﬁne-tuning in transfer learning on the Oxford Flower data (Nilsback & Zisserman, 2008).
We will make the implementation of E XPLAI NN publicly available upon publication.
For the below experiments, running on commodity hardware E XPLAI NN took minutes for MNIST and Flower , and 6 hours for ImageNet , yielding rule sets of hundreds up to 3000 rules, depending on data and choice of layers. 3. R ECOVERING GROUND TRUTH
To evaluate how well E XPLAI NN is able to recover the ground truth from data, we ﬁrst generated synthetic binary data sets of size n = 10000 samples initially ﬁlled with zeros with m = { 10 , 50 , 100 , 200 , 300 , 500 } planted rules with up to 5 items in head and tail, respectively.
For each rule, the frequency is drawn from U ( . 02 , .
08) , the conﬁdence is drawn uniformly from U ( . 5 , 1) . We introduce noise by ﬂipping 0 . 1% of the entries chosen uniformly at random.
Additionally, 5 noise features with the same frequency as the rule features are introduced.
Fischer &
Vreeken (2019) showed that such an MDL model works for conjunctive rules, hence we will focus on investigating performance for mutually exclusive rule heads and noise.
In the ﬁrst set of experiments, we set head items to 1 in a mutual exclusive fashion, in line of ﬁnding rules over the NN
[Figure]
[Figure]
output labels.
E XPLAI NN achieves high recall and precision (see Figure 2) in terms of retrieving exact ground truth rules, and does not retrieve any redundant rules.
Next, we investigate the impact of noise on the performance, generating data of n = 10000 samples and m = 100 rules in a similar setting as above, with head items now initially set co-occuring (conjunctive), but adding varying levels of noise with α = { 0 . 1% , 0 .
5% , 1% , 3% , 5% , 10% } of bitﬂips in the matrix, where 10% noise means more noise than actual signal.
We report the results in Figure 2.
As expected by its design,
E XPLAI NN is fairly robust to noise, even when facing almost the same amount of noise and signal.
3. H OW IS INFORMATION FILTERED
As a ﬁrst simple example we consider MNIST .
We train a simple CNN of 2 convolutional and one fully connected layer using Keras, achieving 99% classiﬁcation accuracy on test data (see App.
B. for details).
We are interested in what the individual ﬁlters learn about the digits, and how E XPLAI NN reveals shared features across several classes.
We compare to average activation maps and single neuron prototypes.
Examining different ﬁlters, we observe that average class activation maps does not reveal the purpose of a ﬁlter, whereas the rules that E XPLAI NN discovers identify which pixels together trigger a ﬁlter.
As an example, consider Fig.
3, in which for ﬁlter 2 in layer 1 we visualize the discovered rules on top of the average activation maps (which just represents the original numbers).
Whereas the rules identify shared structure (indicated by pixels of the same colour, in the same location, for different classes) such as top-left curve for 0, 4, and 9 (depicted in orange), the prototype (see App.
Fig.
9) looks like a maze and does not reveal any insight.
Similarly, for ﬁlterin layer 2, we directly see from the discovered rules that this ﬁlter detects horizontal edges in a class speciﬁc manner, whereas prototyping and activation maps again fail to reveal this information (see
App.
Fig. 10).
Interestingly, we also ﬁnd ﬁlters that learn the negative of an image, with activated areas corresponding to the background of the digit (see App.
Fig.
3. H OW DOES INFORMATION FLOW
Next, we investigate a more complex setting and examine the activations for the ImageNet data set of pretrained VGG-S and GoogLeNet architectures (Chatﬁeld et al., 2014; Szegedy et al., 2015).
We here focus on VGG-S, as for this network there exist optimized prototyping methods that yield highly interpretable images for sets of neurons (M. Øygard, 2016).
We provide the results for GoogLeNet in App. B.2.1.
Starting with rules from the labels to the last layer, we observe that E XPLAI NN yields complex rules spanning multiple labels and multiple neurons in the last fully connected layer (FC7), which together encode the information shared between labels.
Examples include the faces of certain dog breeds, as well as the coloured beaks of birds (see App.
B.2.2).
Notably, if we visualize these neurons individually, it is hard to extract anything meaningful from the images: the information is really encoded in the set of neurons that act together (see App.
Fig. 14).
We also observe cases where rules describe how the network discriminates between very similar classes.
We give an example in
[Figure]
husky AND Malamute
[Figure]
Fig.
4 for the neurons E XPLAI NN discovers to be associated with huskies, with malamutes, and for both classes together.
These dog breeds are visually very similar, sharing a black–white fur pattern, as well as head and ear shapes.
These traits are reﬂected by the neurons corresponding to the rule for both breeds.
Looking closer, we can see that distinct traits, the more deﬁned snout and ears of the husky, respectively the ﬂuffy fur of the malamute, are picked up by the neuron sets discovered for the individual dog breeds.
Beside discovering what shared and distinct traits the network has learned for classes, unlike with class prototypes we also ﬁnd out when it learns differences across samples of the same class.
For example, for the dog breed Great Danes, we discover not one but three rules that upon visualization each correspond to visually very different sub-breeds (see App. Fig. 15).
Next we investigate the information ﬂow within the network.
Starting with the E XPLAI NN rules X → Y from output layer to last fully connected layer FC7, we apply E XPLAI NN to discover rules Y → Z between FC7 and FC6, where we restrict the heads Y to be groups of neurons found as tails in the previous run. We recursively apply this process until we covered a convolutional layer.
This gives us traces of neuronal activity by chaining rules X
→ Y → Z discovered in the iterative runs.
We visualize one such trace in Fig.
5, which gives insight in how the network perceives totem poles.
We ﬁnd that the relevant neurons of FC7 and FC6 yield prototypes that show typical totem pole heads with animalistic faces.
The neuron sets found for the different ﬁlters of the last convolutional layer CONV5, together detect speciﬁc parts of the class, including the vertical pole, and the rooftoplike structure, decorated with animalistic shapes with eyes, that is typically found at the top of a totem.
We provide another example of an information trace in App.
Fig.
16, which shows how the network learns to detect features located typically at certain parts of the image due to the very prevalent side-view photos taken from dogs, a clear sign of overﬁtting by the neural network.
3. W HAT IS THE EFFECT OF FINE TUNING
Finally, we show that E XPLAI NN can elucidate the effect of ﬁne-tuning in transfer learning.
For this we consider Oxford Flower data Nilsback & Zisserman (2008), which consists of 8k images of ﬂowers of 102 classes.
For investigation, we consider both the vanilla VGG-S network trained on
ImageNet from above, and a ﬁne-tuned version from the Caffee model zoo 1 .
We run E XPLAI NN to obtain rules between the output and the ﬁnal layer of both networks, and give an example in
App.
Fig.
The visualizations show, as expected, a strong emphasize on colour and shape of the corresponding class of ﬂowers.
Interestingly, the visualizations of the same neurons for the original
VGG-S network show almost identical shapes and pattern, but with less intense colour.
Furthermore, we observe that some prototypes show animal-like features such as eyes or beaks. [Figure]
D ISCUSSION AND C ONCLUSION
The experiments show that E XPLAI NN is uniquely able to discover distinct groups of neurons that together capture traits shared and distinct between classes, within-class heterogeneity, and how parts of ﬁlters are used to detect shared features, segment background, or detect edges locally.
None of these are revealed by activation maps, which miss the local information that patterns can provide, nor by saliency maps, which show where in an individual input image the network pays attention.
Prototyping is a great tool for visualizing what information neurons encode, but, by itself is limited by the massive number of possible combinations of neurons, requiring already thousands of hours to painstakingly handpick and connect the information of just individual neurons (Olah et al., 2020). Combining E XPLAI NN with prototyping permits exploring networks beyond single neurons, by automatically discovering which neurons act in concert, which information they encode, and how information ﬂows through the network. In particular, we discover distinct groups of neurons in fully connected layers that capture shared respectively distinct traits across classes, which helps in understanding how the network learns generality but still can discriminate between classes.
Due to the local information that our rules provide, we can also detect differences in the perception across samples of a single class, where for example different groups of neurons describe visually different sub-breeds of a class of dogs.
By connecting rules that we ﬁnd across several layers, we can trace how information is gathered and combined to arrive at a classiﬁcation, from ﬁlters that detect typical class speciﬁc features in the image, through fully connected layers where multiple neurons together encode the combined information, up to the ﬁnal classiﬁcation output.
Applying E XPLAI NN to investigate the impact of ﬁne-tuning in transfer learning, we found that for the groups of neurons in the given ﬁne-tuned CNN, surprisingly, the contained information is almost identical to the original CNN, but capturing the traits of the new classes almost perfectly.
For the given transfer task the ﬁne-tuning thus mostly resulted in routing information differently, rather than having to learn to detect new features.
Overall, E XPLAI NN performs well and convincingly answers ‘what’s in the box?’.
It ﬁnds surprising results that help to understand how CNNs perceive the world.
While many important tasks are solved by such networks, recurrent neural networks play an important role in e.g. language processing.
Although rules can likely also help to understand what these models learn, these networks encode an entirely different type of information that is inherently hard to understand and visualize, and hence an exciting challenge for future work.
Here, our main interest was characterizing information ﬂow through neural networks, and hence, we focused on subsequent layers.
E XPLAI NN, however, operates on arbitrary sets of neurons, and hence also naturally allows investigating e.g. residual networks, where the previous two layers contribute information.
E XPLAI NN currently scales to thousands of neurons.
It will make for engaging future work to scale it up to consider entire networks at once.
A MDL FOR ROBUST RULES
In this section we will give extended examples on how to compute the MDL score for a given database and set of rules, elaborate on the error encoding for the rule tails and give a visual toy example on the impact of the extended pattern language for the rule head.
[Figure]
For the given example in Fig.
6, we will now compute the codelength L ( D, M )
=
L ( M ) + L ( D |
M ) of transmitting the whole database D using M ∪ M ind .
Here, we will stick with the simple encoding without error matrices, to make the process of computation more understandable.
For reference, we ﬁrst compute the baseline model, which is given by
[Equation]
log [Equation] (cid:19)
[Equation]
log (cid:18) (cid:19) + 2 log (cid:18) (cid:19) + log (cid:18) (cid:19)
≈ 14 .
+ 93 .
47 + 179 .
64 + 87 .
93 = 375 . 92 .
Thus, sending the data with just the baseline model costs 375 .
92 bits.
Now, we will compute
L ( D, M ∪ M ind ) , we will start with the costs of sending the data L ( D | M ∪ M ind ) [Equation]
log (cid:18) [Equation]
| | [Equation] | (cid:19)(cid:33) + (cid:32) (cid:88)
I ∈I log (cid:18) | D | | T (cid:48)
I | (cid:19)(cid:33) =
log (cid:18) (cid:19)
+ log (cid:18) (cid:19) + log (cid:18) (cid:19)
+ log (cid:18) (cid:19) + log (cid:18) (cid:19)
≈ 29 .
+ 93 .
47 + 26 .
17 + 17 .
30 + 89 .
82 = 256 .
The model costs are composed of the parametric complexities for the (adapted) baseline rules, plus the costs of transmitting what the rule is composed of along with its parametric complexity.
We thus get
[Equation]
[-14 . 88 + 1 . 
 + 2 . 
 + 1 . 
 + 2 . 
-][Equation]
[Equation]
Hence, the model with the complex rule has a smaller codelengh than the baseline, with L ( D, M ∪
M ind )
= 284 .
44 bits.
A. T HE ERROR ENCODING FOR TAILS
For the error encoding for tails, which allow to discover rules in noisy settings (compare Fig.
7a,b), we send where a rule X
→ Y approximately holds according to some parameter k , which deﬁnes the number of items of the tail that have to be present in the transaction.
The errors made by this approximation are then accounted for by sending error correcting matrices
X − X
→ Y and X + X → Y , which account for the destructive, respectively additive noise in the are where the rule applies (compare
Fig. 7c).
Let us ﬁrst assume we are given a k , we will later show how we can optimize for k .
We redeﬁne the transaction sets T Y | X
= { t ∈ D | ( X ⊂ t )
∧ ( | Y ∩ t | ≥ k ) } , which corresponds to the transactions where the rule approximately holds.
We will now slightly abuse notation and indicate the binary input matrix that correspond to D by D , and we subset this matrix using the transaction id lists and item subsets.
Both of these are sets of indices that indicate which rows, respectively columns to use of the matrix.
For example, the submatrix where X holds is given by D [ T X , X ] .
We can now deﬁne the error correcting matrices to be X − X
→ Y = D [ T Y | X , Y ] ×(cid:13) 1 | T Y | X |×| Y | , and
X − X
→ Y = D
[ T X \ T Y | X , Y ] , where ×(cid:13) is the element-wise XOR operator and 1 i × j is a matrix of size i × j ﬁlled with ones.
The receiver, knowing T X and T Y | X , can then reconstruct the original data D
[ T Y | X , Y ] =
| T Y | X |×| Y | ×(cid:13)
X
− X
→ Y , respectively D [ T X \ T Y
| X , Y ] =
X + X → Y .
While this explains the concept of how error correcting matrices can be used to reconstruct the original input, which hence deﬁne a lossless encoding, we are mainly interested in the codelength functions.
To adapt the data costs, we now additionally send the two error matrices, which we can do using binomial codes.
Hence, we get
[Equation]
with the second line providing the codelength of the error matrices, and |X| indicating the number of ones in X .
Our model M now not only consists of rules M ∪ M ind , but also of the set of error correcting matrices.
As the submatrix to which we need to apply the matrix is fully deﬁned by T X , T Y | X and
Y of the corresponding rule, also deﬁning its size, the only adaptation we need for the model costs is the parametric complexities induced by the codes for transmitting the data.
This yields
L
[Equation]
This completes the MDL costs for rules robust to noise in the tail for a given k .
To optimize k , the crucial insight is that the codelength of individual complex rules are independent, as is the data cost.
That means we can optimize a k for each rule separately.
Thus, for a given rule X → Y we can enumerate all | Y | many models for the different thresholds k and let MDL decide which one ﬁts the data best.
A. T HE IMPACT OF THE EXTENDED PATTERN LANGUAGE
Extending the pattern language for rule heads is crucial to be applicable for tracing activation patterns through a neural network.
First of all, we need to start from labels, which are inherently activated mutually exclusive, as we only have a single label as classiﬁcation.
To ﬁnd shared features of labels, it is essential to be able to express disjunctions with rule heads.
Furthermore, the data as well as activation patterns across the data are very noisy.
Thus, determining where a rule applies just based on conjunctions of features can give a very twisted look of the data at hand, as visualized in Fig.
That is the reason to introduce a more ﬂexible language similar to approximate rule tails, which solves these issues.
A. S EARCH COMPLEXITY
The size of the search space implied by our model class M is in O
(2 | I
i |×| I j |× 2 | Ii | + | Ij | ) .
For two layers I i , I j , we enumerate all possible rules by (cid:16) | I i | (cid:88) k
= k × (cid:18)
| I i | k (cid:19)(cid:17) (cid:124) (cid:123)(cid:122) (cid:125) Possibilities for head × (cid:16) | I j | (cid:88) l = l × (cid:18) | I j | l (cid:19)(cid:17) (cid:124) (cid:123)(cid:122) (cid:125) Possibilities for tail ≤ |
I i | (cid:16) | I i | (cid:88)
k = (cid:18) | I i | k
(cid:19)(cid:17) × | I j | (cid:16) | I j | (cid:88) l = (cid:18) | I j | l (cid:19)(cid:17) = | I i | 2 | I i | × |
I j | 2 | I j | = | I j | | I i | 2 | I i | + | I j | , where the ﬁrst sum enumerates all heads of size k , the binomial coefﬁcient describes the ways of drawing heads of such size, and the term k is the number of models given by the robust head encoding.
Similarly, the second sum enumerates all tails of size l , the binomial coefﬁcient describes the drawing of such tails, and the term l is the number of ways to place the error correcting matrices for the robust tail encoding.
As in theory we can have any subset of these rules as a model, we thus get approximately 2 ( | I j |×|
I
i |× 2 | Ii | + | Ij | ) many different models.
A. A LGORITHM P SEUDOCODE
E XPLAI NN explores the search space of rule sets in an iterative fashion, either generating new rules with a single item in the tail, or merging two existing rules, thus generating more complex rules with multiple items in the tail.
Using these two steps, we can generate all potential candidate rules to add to the model, and evaluate their respective gain in terms of MDL.
For a rule r (cid:48) , we will say model M (cid:48) =
M ⊕ r (cid:48) is the reﬁned model, with the reﬁnement operator ⊕ adding the rule r (cid:48)
=
X → Y to M , removing the merged rules that led to r (cid:48) , if any, and updating the singleton transaction lists T A for all items in the tail A ∈ Y .
Here, we will provide the pseudocode for the two candidate generation functions for new rules and for merging rules in the general setting alongside the complete algorithm of E XPLAI NN.
For generating a new rule with a head using the extended pattern language we use the approach described in the main paper, gathering all conﬁdence values for a given neuron A in I j for all potential head neurons
I i .
We keep all potential head neurons with conﬁdence value beyond θ in a list H A sorted descending on conﬁdence and merge the ﬁrst t neurons in the list to form the head.
Going over all t = 1 .. | H A | allows us to greedily optimize for the best of all relevant heads for the given item,. Input Dataset layers I
, I , Model , tail item A , threshold θ Output reﬁnement M head items, in decreasing order of conﬁdence
σ
// Update best rule set
The key component is hidden in the gain estimate in line 10, which for the given rule X
→ A determines the best value k of items in the head needed for a rule to apply.
That is, we test all for all transactions sets determining where the rule applies T k X
= { t ∈ D | | X ∪ t | ≥ k } which one gives the best gain.
To generate new rules going from the output layer to a hidden layer, we want to mine rules with disjunctive heads, which means we only have to consider T 1 X – corresponding to a disjunction – in the search process.
To generate candidates from existing rules in
M , we use an extended search scheme that allows to merge pairs of rules with approximately equal heads, hav- ing up to µ dissimilar items, measured by the symmetric set differences between (cid:9) .
Input: Dataset D , Model M , overlap threshold µ
Output: Candidates C sorted by gain ∆ 1 C ← ∅ ; //
Candidate rule merges 2 for r 1 =
X 1 → Y 1 ∈ M, r 2 =
X 2 → Y 2 ∈ M do if | X 1 (cid:9) X 2 | ≤ µ then ∆ ∩ ← L ( D, M ⊕ { X 1 ∩ X 2 → Y 1 ∪ Y 2 } ) − L ( D, M ) ; // Gain of adding conjunction of heads if ∆ ∩ < 0 then insert ( X 1 ∩ X 2 → Y 1 ∪ Y 2 , ∆ ∩ ) into C ; // Add to candidates ∆ ∪ ← L ( D, M ⊕ { X 1 ∪ X 2 → Y 1 ∪ Y 2 } ) − L ( D, M ) ; // Gain of adding disjunction of heads if ∆ ∪ < 0 then insert ( X 1 ∪ X 2 → Y 1 ∪ Y 2 , ∆ ∪ ) into C ; //
Add to candidates 10 return C
Using the candidate generation methods, we can now write down the pseudocode for E XPLAI NN, which iteratively generates candidates and commits to the candidate with highest gain, until there is no more candidate that yields any gain in terms of MDL. Algorithm
Input: Dataset D over layers I
i , I j , frequency threshold θ , overlap threshold µ
Output: Best model M ∗ 1 M ← {∅ → A | A ∈ I j } ; // Initialize model with baseline rule set 2 foreach A ∈
I j do
R (cid:48) ← G EN C ANDIDATE N EW
( D, M, A, θ ) ; // App.
Alg.
M (cid:48) ← M ⊕ R (cid:48) ;
if L ( D, M (cid:48) ) < L ( D, M ) then M ← M (cid:48)
B E XPERIMENTS AND D ATA
Here, we detail the setup and training of the individual networks, and provide further experimental results.
B. MNIST TRAINING
We trained a CNN on the MNIST data set using the Keras framework, using 60000 images for training and 10000 images as hold out test set for evaluation.
The network consists of 2 convolutional layers, with 20 ﬁlters in the ﬁrst layer and 40 ﬁlters in the second layer, each using 3x3 kernels and 2x2 maxpooling.
The convolutional layers are followed by a Dropout layer with dropout rate .
25 , and the ﬂattened outputs are passed on to a fully connected layer with 64 nodes with ReLU activations.
Then follows a dropout layer with rate .
5 and the output layer of size 10 with softmax activations.
The network was trained using AdaDelta with default parameters based on categorical cross entropy loss over 12 epochs using a batch size of 128.
We gathered binarized activations across all ﬁlters and applied E XPLAI NN to build rules from the output layer to the ﬁrst respectively second convolutional layer.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
B.
I MAGENET FURTHER RESULTS
B.2.
G OOG L E N ET RESULTS
L ab e l s
L a s t l a ye r
Curly Coated Retriever
[Figure]
Obelisk
[Figure]
B.2.
VGG SHARED NEURONS
L ab e l s
F C
Japanese spaniel
PekineseShih-Tzu
Lhasa
AffenpinscherPug
Brabancon griffon
Viaduct
Triumphal arch
Red-breasted merg.
Redshank
Oystercatcher
Tusker
Indian elephant
African elephant
[Figure]
We discovered many shared traits that the networks is able to pick up across classes, which are encapsulated in groups of neurons in the last layer.
For example, there are neurons that capture the red beaks of different birds, arch like structures of buildings, tusks of elephants and as an object itself, and the ugly face of a whole group of different dog breeds (see App.
Fig. 13).
Even if we would visualize all neurons individually and handpick them, without the knowledge about the rule, e.g. for the ugly dogs, it is very hard to interpret what they encode, and only few of them give hints about what the information is (see App.
Fig. 14).
In practice, we only visualize class prototypes, which can be very misleading, as shown in the next section.
B.2. R ESOLVING THE MEANING OF CLASS PROTOTYPES Class prototype for Great Danes
A standard technique to capture traits that a neural network learns about a class is prototyping the given class label.
These can offer hints on what the network learns globally about the class but very often lead to uninterpretable results.
We provide one such example prototype for the Great Dane class in Imagenet of the VGG network in App. Fig.
which does not provide any clue what the network learns.
The rules discovered by E XPLAI NN however show, that different groups of neurons in the last layer lead to the Great Dane classiﬁcation,
each encoding a distinct type of fur colour and pattern that appear with this breed.
The class prototype is a mixture of these different types, which explains the difﬁculty to interpret that prototype. [Figure]
B. O XFORD F LOWER DATA
B. A DDITIONAL RESULTS
In App. Fig.
18, 19, we provide additional results based on prototyping for rules found for ImageNet data and the VGG-S network.
We focus on rules with multiple neurons in the tail, as such class and multiclass prototypes can hardly be found by hand.
Overall, we observed that the larger the number of neurons in the tail, the sharper and more interesting the resulting protoype.
Furthermore, we found that for many prototypes spanning multiple classes, we discover multiple rules for some of these classes (e.g. Black Grouse ) and the protoypes indicate that only a fraction of information, such as patterns, a colored leg or beak, or a color patch, is used from each group of neurons such that together they arrive at the class prediction.
In App.
Fig.
18, the ﬁrst row of the panel are examples of neuron groups that learn typical shapes of objects, such as Sombrero or Gondola .
The second row contain groups of neurons capturing typical patterns and colors for individual classes, such as yellow patches on black skin of the Fire
Salamander , red caps with white dots of the Agaric mushroom, the typical leaf with red veins of Sorrel or the wings of a Monarch butterﬂy.
The third row contains common features between two classes that are together captured by the same group of neurons, like the arch-like structures and round rooftops found for certain Triumphal Archs and Mosques , the layered and intertwined worm-like shapes of many Fur Coats and the Gyromira mushroom, or the characteristic traditional covering of yurts and the front part of dogsleds.
In App.
Fig.
19, groups of neurons that are shared between multiple classes are visualized both revealing surprising similarities, as well as conﬁrming that the network learns similarities that we also use as a human.
In the ﬁrst row, the neurons described by the ﬁrst two images capture the typical shape and red color of the ears shared between the Red Fox and the Lesser Panda , respectively the insect legs and shiny turquoise color of the body of Tiger Beetles and Damselflies .
Intriguingly, the network also learns a roundish shape and distinct pattern between the Jackfruit and the Squirrel Monkey .
At this point, we would like to invite the reader to look up how the top of the head of such a monkey looks like, it resembles surprisingly well the size, color, shape, and texture of a Jackfruit.
For the last picture in the ﬁrst row of this panel, we see dotted wings that clearly are related with the associated labels Cabbage Butterfly , and Sulphut
Butterfly .
But opposed to visualizations related to other Butterﬂies (given in both panels), the wings are all oriented in a distinct way, which resemble the cap of a dotted mushroom, which might explain the association with the Agaric mushroom.
In the second row, we observe that the network captures common features shared between similar classes - in this case closely related animals - with the same set of neurons, which matches human intuition.
Sombrero
Gong
Barn
Gondola
Europ.
Fire Salam.
Agaric
Sorrel
Monarch
[Figure]
Rapeseed
Mosque
Triumphal Arch
Fur Coat
Gyromira
DogsledYurt
Red Fox
Lesser Panda
Tiger Beetle
Damselfly
Jackfruit
Squirrel Monkey
Cabbage Butterfly
Sulphur Butterfly
Agaric
A. S. S. Terrier
Bull Mastif
Saint Bernard
Water Ouzel
Black Grouse
Black Footed Ferret
Leopard
Snow Leopard
Cheetah
Ringlet
Cabbage Butterfly
Sulphur Butterfly