Original: Efﬁcient Decompositional Rule Extraction for Deep Neural Networks
Revised: Efﬁcient Decompositional Rule Extraction for Deep Neural Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: In recent years, there has been signiﬁcant work on increasing both interpretabilityand debuggability of a Deep Neural Network (DNN) by extracting a rule-basedmodel that approximates its decision boundary.
Revised: In recent years, there has been signiﬁcant work on increasing both interpretability and debuggability of a Deep Neural Network (DNN) by extracting a rule-based model that approximates its decision boundary.

Original: Nevertheless, current DNN ruleextraction methods that consider a DNN’s latent space when extracting rules,known as decompositional algorithms, are either restricted to single-layer DNNs orintractable as the size of the DNN or data grows.
Revised: Nevertheless, current DNN rule extraction methods that consider a DNN’s latent space when extracting rules, known as decompositional algorithms, are either restricted to single-layer DNNs or intractable as the size of the DNN or data grows.

Original: In this paper, we address theselimitations by introducing ECLAIRE, a novel polynomial-time rule extractionalgorithm capable of scaling to both large DNN architectures and large trainingdatasets.
Revised: In this paper, we address these limitations by introducing ECLAIRE, a novel polynomial-time rule extraction algorithm capable of scaling to both large DNN architectures and large training datasets.

Original: We evaluate ECLAIRE on a wide variety of tasks, ranging from breastcancer prognosis to particle detection, and show that it consistently extracts moreaccurate and comprehensible rule sets than the current state-of-the-art methodswhile using orders of magnitude less computational resources.
Revised: We evaluate ECLAIRE on a wide variety of tasks, ranging from breast cancer prognosis to particle detection, and show that it consistently extracts more accurate and comprehensible rule sets than the current state-of-the-art methods while using orders of magnitude less computational resources.

Original: We make all ofour methods available, including a rule set visualisation interface, through theopen-source REMIX library.
Revised: We make all of our methods available, including a rule set visualisation interface, through the open-source REMIX library (https://github.com/mateoespinosa/remix).

Original:  Introduction
Revised:  Introduction

Original: As the ﬁeld of Artiﬁcial Intelligence (AI) transitions from traditional symbolic methods into opaquemodels like Deep Neural Networks (DNNs) [1] there has been a great amount of concern over thepotential consequences of using these methods in safety-critical environments [
Revised: As the ﬁeld of Artiﬁcial Intelligence (AI) transitions from traditional symbolic methods into opaque models like Deep Neural Networks (DNNs) [1], there has been a great amount of concern over the potential consequences of using these methods in safety-critical environments [2, 3].

Original: The use ofsuch “black-box” models in practice  not onlycan lead to uncertainty and delays when debuggingand root-causing mistakes in a model’s behaviour (as in Uber’s infamous accident [5, 6]), but it alsogoes against the long-held conventions of using transparent, interpretable models for life threateningdecisions [7, 8].
Revised: The use of such “black-box” models in practice can not only lead to uncertainty and delays when debugging and root-causing mistakes in a model’s behaviour (as in Uber’s infamous accident [4, 5]), but it also goes against the long-held conventions of using transparent, interpretable models for life threatening decisions [6, 7].

Original: Furthermore, the lack of transparency in DNNs has been shown to lead to usermistrust [9, 10] and to models that misleadingly show promising results for the wrong reasons [11, 12].
Revised: Furthermore, the lack of transparency in DNNs has been shown to lead to user mistrust [8] and to models that misleadingly show promising results for the wrong reasons [9].

Original: Such opacity perpetuates a barrier for the practical use of DNNs despite their celebrated success intasks ranging from computer vision [13, 14, 15] and language modelling [16, 17, 18] to diagnosis[19, 20].
Revised: Such opacity perpetuates a barrier for the practical use of DNNs despite their celebrated success in tasks ranging from computer vision [10] and language modelling [11] to diagnosis [12].

Original: As a reaction to these concerns, a new line of work in the ﬁeld of eXplainable AI (XAI) has beendeveloped to explore ways of demystifying a DNN’s decision boundary and to facilitate debugging anddeployment of these models.
Revised: As a reaction to these concerns, a new line of work in the ﬁeld of eXplainable AI (XAI) has been developed to explore ways of demystifying a DNN’s decision boundary and to facilitate debugging and deployment of these models.

Original: A promising proposal in this direction is to make use of the latent spacelearnt by a DNN to construct a rule-based model (a set or list of IF-THEN rules) that approximatesits decision boundary [21, 22].
Revised: A promising proposal in this direction is to make use of the latent space learnt by a DNN to construct a rule-based model (a set or list of IF-THEN rules) that approximates its decision boundary [13, 14].

Original: Such approaches complement a DNN’s high performance by offeringtransparency and debuggability in the form of both “local” interpretability (i.e., understanding howa single prediction was made) as well as “global” interpretability (i.e., discovering global patternsin the data that explain how the DNN acts).
Revised: Such approaches complement a DNN’s high performance by offering transparency and debuggability in the form of both “ local ” interpretability (i.e., understanding how a single prediction was made) as well as “ global ” interpretability (i.e., discovering global patterns in the data that explain how the DNN acts).

Original: This is in contrast to existing XAI alternatives, suchas feature importance methods [23, 24], sample importance methods [25, 26] and counterfactualexplanations [27, 28], where a method only provides either local or global interpretability in theirbasic form.
Revised: This is in contrast to existing XAI alternatives, such as feature importance methods [15, 16], sample importance methods [17, 18] and counterfactual explanations [19, 20], where a method only provides either local or global interpretability in their basic form.

Original: Moreover, aided by publicly available rule set visualisation tools [29, 30], as well as the possibility of injecting expert hand-crafted rules into rule sets (e.g., as suggested in [31]), existingwork and open-source frameworks for deploying rule-based models can signiﬁcantly assist modeldevelopers, legal moderators, and model users with debugging and deploying a DNN by analysing itsapproximating rule-based model.
Revised: Moreover, aided by publicly available rule set visualisation tools [21, 22], as well as the possibility of injecting expert hand-crafted rules into rule sets (e.g., as suggested in [23]), existing work and open-source frameworks for deploying rule-based models can signiﬁcantly assist model developers, legal moderators, and model users with debugging and deploying a DNN by analysing its approximating rule-based model.

Original: Nevertheless, previous rule extraction methods that consider the inner workings of a DNN to constructtheir rules, commonly referred to as “decompositional” methods [32], tend to have severe limitations.
Revised: Nevertheless, previous rule extraction methods that consider the inner workings of a DNN to construct their rules, commonly referred to as “decompositional” methods [24], tend to have severe limitations.

Original: First, with very few exceptions, most decompositional algorithms have been designed to work onlyon shallow architectures [22, 21].
Revised: First, with very few exceptions, most decompositional algorithms have been designed to work only on shallow architectures [14, 13].

Original: Second, to the best of our knowledge, all of the approaches that areable to operate on non-trivial DNNs suffer from intractability issues and, therefore, are impracticalfor tasks that require very deep architectures or have a large training dataset [33, 31].
Revised: Second, to the best of our knowledge, all of the approaches that are able to operate on non-trivial DNNs suffer from intractability issues and, therefore, are impractical for tasks that require very deep architectures or have a large training dataset [25, 23].

Original: Both of theseare in stark contrast with the current direction in DNN architecture design where deep models trainedon large datasets dominate over shallow models trained with small datasets [34].
Revised: Both of these are in stark contrast with the current direction in DNN architecture design where deep models trained on large datasets dominate over shallow models trained with small datasets [26].

Original: In this paper we address these limitations by proposing ECLAIRE, a novel decompositional ruleextraction algorithm for DNNs.
Revised: In this paper we address these limitations by proposing ECLAIRE, a novel decompositional rule extraction algorithm for DNNs.

Original: Our method, depicted in Figure 1, avoids the intractability commonto other decompositional methods by making use of a two-step process that efﬁciently uses a DNN’slatent space to guide its rule extraction.
Revised: Our method, depicted in Figure 1, avoids the intractability common to other decompositional methods by making use of a two-step process that efﬁciently uses a DNN’s latent space to guide its rule extraction.

Original: Our main contributions can be summarised as follows:
Revised: Our main contributions can be summarised as follows:

Original: • We introduce ECLAIRE, a novel decompositional rule extraction algorithm which, to thebest of our knowledge, is the ﬁrst polynomial-time decompositional method applicable to arbitrary DNNs.
Revised: • We introduce ECLAIRE, a novel decompositional rule extraction algorithm which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional method applicable to arbitrary DNNs.

Original: • We show that ECLAIRE’s rule sets outperform those extracted by current state-of-the-artdecompositional methods in both predictive performance and comprehensibility (measuredas the number and length of extracted rules).
Revised: • We show that ECLAIRE’s rule sets outperform those extracted by current state-of-the-art decompositional methods in both predictive performance and comprehensibility (measured as the number and length of extracted rules).

Original: Furthermore, we show that ECLAIRE is able to achieve this while using signiﬁcantly less computational resources than competing methods.
Revised: Furthermore, we show that ECLAIRE is able to achieve this while using signiﬁcantly less computational resources than competing methods.

Original: • We open-source our methods, together with a rule visualisationand interaction
Revised: • We open-source our methods, together with a rule visualisation

Original:  library, aspart of the REMIX package 1 .
Revised: and interaction library, as part of the REMIX package (https://github.com/mateoespinosa/remix).

Original:  Related Work
Revised:  Related Work

Revised: Previous work on explaining a DNN’s prediction using rule-based explanations can be divided into two lines of work: (i) rule-based model-agnostic local explainability methods and (2) DNN rule extraction methods.

Revised: While members of the former family of methods, which include algorithms such as LORE [27] and Anchors [28], are able to generate rule-based explanations for the behaviour of a

Revised: DNN in a point’s local neighbourhood, they are unable to provide insights on any global patterns captured by the DNN’s decision boundary.

Revised: Hence, given that we are interested in extracting rule-based models that provide both local and global interpretability in a DNN, in this section we focus on DNN rule extraction methods.

Original: The design of DNN rule extraction methods can be divided following a taxonomy proposed by
Revised: The design of DNN rule extraction methods can be divided following a taxonomy proposed by

Original: Andrews et al.
Revised: Andrews et al.

Original: [32] in which a method is classiﬁed as being in one of three possible categories:“ pedagogical ”, “ decompositional ”, and “ eclectic ”.
Revised: [24] in which a method is classiﬁed as being in one of three possible categories: “ pedagogical ”, “ decompositional ”, and “ eclectic ”.

Original: Pedagogical methods, such as HYPINV [35],
Revised: Pedagogical methods, such as HYPINV [29],

Original: TREPAN [36], BIO-RE [37], and Validity Interval Analysis [38], extract rules from a DNN bytreating it as a black box.
Revised: TREPAN [30], and Validity Interval Analysis [31] extract rules from a DNN by treating it as a black box.

Original: A limitation of these methods, however, is that by ignoring a DNN’slatent space, they do not take advantage of different hierarchical representations that a DNN wasable to learn from its training set.
Revised: A limitation of these methods, however, is that by ignoring a DNN’s latent space, they do not take advantage of different hierarchical representations that a DNN was able to learn from its training set.

Original: In contrast, decompositional rule extraction algorithms addressthis limitation by explicitly using a DNN’s latent space to guide their rule extraction.

Original: In this workwe are concerned with extracting rules from generic multi-layered architectures, and thus in thissection we focus on discussing only those decompositional methods that are able to extract rulesfrom arbitrary DNNs.
Revised: In this work we are concerned with extracting rules from generic multi-layered architectures, and thus in this section we focus on discussing only those decompositional methods that are able to extract rules from arbitrary DNNs.

Original: This focus renders the majority of decompositional approaches, such as thosein [39, 40, 41, 42, 43], out of scope for our current work due to their restrictive application to onlya subset of architectures.
Revised: This focus renders the majority of decompositional approaches, such as those in [32, 33, 34, 35, 36], out of scope for our current work due to their restrictive application to only a subset of architectures.

Original: Instead, we bring attention to the only two decompositional algorithmsapplicable to arbitrary DNNs: DeepRED [33] and, an optimised version of it, REM-D [31].
Revised: Instead, we bring attention to the only two decompositional algorithms applicable to arbitrary DNNs: DeepRED [25] and, an optimised version of it, REM-D [23].

Original: Borrowing inspiration from CRED [44], an earlier single-layer decompositional method, both
Revised: Borrowing inspiration from CRED [37], an earlier single-layer decompositional method, both

Original: DeepRED and REM-D use decision trees to induce rule sets that map input features to the truth valueof each of the DNN’s output classes.
Revised: DeepRED and REM-D use decision trees to induce rule sets that map input features to the truth value of each of the DNN’s output classes.

Original: The crux of both methods lies in their substitution step in whicha set of rules 
Revised: The crux of both methods lies in their substitution step in which a set of rules R h

Original: R h i (cid:55)→ l mapping activations in the i th layer to the truth value of class l is rewritten to bea function of activations in the ( i − 1) th layer.
Revised: i (cid:55)→ l mapping activations in the i th layer to the truth value of class l is rewritten to be a function of activations in the ( i − 1) th layer.

Original: This substitution is performed in a term-wise manner:each term of a rule’s premise in R h
Revised: This substitution is performed in a term-wise manner: each term of a rule’s premise in R h

Original: i (cid:55)→ l is replaced by a set of new rules which depend on the hiddenactivations of layer ( i − 1) .
Revised: i (cid:55)→ l is replaced by a set of new rules which depend on the hidden activations of layer ( i − 1) .

Original: This substitution, however, requires an exponential post-processingstep in which rules substituted for a given term need to be distributed in a cartesian-product fashionwith rules that were substituted for all terms in the same premise.
Revised: This substitution, however, requires an exponential post-processing step in which rules substituted for a given term need to be distributed in a cartesian-product fashion with rules that were substituted for all terms in the same premise.

Original: This results in both REM-D and
Revised: This results in both REM-D and

Original: DeepRED having an exponential asymptotic run time [31].
Revised: DeepRED having an exponential asymptotic runtime [23].

Original: While algorithmically similar, these methods differ in two implementation-wise optimisations: (1)
Revised: While algorithmically similar, these methods differ in two implementation-wise optimisations: (1)

Original: REM-D uses C5.0 [45] for intermediate decision tree construction rather than C4.5 [46] (a previousiteration of C5.0 with  worse empirical performance and efﬁciency than C5.0) and (2) REM-D savesmemory usage by avoiding storing all intermediate rulesand instead substitutes them as soon as theyhave been extracted.
Revised: REM-D uses C5.0 [38] for intermediate decision tree construction rather than C4.5 [39] (a less efﬁcient iteration of C5.0 with typically worse performance) and (2) REM-D reduces memory consumption by substituting intermediate rules as soon as they have been extracted.

Original: REM-D’s use of C5.0 for intermediate rule extraction, and its early substitution,allows it to empirically outperform DeepREDnot only in terms of its scalability but also in the sizeand ﬁdelity of the resulting rule sets [47].
Revised: REM-D’s use of C5.0 for intermediate rule extraction, and its early substitution, allow it to empirically outperform DeepRED in terms of its scalability as well as in the size and ﬁdelity of its rule sets [40].

Original: The last family of rule extraction methods are eclectic algorithms, a hybrid between pedagogicaland decompositional methods.
Revised: The last family of rule extraction methods are eclectic algorithms, a hybrid between pedagogical and decompositional methods.

Original: Nevertheless, while eclectic methods such as RX [48], MofN [49], FERNN [50], and ERENN_MHL [51] have been shown to extract competitive approximating rulesets, these algorithms either assume that the underlying DNN has only one hidden layer or theyrequire full control over the DNN’s training pipeline (e.g., they require a speciﬁc training procedureto be used for the DNN).
Revised: While eclectic methods such as RX [41], MofN [42], and ERENN_MHL [43] have been shown to extract competitive approximating rule sets, these algorithms either assume that the underlying DNN has only one hidden layer or they require full control over the DNN’s training pipeline (e.g., they require a speciﬁc training procedure to be used for the DNN).

Original:  ECLAIRE
Revised:  ECLAIRE

Original: Problem Setup Assume we are given a set of unlabelled training samples
Revised: Problem Setup Assume we are given a set of unlabelled training samples

Original: X
Revised: X

Original: = { x ( i ) ∈
Revised: = { x ( i ) ∈

Original: R m } Ni = 1and a pretrained DNN f θ : R m (cid:55)→ [0 , 1] L such that for all x ∈ X , f θ ( x ) outputs a probabilitydistribution over labels in set Y = { l 1 , l 2 , · · · , l L } .
Revised: R m } Ni = and a pre-trained DNN f θ : R m (cid:55)→ [0 , 1] L such that for all x ∈ X , f θ ( x ) outputs a probability distribution over labels in set Y = { l 1 , l 2 , · · · , l L } .

Original: Furthermore, assume f θ has d hidden layers andlet h i ( x ) be the output of the i -th layer of f θ when fed with input x .
Revised: Furthermore, assume f θ has d hidden layers and let h i ( x ) be the output of the i -th layer of f θ when fed with input x .

Original: In this setup, we let h 

Revised: In this setup, we let h

Original: ( x ) = xbe the input activation layer and h d +1 ( x ) =
Revised: ( x ) = x be the input activation layer and h d +1 ( x ) =

Original: f θ ( x ) be f θ ’s output probability distribution.
Revised: f θ ( x ) be f θ ’s output probability distribution.

Original: Our goalis to use samples in X to construct a rule set R x
Revised: Our goal is to use samples in X to construct a rule set R x

Original: (cid:55)→ ˆ y of IF-THEN rules of the form 
Revised: (cid:55)→ ˆ y of IF-THEN rules of the form IF

Revised: [Equation]

Revised: (cid:122)

Original: IF (cid:16) term (cid:122) (cid:125)(cid:124) (cid:123) ( x i > v i ) ∧ ( x j ≤ v j ) ∧ · · · ∧ ( x n > v n ) (cid:124) (cid:123)(cid:122) (cid:125) premise/antecedent (cid:17) THEN l k (cid:124)(cid:123)(cid:122)(cid:125) conclusion such that, when subjected to a majority voting 2 with input x , it accurately predicts the result ofevaluating arg max i f θ ( x ) i .
Revised: (cid:125)(cid:124) (cid:123) ( x i > v i ) ∧ ( x j ≤ v j ) ∧ · · · ∧ ( x n > v n ) (cid:124) (cid:123)(cid:122) (cid:125) premise/antecedent (cid:17) THEN l k (cid:124)(cid:123)(cid:122)(cid:125) [Table] such that, when subjected to a majority voting 1 with input x , it accurately predicts the result of evaluating arg max i f θ ( x ) i .

Original: In alignment with existing decompositional methods (e.g., REM-D and
Revised: In alignment with existing decompositional methods (e.g., REM-D and

Original: DeepRED), in our work we constrain each rule’s premise to be a conjunction of termssuch that eachterm is of the form ( x i > v i ) or ( x i ≤ v i ) , where x i is the i -th input feature of sample x and v i ∈ Ris a learnt threshold.
Revised: DeepRED), in our work we constrain each rule’s premise to be a conjunction of terms of the form ( x i > v i ) or ( x i ≤ v i ) , where x i is the i -th input feature of sample x and v i ∈ R is a learnt threshold.

Original: Efﬁcient Clause-wise Rule Extraction In order to introduce our method, we begin with thefollowing observation: a neural network f θ with d hidden layers provides access to d distinct representations for samples x ∈ X .
Revised: Efﬁcient Clause-wise Rule Extraction In order to introduce our method, we begin with the following observation: a neural network f θ with d hidden layers provides access to d distinct representations for samples x ∈ X .

Original: This allows us to extend our support training set X by creating dnew labelled training sets (cid:8) { (cid:0)
Revised: This allows us to extend our support training set X by creating d new labelled training sets (cid:8) { (cid:0)

Original: h i ( x ) , arg max i f θ ( x ) i (cid:1) | x ∈ X } (cid:9) d i =1 from which one can induce drule sets
Revised: h i ( x ) , arg max i f θ ( x ) i (cid:1) | x ∈ X } (cid:9) d i =1 from which one can induce d rule sets

Original: R h i (cid:55)→ ˆ y that map each hidden layer’s output to f θ ’s label predictions ˆ y = arg max i f θ ( x ) i .
Revised: R h i (cid:55)→ ˆ y that map each hidden layer’s output to f θ ’s label predictions ˆ y = arg max i f θ ( x ) i .

Original: The crux of our method consists of efﬁciently unifying all d rule sets { R h 1 (cid:55)→ ˆ y , · · · , R h d (cid:55)→ ˆ y } intoa single rule set R x
Revised: The crux of our method consists of efﬁciently unifying all d rule sets { R h 1 (cid:55)→ ˆ y , · · · , R h d (cid:55)→ ˆ y } into a single rule set R x

Original: (cid:55)→ ˆ y that maps input samples x ∈ X to f θ ’s predicted labels.
Revised: (cid:55)→ ˆ y that maps input samples x ∈ X to f θ ’s predicted labels.

Original: In the same waythat bagging and feature subsampling are key to the ability of random forests to reduce variance andoverﬁtting [52], making use of d different representations of the same dataset to construct R x
Revised: In the same way that bagging and feature subsampling are key to the ability of random forests to reduce variance and overﬁtting [44], making use of d different representations of the same dataset to construct R x

Original: (cid:55)→ ˆ y cansigniﬁcantly help reducing the variance and overﬁtting that would otherwise be present in vanilla ruleinduction algorithms.
Revised: (cid:55)→ ˆ y can signiﬁcantly help reducing the variance and overﬁtting that would otherwise be present in vanilla rule induction algorithms.

Original: be the premise of the j th rule whose consequence is TRUE .

Original: For this example, we assume that the number of rules with TRUE as a conclusion in I x

Original: ( t 1 ∧ t 2 ) and I x (cid:55)→ ( t t t 4 ) is N 1 and N 2 , respectively, which results in rule set R x

Original: (cid:55)→ ˆ y having N 1 + N 2 rules after substitution.

Original: Motivated by these observations, we introduce E fﬁcient CLA use-w I se R ule E xtraction (ECLAIRE).
Revised: Motivated by these observations, we introduce E fﬁcient CLA use-w I se R ule E xtraction (ECLAIRE).

Original: Our method, summarised in Algorithm 1, uses an input DNN f θ and a set of training samples X toincrementally construct a rule set R x
Revised: Our method, summarised in Algorithm 1, uses an input DNN f θ and a set of training samples X to incrementally construct a rule set R x

Original: (cid:55)→ ˆ y which approximates f θ (line 1 ).
Revised: (cid:55)→ ˆ y which approximates f θ (line 1 ).

Original: More speciﬁcally, ECLAIREiterates over each hidden layer
Revised: More speciﬁcally, ECLAIRE iterates over each hidden layer

Original: h i and uses a decision tree generated via C5.0 to induce a rule set 
Revised: h i and uses a general rule induction algorithm ψ ( · ) to induce a rule set R h i

Original: R h i (cid:55)→ ˆ y that maps h i ’s activations to the DNN’s output predictions3 (lines 2 to 5 ).
Revised: (cid:55)→ ˆ y that maps h i ’s activations to the DNN’s output predictions (lines 2 to 5 ).

Original: ECLAIRE thenproceeds to do a clause-wise substitution of rules in R h
Revised: ECLAIRE then proceeds to do a clause-wise substitution of rules in R h

Original: i (cid:55)→ ˆ y .
Revised: i (cid:55)→ ˆ y .

Original: This substitution, depicted in Figure 2,iterates over all of the rules in R h
Revised: This substitution, depicted in Figure 2, iterates over all of the rules in R h

Original: i (cid:55)→ ˆ y (lines 6-7) and constructs a new temporary rule set I x (cid:55)→ pmapping input feature activations x to the truth value of each rule’s premise p (lines 8-9).
Revised: i (cid:55)→ ˆ y (lines 6-7) and constructs a new temporary rule set I x (cid:55)→ p mapping input feature activations x to the truth value of each rule’s premise p (lines 8-9).

Original: Note thatin Algorithm 1 we use p ( x(cid:48) ( i ) ) to represent the result of evaluating premise p (which is a conjunctionof terms) on sample x(cid:48)
Revised: Note that in Algorithm 1 we use p ( x ( i ) ) to represent the result of evaluating premise p (which is a conjunction of terms) on sample x

Original: ( i ) .
Revised: ( i ) .

Original: Finally, we add to R x
Revised: Finally, we add to R x

Original: (cid:55)→ ˆ
Revised: (cid:55)→ ˆ

Original: y
Revised: y

Original: all of the premises in I x
Revised: all of the premises in I x

Original: (cid:55)→ p that have TRUE as aconclusion (lines 10-12).
Revised: (cid:55)→ p that have TRUE as a conclusion (lines 10-12).

Original: In other words, we add all the rules that approximate premise p as beingtrue and map those rules to the class which p originally predicted.
Revised: In other words, we add all the rules that approximate premise p as being true and map those rules to the class which p originally predicted.

Original: This process is repeated for allintermediate hidden layers and terminates by outputting R x
Revised: This process is repeated for all intermediate hidden layers and terminates by outputting R x

Original: (cid:55)→ ˆ y as its result.
Revised: (cid:55)→ ˆ y as its result.

Original: ECLAIRE’s clause-wise substitution allows it to avoid the expensive term redistribution present inother methods such as REM-D and DeepRED. This manifests itself in ECLAIRE’s runtime whichcan be shown to be as follows: Theorem 1 (ECLAIRE Runtime Complexity) .
Revised: [Table]

Original: Given a neural network with d hidden layers, suchthat there are at most m activations in any of its layers, and N training samples, ECLAIRE’s runtimegrows as a function of O (cid:16) dN 3 m (cid:17) . 
Revised: ) 8: ˆ y (1) p , ˆ y (2) p , . . . , ˆ y ( N ) p ← p (cid:0) x (cid:48) (1) (cid:1) , p (cid:0) x (cid:48) (2) (cid:1) , . . . , p (cid:0) x (cid:48) ( N ) (cid:1) 9:

Original: Theorem 1, whose formal proof can be found in Appendix A, is indicative of a signiﬁcant computational advantage in ECLAIRE over previous decompositional methods: while both REM-D and DeepRED are known to have an exponential runtime with respect to both the training set and thedepth of the network [31], ECLAIRE exhibits a polynomial-time growth with respect to its input size. [Table]{+I h 
+}
Revised: {+I h 
+}

Original: µ, { ( x  (1) , ˆ y (1) p ) , ( x  (2) , ˆ y (2) p ) , ..., ( x  ( N ) , ˆ y ( N ) p ) } (cid:1) 10: for clause c ∈ getPremisesByConclusion (cid:0) I x{+h 
+}
Revised: → p ← ψ (cid:0) { ( x (cid:48) (1) , ˆ y (1) p ) , ( x (cid:48) (2) , ˆ y (2) p ) , ..., ( x (cid:48) ( N ) , ˆ y ( N ) p ) } (cid:1) 10: for clause c ∈ getPremisesByConclusion (cid:0) I {+h 
+}

Original: → p , TRUE (cid:1) do 11:
Revised: → p , TRUE (cid:1) do 11:

Original: R x
Revised: R x

Original: (cid:55)→ ˆ y ← R x
Revised: (cid:55)→ ˆ y ← R x

Original: (cid:55)→ ˆ y ∪ (cid:110) IF c THEN getConclusion ( r ) (cid:111) 12: end for 13: end for 14: end for 15: return R x
Revised: (cid:55)→ ˆ y ∪ (cid:110) IF c THEN getConclusion ( r ) (cid:111) 12: end for 13: end for 14: end for 15: return R x

Original: (cid:55)→ ˆ y
Revised: (cid:55)→ ˆ y

Original: Furthermore, ECLAIRE extracts a hidden layer’s intermediate rule set without relying on the rulesets extracted for other layers.
Revised: When using a polynomial-time rule extractor ψ ( · ) , ECLAIRE’s clause-wise substitution allows it to avoid the expensive and exponential term redistribution that is required in REM-D and DeepRED. This manifests itself in ECLAIRE’s runtime which can be shown to be as follows: Theorem 1 (ECLAIRE Runtime Complexity) . Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).

Revised: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .

Revised: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max (3 ,p n +1) m p m (cid:17) .

Revised: Theorem 1, whose proof is in Appendix A, highlights a signiﬁcant computational advantage in

Revised: ECLAIRE over previous decompositional methods: while both REM-D and DeepRED are known to have an exponential runtime with respect to the training set and the depth of the network [23],

Original: This means that ECLAIRE  istopology agnostic  and it can be easilyparallelised, somethingthat we take advantage of in practice by distributing the work in the mainloop (lines 3-15) across n threads threads. 
Revised: ECLAIRE exhibits a polynomial-time growth with respect to its input size when using a polynomial- time rule induction method for ψ ( · ) . Furthermore, ECLAIRE’s use of a DNN’s intermediate representations without regards of their topological order in the network implies that, in stark contrast to REM-D and DeepRED, it is agnostic to the DNN’s network topology and it can be easily parallelised , something we take advantage of in practice by distributing the work in the main loop (lines 3-15) across n threads threads. Notice that this is not possible in both REM-D and DeepRED as they extract rules in a sequential manner, processing the DNN’s layers in reverse topological order.

Original:  Experiments
Revised:  Experiments

Original: Setup For all experiments we perform a 5-fold stratiﬁed cross-validation and, for each metric ofinterest, we show both the mean and standard deviation errors across all 5 folds.
Revised: Setup For all experiments we perform a 5-fold stratiﬁed cross-validation and, for each metric of interest, we show both the mean and standard deviation errors across all 5 folds.

Original: Furthermore, allexperiments are carried on a 2016 MacBook Pro with a 2.7 GHz Quad-Core Intel i7 processor and[-
-]
Revised: Furthermore, all experiments are carried on a 2016 MacBook Pro with a 2.7 GHz Quad-Core Intel i7 processor and[-
-]

Original: GB of RAM.
Revised: GB of RAM.

Original: Finally, we limit each individual experiment to span at most n threads = 6 threads whenusing ECLAIRE and we terminate an experiment if it takes longer than six hours to complete.
Revised: Finally, we limit each individual experiment to span at most n threads = 6 threads when using ECLAIRE and we terminate an experiment if it takes longer than six hours to complete.

Original: Datasets and Tasks We evaluate our methods on six tasks.
Revised: Datasets and Tasks We evaluate our methods on six tasks.

Original: We begin by using a synthetic task(XOR) to contrast ECLAIRE’s rule sets to those extracted by the current state-of-the-art methodsin a controlled setting.
Revised: We begin by using a synthetic task to contrast ECLAIRE’s rule sets to those extracted by the current state-of-the-art methods in a controlled setting.

Original: We then evaluate ECLAIRE on two breast cancer prognosis tasks basedon the METRABRIC [53] dataset to highlight ECLAIRE’s applicability to real-world tasks inwhich transparency is crucial.
Revised: We then evaluate ECLAIRE on two breast cancer prognosis tasks based on the METRABRIC [45] dataset to highlight ECLAIRE’s applicability to real-world tasks in which transparency is crucial.

Original: We proceed to evaluate ECLAIRE’s scalability to large datasets andarchitectures by extracting rules from models trained on two large particle physics datasets (MAGIC[54, 55] and MiniBooNE [56, 55]).
Revised: We proceed to evaluate ECLAIRE’s scalability to large datasets and architectures by extracting rules from models trained on two large particle physics datasets (MAGIC [46, 47] and MiniBooNE [48, 47]).

Original: Finally, we use the Letter Recognition [57, 55] dataset to explore ECLAIRE’s ability to extend to non-binary classiﬁcation tasks.
Revised: Finally, we use the Letter Recognition [49, 47] dataset to explore ECLAIRE’s ability to extend to non-binary classiﬁcation tasks.

Original: A summary of each dataset, includingits size and label distribution, can be found in Appendix B.
Revised: A summary of each dataset, including its size and label distribution, can be found in Appendix B.

Original: Baselines and Metrics In our experiments, we compare ECLAIRE against the  other two decompositional methods that share its applicability: REM-D and DeepRED.
Revised: Baselines and Metrics In our experiments, we compare ECLAIRE against the only other two decompositional methods that share its applicability: REM-D and DeepRED.

Original: Due to a lack of an up-to-dateimplementation of DeepRED, we follow the work in [47] and use a variation of DeepRED where C5.0 is used instead of C4.5.
Revised: Due to a lack of an up-to-date implementation of DeepRED, we follow the work in [40] and use a variation of DeepRED where C5.0 is used instead of C4.5.

Original: This results in an augmented DeepRED that outperforms its originalcounterpart.
Revised: This results in a variant of DeepRED that outperforms its C4. counterpart.

Original: For both this implementation of DeepRED, which we refer to as DeepRED * , and theimplementation of REM-D, we use as our starting point the code published in [31, 47].
Revised: For both this implementation of DeepRED, which we refer to as DeepRED * , and the implementation of REM-D, we use as our starting point the code published by Shams et al.

Revised: When evaluating ECLAIRE, we use rule sets induced from C5.0 trees as our intermediate rule extractor ψ ( · ) unless speciﬁed otherwise.

Revised: Although this constraints our evaluation to a single rule extractor (rather than using more recent rule extractors such as Bayesian rule sets [50] or Column

Revised: Generation [51]), this allows us to fairly evaluate the algorithmic design of ECLAIRE against that of DeepRED * and REM-D without strongly biasing our results to our choice of intermediate rule extractor.

Revised: Nevertheless, notice that from Theorem 1, and the fact that C5.0’s runtime complexity grows as O ( N 2 m ) [38], it follows that the runtime complexity of this instantiation of ECLAIRE grows as

Original: Finally,to highlight the beneﬁts of using a DNN’s latent space for rule extraction, we include two nondecompositional baselines: C5.0 as a baseline end-to-end rule induction method and PedC5.0 [31, 47]as a baseline pedagogical algorithm.
Revised: O ( dN 3 m ) . Finally, to highlight the beneﬁts of using a DNN’s latent space for rule extraction, we include two non-decompositional baselines: C5.0 as a baseline end-to-end rule induction method and PedC5.0 [23, 40] as a baseline pedagogical algorithm.

Original: The latter method uses C5.0 to induce a rule set from a trainingset where each sample is labelled using the DNN’s prediction for that sample rather than its true label.
Revised: The latter method uses C5.0 to induce a rule set from a training set where each sample is labelled using the DNN’s prediction for that sample.

Original: When comparing any two methods, we evaluate the accuracy of their rule sets, their ﬁdelity withrespect to the input DNN (deﬁned as the predictive accuracy when sample x ( i ) is assigned label ˆ y ( i ) =
Revised: When comparing any two methods, we evaluate the ﬁdelity of the extracted rule sets with respect to the input DNN (deﬁned as the predictive accuracy when sample x ( i ) is assigned label ˆ y ( i ) =

Original: arg max j f θ ( x ( i ) )
Revised: arg max j f θ ( x ( i ) )

Original: j ), and their comprehensibility as measured by their number of rules andtheir average rule length.
Revised: j ) and their comprehensibility as measured by their number of rules and their average rule length.

Original: Finally, as a way to evaluate the tractability of our methods, we keep trackof the resources utilised by each method in terms of both wall-clock time and memory.
Revised: Furthermore, as a way to evaluate the tractability of our methods, we keep track of the resources utilised by each method in terms of both wall-clock time and memory.

Revised: Finally, given that there may be a direct beneﬁt from using the rule sets extracted from the DNN as standalone interpretable models (e.g., as a form of model distillation that generates stronger rule-based models), we include each rule set’s accuracy and AUC as part of our evaluation.

Original: Model and Hyperparameter Selection
Revised: Model and Hyperparameter Selection

Revised: Although ECLAIRE is applicable to arbitrary DNN architectures, in order for us to simplify architecture selection during evaluation, in our experiments we focus on extracting rule sets from multilayer perceptrons (MLPs).

Original: In order to simulate ECLAIRE’s use case, we maintain astrict separation between ﬁne-tuning the DNN we train for a task and ﬁne-tuning the rule extractionalgorithm we apply to that model.
Revised: Moreover, in order to simulate ECLAIRE’s use case, we maintain a strict separation between ﬁne-tuning the DNN we train for a task and ﬁne-tuning the rule extraction algorithm we apply to that model.

Original: Speciﬁcally, we train several MLP architectures for each task andextract rules only from the best performing model.
Revised: More speciﬁcally, we train several MLP architectures for each task and extract rules only from the best performing model.

Original: During rule extraction, we vary only the minimumnumber of samples µ  we use for splitting a node in C5.0.
Revised: During rule extraction, we control the growth of intermediate rule sets induced via C5.0 by varying only the minimum number of samples µ that we require C5.0 to have before splitting a node.

Original: Because µ requires different ﬁne-tuningstrategies across different tasks 4 , in this section we report only the highest performing rule set weobtain after varying µ across a task-and-method-speciﬁc spectrum.
Revised: Because µ requires different ﬁne-tuning strategies across different tasks 2 , in this section we report only the highest performing rule set we obtain after varying µ across a task-and-method-speciﬁc spectrum.

Original: Unless speciﬁed otherwise, allother C5.0’s hyperparameters, with the exception of winnowing which is always set to true, are leftas their default values.
Revised: Unless speciﬁed otherwise, all other C5.0’s hyperparameters, with the exception of winnowing which is always set to true, are left as their default values.

Original: We include all details of each task’s architecture and hyperparameter selection,including the search spectrum used for µ for each task, in Appendix C.
Revised: We include all details of each task’s architecture and hyperparameter selection, including the search spectrum used for µ for each task, in Appendix C.

Original: 4.1 XOR: Evaluating Rule Set Interpretability in Controlled Setup
Revised: 4.1 XOR: Evaluating Rule Set Interpretability in Controlled Setup

Original: In this section, we use a synthetic dataset to evaluate the performance of ECLAIRE and our baselinesin a controlled environment known to be challenging for vanilla rule induction algorithms.
Revised: In this section, we use a synthetic dataset to evaluate the performance of ECLAIRE and our baselines in a controlled environment known to be challenging for vanilla rule induction algorithms and pedagogical rule extraction methods.

Original: For this,we make use of a variant of the XOR classiﬁcation task, in a form commonly used for studyingfeature selection methods [8, 58, 59], in which we generate a dataset { ( x ( i ) , y i ) } 1000 i =1 with 100010-dimensional samples such that every data point x ( i ) ∈ [0 , 1] 10 is constructed by independentlysampling each dimension from a uniform distribution in [0 , 1] .
Revised: For this, we make use of a variant of the XOR classiﬁcation task, in a form commonly used for studying feature selection methods [7, 52, 53], in which we generate a dataset { ( x ( i ) , y i ) } 1000 i =1 with 1000 10-dimensional samples such that every data point x ( i ) ∈ [0 , 1] 10 is constructed by independently sampling each dimension from a uniform distribution in [0 , 1] .

Original: In this dataset, we assign a binarylabel y i ∈ { 0 , 1 } to each point x
Revised: In this dataset, we assign a binary label y i ∈ { 0 , 1 } to each point x

Original: ( i ) by XOR-ing the result of rounding its ﬁrst two dimensions asy i = round (cid:0) x
Revised: ( i ) by XOR-ing the result of rounding its ﬁrst two dimensions as y i = round (cid:0) x

Original: ( i ) 1 (cid:1)
Revised: ( i ) 1 (cid:1)

Original: ⊕ round (cid:0) x
Revised: ⊕ round (cid:0) x

Original: ( i ) 2 (cid:1) .
Revised: ( i ) 2 (cid:1) .

Original: In order for a classiﬁer to be able to achieve a high performance in this task, it must be able tolearn that only the ﬁrst two dimensions are meaningful for predicting the output label.
Revised: In order for a classiﬁer to be able to achieve a high performance in this task, it must learn that only the ﬁrst two dimensions are meaningful for predicting the output label.

Original: This propertymakes this dataset an illustrative example of a task in which decision trees are unable to obtain a highperformance without a large training dataset [60, 33].
Revised: This property makes this dataset an illustrative example of a task in which decision trees are unable to obtain a high performance without a large training dataset [54, 25].

Original: Therefore, this task highlights the competitiveadvantages ofusing a DNN’s latent space to guide a rule set construction.
Revised: In this section, we hypothesise that decompositional methods will outperform both vanilla rule induction algorithms and pedagogical rule extraction methods as their use of a DNN’s latent representations during training results in an implicit data augmentation process. Furthermore, we hypothesise that within our decompositional baselines, ECLAIRE will outperform other methods given that its two-step process avoids the need of nesting multiple sequential approximations when merging rule sets as done in both REM-D and DeepRED * .

Original: To test ECLAIRE and ourbaselines in this task, we extract rules from an MLP with hidden layer sizes { 64 , 32 , 16 } trained onthis dataset and report our ﬁndings in the XOR rows of Table 1.
Revised: To test both hypothesises, we extract rules from an MLP with hidden layer sizes { 64 , 32 , 16 } trained on this dataset and report our ﬁndings in the XOR rows of Table 1.

Original: Theachieved MLP’s test accuracy is96 . 6% ± 1 . 9% and itstest AUC is 96 .
Revised: The MLP’s test accuracy is 96 . 6% ± 1 . 9% and its AUC is 96 .

Original: 6% ± 1 . 9% .
Revised: 6% ± 1 . 9% .

Original:  Table 1 shows that ECLAIRE outperforms all other decompositional methods in the XOR task in terms ofboth accuracy and ﬁdelity.
Revised: As hypothesised, Table 1 shows that ECLAIRE outperforms all other decompositional methods in the XOR task in terms of both accuracy and ﬁdelity.

Original: We further observe that ECLAIRE uses signiﬁcantly less resources than REM-Dand DeepRED * while generating signiﬁcantly fewerrules than both.
Revised: We further observe that ECLAIRE uses signiﬁcantly less resources than REM-D and DeepRED * while generating signiﬁcantly fewer rules than both.

Original: Moreover, note that, as expected, wealso observe that both C5.0 and PedC5.0 are unable tocapture any meaningful rules and instead generate anempty rule set that acts as a random classiﬁer.
Revised: Moreover, note that, as expected, we also observe that both C5.0 and PedC5.0 are unable to capture any meaningful rules and instead generate an empty rule set that acts as a random classiﬁer.

Original: Finally,we observe that the standard deviation of ECLAIRE’srule set sizes is relatively small compared to its mean.
Revised: Finally, we observe that the standard deviation of ECLAIRE’s rule set sizes is relatively small compared to its mean.

Original: This is in stark contrast with the observed standard deviations of rule set sizes in both REM-D and DeepRED * ,which tend to be even greater than their means.
Revised: This is in stark contrast with the observed standard deviations of rule set sizes in both REM-D and DeepRED * , which tend to be even greater than their means.

Original: Suchbehaviour indicates that  ECLAIRE is more robust tochanges in its training data and input DNN than REM-Dand DeepRED * .
Revised: Such behaviour indicates that [Table] ECLAIRE is more robust to changes in its training data and input DNN than REM-D and DeepRED * .

Original: We conjecture a hypothesis to explainthese results in Section 5.
Revised: We conjecture a hypothesis to explain these results in Section 5.

Original: Our results also indicate that ECLAIRE is better at extracting rules that correctly capture theunderlying generating process of the XOR dataset, therefore providing better global insights onwhat a DNN learnt from its training data.
Revised: Our results also indicate that ECLAIRE is better at extracting rules that correctly capture the underlying generating process of the XOR dataset, therefore providing better global insights on what a DNN learnt from its training data.

Original: This can be seen when analysing which features wereconsidered important in the rules extracted by ECLAIRE.
Revised: This can be seen when analysing which features were considered important in the rules extracted by ECLAIRE.

Original: Such analysis, shown in Figure 3, indicatesthat rules in ECLAIRE are able to identify the importance that the ﬁrst two features (i.e., x 1 and[Table] x 2 ) have on the resulting output class by very rarely conditioning on other features.
Revised: Such analysis, shown in Figure 3, indicates that rules in ECLAIRE are able to identify the importance that the ﬁrst two features (i.e., x 1 and x 2 ) have on the resulting output class by very rarely conditioning on other features.

Original: This is at oddswith what we see in rules extracted by REM-D and DeepRED * , where a signiﬁcant fraction of rulepremises heavily rely on features other than x 1 and x 2 .
Revised: This is at odds with what we see in rules extracted by REM-D and DeepRED * , where a signiﬁcant fraction of rule premises heavily rely on features other than x 1 and x 2 .

Original: 4.2 METABRIC: Medical Applications
Revised: 4.2 METABRIC: Medical Applications

Original: In this section we evaluate ECLAIRE in real and safety-critical tasks in which interpretable models arefavoured.
Revised: In this section we evaluate ECLAIRE in real and safety-critical tasks in which interpretable models are favoured.

Original: To achieve this, we evaluate our methods on two medical tasks, based on the METABRIC[53] dataset, deﬁned byShams et al.
Revised: To achieve this, we evaluate our methods on two medical genomics tasks deﬁned by

Original: in [31]. The speciﬁc tasks we consider are: • Immunohistochemical subtype prediction (MB-ER): for this task we predict immunohis-
Revised: Shams et al.

Original: tochemical subtypes in breast cancer patients using 1000 mRNA expression patterns. Each sample can be classiﬁed as one of two types, ER+ or ER-, which are crucial for determining a patient’s treatment.
Revised: in [23] based on the METABRIC dataset.

Original: • Histological tumour subtype prediction (MB-HIST): for this task we  predict  histological  subtypes  ofbreast cancer tumours using 1004 mRNA expression proﬁles.Each sample
Revised: More speciﬁcally, we extract rules from models trained to predict immunohistochenical subtypes (MB-ER) and histological tumour subtypes (MB-HIST) from a sequence of mRNA expression proﬁles.

Original: can be classiﬁed as either Invasive Lobular Carcinoma (ILC) or Invasive Ductal Carcinoma (IDC), two most common breast cancer histological subtypes [
Revised: For further details, refer to Appendix B.

Original: Using the same DNN architecture and hyperparameters as in [31], we train two MLPs with hiddenlayers { 128 , 16 } , one trained on MB-ER and one trained on MB-HIST, and extract rules fromthese models.
Revised: Using the same DNN architecture and hyperparameters as in [23], we train two MLPs with hidden layers { 128 , 16 } , one trained on MB-ER and one trained on MB-HIST, and extract rules from these models.

Original: In MB-ER, our MLP achieves a test accuracy of 97 .
Revised: In MB-ER, our MLP achieves a test accuracy of 97 .

Original: 9% ± 0 .
Revised: 9% ± 0 .

Original: 3% and test AUC of95 .
Revised: 3% and test AUC of 95 .

Original: 6% ± 1 . 1% while in MB-HIST our MLP achieves a test accuracy of 91 .
Revised: 6% ± 1 . 1% while in MB-HIST our MLP achieves a test accuracy of 91 .

Original: 2% ± 3 .
Revised: 2% ± 3 .

Original: 5% and a test
Revised: 5% and a test

Original: AUC of 88 .
Revised: AUC of 88 .

Original: 4% ± 3% .
Revised: 4% ± 3% .

Original: We summarise these results in the MB-ER and MB-HIST rows of Table 1.
Revised: We summarise these results in the MB-ER and MB-HIST rows of Table 1.

Original: As in XOR, in both of these tasks we observe that ECLAIRE is able to efﬁciently extract rule sets thatare not only signiﬁcantly smaller than those produced by DeepRED * and REM-D (up to 32x fewerrules on MB-ER) but also higher-performing in their predictive accuracy.
Revised: As in XOR, in both of these tasks we observe that ECLAIRE is able to efﬁciently extract rule sets that are not only signiﬁcantly smaller than those produced by DeepRED * and REM-D (up to 32x fewer rules on MB-ER) but also higher-performing in their predictive accuracy.

Original: Only in MB-HIST we see ECLAIRE underperforming in predictive accuracy when compared to REM-D.
Revised: Only in MB-HIST we see ECLAIRE underperforming in predictive accuracy when compared to REM-D.

Original: However, given thehigh class imbalance in this dataset (91.3% of samples are of one class), AUC is a more meaningfulmetric than accuracy.
Revised: However, given the high class imbalance in this dataset (91.3% of samples are of one class), AUC is a more meaningful metric than accuracy.

Revised: ECLAIRE’s superior AUC in MB-HIST indicates that it is able to extract higher-ﬁdelity rule sets than our baselines in binary tasks with a high class imbalance, a signiﬁcant advantage for several real-world tasks.

Original: Finally, note that although we see similar beneﬁts in ECLAIRE’s runtime asin XOR, in these tasks ECLAIRE’s parallelism overhead leads to a higher memory consumptionthan REM-D and DeepRED * .
Revised: Finally, note that although we see similar beneﬁts in ECLAIRE’s runtime as in XOR, in these tasks ECLAIRE’s parallelism overhead leads to a higher memory consumption than REM-D and DeepRED * .

Original: Nevertheless, although ECALIRE’s rule sets are not as small as thoseproduced by PedC5.0 and C5.0, they have fewer than 50 rules, making them easy to inspect anddebug for practitioners that may require such a model for prognosis purposes.
Revised: Nevertheless, although ECALIRE’s rule sets are not as small as those produced by PedC5.0 and C5.0, they have fewer than 50 rules, making them easy to inspect and debug for practitioners that may require such a model for prognosis purposes.

Original: 4.3 MAGIC and MiniBooNE: Scalability Study
Revised: 4.3 MiniBooNE and MAGIC: Scalability Study

Original: In this section we discuss a series of experiments that evaluate how ECLAIRE and our baselinesscale when the number of samples and the depth of the architecture increase.
Revised: In this section we discuss a series of experiments that evaluate how ECLAIRE and our baselines scale when the number of samples and the depth of the architecture increase.

Original: For this, we use twoparticle physics binary classiﬁcation tasks that offer a large training set: MAGIC [54, 55], with19,020 training samples, and MiniBooNE [61, 55], with 130,065 training samples.
Revised: For this, we use two particle physics binary classiﬁcation tasks that offer a large training set: MAGIC [46, 47], with 19,020 training samples, and MiniBooNE [55, 47], with 130,065 training samples.

Original: We train an MLPwith hidden layers { 64 , 32 , 16 } on MAGIC and an MLP with hidden layers { 128 , 64 , 32 , 16 , 8 } on
Revised: We train an MLP with hidden layers { 64 , 32 , 16 } on MAGIC and an MLP with hidden layers { 128 , 64 , 32 , 16 , 8 } on

Original: MiniBooNE and report those results in Table 1.
Revised: MiniBooNE and report those results in Table 1.

Original: The MLP trained on MAGIC achieves a test accuracyof 86 .
Revised: The MLP trained on MAGIC achieves a test accuracy of 86 .

Original: 3% ± 1 . 8% and a test AUC of 86 .
Revised: 3% ± 1 . 8% and a test AUC of 86 .

Original: 3% ± 1 . 8% while the MLP trained on MiniBooNE achievesa test accuracy of 97.9% ± 0.3% and a test AUC of 93% ± 0.2%.
Revised: 3% ± 1 . 8% while the MLP trained on MiniBooNE achieves a test accuracy of 97.9% ± 0.3% and a test AUC of 93% ± 0.2%.

Original: In these experiments, we came across signiﬁcant difﬁculties when using REM-D and DeepRED * .
Revised: In these experiments, we came across signiﬁcant difﬁculties when using REM-D and DeepRED * .

Original: This was because most of their conﬁgurations led to runs that took more than 6 hours in total, eitherdue to intractable rule extraction or intractable inference caused by large extracted rule sets.
Revised: This was because most of their conﬁgurations led to runs that took more than 6 hours in total, either due to intractable rule extraction or intractable inference caused by large extracted rule sets.

Original: Thus,we increased the value of µ to be a signiﬁcant fraction of N , the number of points in the dataset,and we show the best value we obtain by varying µ as indicated inour hyperparameter selection in Appendix C.3.4.
Revised: Thus, we increased the value of µ to be a signiﬁcant fraction of N , the number of points in the dataset, and we show the best value we obtain by varying µ as indicated in Appendix C.3.4.

Original: Nevertheless, notice that although such amounts of pruning lead to smaller rule setsthan ECLAIRE’s in MiniBooNE, these are overpruned rule sets which underperform in the test set.
Revised: Nevertheless, notice that although such amounts of pruning lead to smaller rule sets than ECLAIRE’s in MiniBooNE, these are overpruned rule sets which underperform in the test set.

Original: Our experiments show that ECLAIRE can scale to both large architectures and large datasets withoutthe need for excessively large values of µ (i.e., no need for excessive intermediate rule set pruning).
Revised: Our experiments show that ECLAIRE can scale to both large architectures and large datasets without the need for excessively large values of µ (i.e., no need for excessive intermediate rule set pruning).

Original: Furthermore, we can see that ECLAIRE outperforms all of the baselines in both tasks in terms ofits AUC and ﬁdelity.
Revised: Furthermore, we can see that ECLAIRE outperforms all of the baselines in both tasks in terms of its AUC and ﬁdelity.

Original: Only in its accuracy it is marginally outperformed by C5.0 and PedC5.0 in the
Revised: Only in its accuracy it is marginally outperformed by C5.0 and PedC5.0 in the

Original: MiniBooNE task.
Revised: MiniBooNE task.

Original: Nevertheless, given the class imbalance in this task, the beneﬁt from ECLAIRE’s
Revised: Nevertheless, given the class imbalance in this task, the beneﬁt from ECLAIRE’s

Original: AUC improvements out-weights its marginally worse accuracy.
Revised: AUC improvements out-weights its marginally worse accuracy.

Original: 4.4 Letter Recognition: Multiclass Classiﬁcation
Revised: 4.4 Letter Recognition: Multiclass Classiﬁcation

Original: Although both REM-D and DeepRED were developed to be applicable to multiclass tasks, they havenever been formally evaluated in such tasks.
Revised: Although both REM-D and DeepRED were developed to be applicable to multiclass tasks, they have never been formally evaluated in such tasks.

Original: In this section we ﬁll this gap by extracting rules from amodel trained on the Letter Recognition Dataset [57, 55], a classiﬁcation task with 26 target labels.
Revised: In this section we ﬁll this gap by extracting rules from a model trained on the Letter Recognition Dataset [49, 47], a classiﬁcation task with 26 target labels.

Original: We proceed by extracting rules from a trained MLP with hidden layers { 128 , 64 } which achieves atest accuracy of 95% ± 0 .
Revised: We proceed by extracting rules from a trained MLP with hidden layers { 128 , 64 } which achieves a test accuracy of 95% ± 0 .

Original: We summarise our results in Table 1.
Revised: We summarise our results in Table 1 and include further experiments using an alternative tree induction algorithm in Appendix C.3.5.

Original: In this task, as in other data-intensive tasks, we ﬁrst observe a difﬁcultly in tractably extractingrules using REM-D and DeepRED * without large values of µ .
Revised: As in other data-intensive tasks, we ﬁrst observe a difﬁcultly in tractably extracting rules using REM-D and DeepRED * without large values of µ .

Original: Even when we ﬁnd a conﬁgurationfor these methods that terminates in the allotted time, we are unable to observe results that aresigniﬁcantly better than random.
Revised: Even when we ﬁnd a conﬁguration for these methods that terminates in the allotted time, we are unable to observe results that are signiﬁcantly better than random.

Original: Second, although ECLAIRE is able to outperform both DeepRED *and REM-D, it struggles to outperform PedC5.0 and C5.0.
Revised: Second, although ECLAIRE is able to outperform both DeepRED * and REM-D, it struggles to outperform PedC5.0 and C5.0.

Original: We believe this to be the case due topossible class imbalance in intermediate rule sets: when performing a clause-wise substitution inintermediate rules, and as the number of classes increases, the number of samples that satisfy thepremise of an intermediate rule becomes smaller.
Revised: We believe this to be the case due to possible class imbalance in intermediate rule sets: when performing a clause-wise substitution in intermediate rules, and as the number of classes increases, the number of samples that satisfy the premise of an intermediate rule becomes smaller.

Original: This implies that when using C5.0 to substitute anintermediate clause, it has to face a highly imbalanced binary classiﬁcation task, something knownto be problematic for decision trees [62].
Revised: This implies that when using C5.0 to substitute an intermediate clause, it has to face a highly imbalanced binary classiﬁcation task, something known to be problematic for decision trees [56].

Original: While we are able to slightly alleviate this in our resultsthrough C5.0’s support for independent class weights, it remains as future work to explore bettermechanisms for handling this imbalance.
Revised: While we are able to slightly alleviate this in our results through C5.0’s support for independent class weights, it remains as future work to explore better mechanisms for handling this imbalance.

Original: An interesting observation is that when we include the input features as one of the representationswhich ECLAIRE can use to extract intermediate rule sets, it results in a rule set that outperforms
Revised: An interesting observation is that when we include the input features as one of the representations which ECLAIRE can use to extract intermediate rule sets, it results in a rule set that outperforms

Original: C5.0 and PedC5.0.
Revised: C5.0 and PedC5.0.

Original: This result, shown at the bottom of Table 1 as ECLAIRE * , suggests that using a
Revised: This result, shown at the bottom of Table 1 as ECLAIRE * , suggests that using a

Original: DNN’s input features as one of the representations that ECLAIRE has access to can be beneﬁcial forthe performance of its end rule set.
Revised: DNN’s input features as one of the representations that ECLAIRE has access to can be beneﬁcial for the performance of its end rule set.

Original: This can be thought of as an eclectic variation of ECLAIRE andwe leave it as future research to explore whether this improvement generalises across domains.
Revised: This can be thought of as an eclectic variation of ECLAIRE and we leave it as future research to explore whether this improvement generalises across domains.

Original:  Discussion
Revised:  Discussion

Original: Variance Reduction in ECLAIRE In [31], Shams et al.
Revised: Variance Reduction in ECLAIRE In [23], Shams et al.

Original: observe a very high variance in thesize of rule sets extracted with REM-D and postulate that this is due to different neural networkinitialisations leading to latent spaces that vary in interpretability properties.
Revised: observe a very high variance in the size of rule sets extracted with REM-D and postulate that this is due to different neural network initialisations leading to latent spaces that vary in interpretability properties.

Original: Nevertheless, our resultsin Table 1 suggest that this may not be the case, as ECLAIRE’s variance is signiﬁcantly smallerthan that observed in both REM-D and DeepRED * .
Revised: Nevertheless, our results in Table 1 suggest that this may not be the case, as ECLAIRE’s variance is signiﬁcantly smaller than that observed in both REM-D and DeepRED * .

Original: Inspired by results on convergence learning [63, 64], in this paper we challenge this hypothesis by arguing that the variance observed in REM-D,and therefore in DeepRED, is not the result of different initialisations leading to different levels ofinterpretability in a DNN’s latent space, but it is rather the result of a crucial design factor in REM-D:
Revised: Inspired by results on convergence learning [57, 58], in this paper we challenge this hypothesis by arguing that the variance observed in REM-D, and therefore in DeepRED, is not the result of different initialisations leading to different levels of interpretability in a DNN’s latent space, but it is rather the result of a crucial design factor in REM-D:

Original: REM-D uses rule sets induced from decision trees to substitute terms in its intermediate rule sets,forcing the need of a cartesian-product redistribution of terms afterwards.
Revised: REM-D uses rule sets induced from decision trees to substitute terms in its intermediate rule sets, forcing the need of a cartesian-product redistribution of terms afterwards.

Original: Decision trees, being highvariance classiﬁers [65], are highly sensitive to their training sets.
Revised: Decision trees, being highvariance classiﬁers [59], are highly sensitive to their training sets.

Original: This implies that when REM-Dsubstitutes the same term in an intermediate rule using two different datasets (as it would happenacross two different training folds), it can generate vastly different rule sets.
Revised: This implies that when REM-D substitutes the same term in an intermediate rule using two different datasets (as it would happen across two different training folds), it can generate vastly different rule sets.

Original: Hence, because during itsterm-wise substitution REM-D replaces each term in an intermediate rule’s premise with a set of rulesextracted from a decision tree, the variance in the number of rules after one substitution grows in amultiplicative fashion.
Revised: Hence, because during its term-wise substitution REM-D replaces each term in an intermediate rule’s premise with a set of rules extracted from a decision tree, the variance in the number of rules after one substitution grows in a multiplicative fashion.

Original: This is because all newly substituted terms in a clause need to be redistributedin a cartesian-product (a visual representation of this process can be found in Appendix D).
Revised: This is because all newly substituted terms in a clause need to be redistributed in a cartesian-product (a visual representation of this process can be found in Appendix D).

Original: Therefore,we argue that the signiﬁcant reduction in variance we observe in our evaluation of ECLAIRE is dueto the fact that ECLAIRE performs a clause-wise substitution rather than a term-wise substitution,hence avoiding a multiplicative growth in the number of rules.
Revised: Therefore, we argue that the signiﬁcant reduction in variance we observe in our evaluation of ECLAIRE is due to the fact that ECLAIRE performs a clause-wise substitution rather than a term-wise substitution, hence avoiding a multiplicative growth in the number of rules.

Original: This hypothesis explains the results in
Revised: This hypothesis explains the results in

Original: Table 1 and highlights a signiﬁcant advantage of ECLAIRE over other decompositional methods.
Revised: Table 1 and highlights a signiﬁcant advantage of ECLAIRE over other decompositional methods.

Original: Growth Coping Mechanisms Despite ECLAIRE’s superior performance compared to our baselines,
Revised: Growth Coping Mechanisms Despite ECLAIRE’s superior performance compared to our baselines,

Original: Table 1 shows that ECLAIRE can still take a considerable amount of time to extract high-performingrule sets in large datasets and architectures.
Revised: Table 1 shows that ECLAIRE can still take a considerable amount of time to extract high-performing rule sets in large datasets and architectures.

Original: Furthermore, we observe a growth in the number of rulesextracted by ECLAIRE as the amount of data and the model size increase.
Revised: Furthermore, we observe a growth in the number of rules extracted by ECLAIRE as the amount of data and the model size increase.

Original: As a way to alleviate thisgrowth, we explore four different coping mechanisms: (1) intermediate rule pruning , where, as in[31], we increase the value of µ to generate smaller intermediate rule sets, (2) hidden representationsubsampling , where we generate intermediate rule sets only from a subset of the DNN’s intermediaterepresentations, (3) training set subsampling , where, as in [33], only a fraction of the DNN’s trainingset is used for rule extraction, and (4) intermediate rule ranking , wherewe drop the lowest p % ofrules in intermediate rule sets as ranked by their conﬁdence levels.
Revised: As a way to alleviate this growth, we explore four different coping mechanisms: (1) intermediate rule pruning , where, as in [23], we increase the value of µ to generate smaller intermediate rule sets, (2) hidden representation subsampling , where we generate intermediate rule sets only from a subset of the DNN’s intermediate representations, (3) training set subsampling , where, as in [25], only a fraction of the DNN’s training set is used for rule extraction, and (4) intermediate rule ranking , where the lowest p % of rules in intermediate rule sets, as ranked by their conﬁdence levels, are dropped.

Original: We summarise these experimentsin Appendix E.
Revised: We summarise these experiments in Appendix E.

Original: Our results indicate that the performance of ECLAIRE’s rule sets is robust to heavy training datasubsampling as well as signiﬁcant intermediate representation subsampling and intermediate rulepruning via conﬁdence ranking.
Revised: Our results indicate that the performance of ECLAIRE’s rule sets is robust to heavy training data subsampling as well as signiﬁcant intermediate representation subsampling and intermediate rule pruning via conﬁdence ranking.

Original: Speciﬁcally, we observe that ECLAIRE is very data efﬁcient : one cansubsample up to 50% of the training set during rule extraction, causing a ∼ 50% drop in extractiontime and rule set size, without observing an absolute loss of more than 0 .
Revised: Speciﬁcally, we observe that ECLAIRE is very data efﬁcient : one can subsample up to 50% of the training set during rule extraction, causing a ∼ 50% drop in extraction time and rule set size, without observing an absolute loss of more than 0 .

Original: 5% in the resulting rule setﬁdelity.
Revised: 5% in the resulting rule set ﬁdelity.

Original: Similar results are observed when one subsamples every other two hidden representationswhen generating intermediate rule sets or drops the lowest 25% of rules in intermediate rule sets.
Revised: Similar results are observed when one subsamples every other two hidden representations when generating intermediate rule sets or drops the lowest 25% of rules in intermediate rule sets.

Original: Nevertheless, our experiments also highlight a clear limitation in ECLAIRE: for achieving optimalperformance, ECLAIRE requires signiﬁcant task-speciﬁc ﬁne tuning of its µ hyperparameter.
Revised: Nevertheless, our experiments also highlight a clear limitation in ECLAIRE: for achieving optimal performance, ECLAIRE requires signiﬁcant task-speciﬁc ﬁne tuning of its µ hyperparameter.

Original: Whilein practice we observe that selecting µ from [10 − 5 N, 10 − 4 N ] yields good results, we leave theexploration of more systematic strategies for selecting µ to future work.
Revised: While in practice we observe that selecting µ from [10 − 5 N, 10 − 4 N ] yields good results, we leave the exploration of more systematic strategies for selecting µ to future work.

Original:  Conclusion
Revised:  Conclusion

Original: With the ubiquitous use of DNNs, it has become increasingly important to be able to explain, aswell as debug, the decisions made by production-sized DNNs.
Revised: With the ubiquitous use of DNNs, it has become increasingly important to be able to explain, as well as debug, the decisions made by production-sized DNNs.

Original: In this work we aim to ﬁll this gapby proposing ECLAIRE, a novel decompositional rule extraction algorithm, which, to the best ofour knowledge, is the ﬁrst polynomial-time decompositional algorithm able to scale to both largetraining sets and deep architectures.
Revised: In this work we aim to ﬁll this gap by proposing ECLAIRE, a novel decompositional rule extraction algorithm, which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional algorithm able to scale to both large training sets and deep architectures.

Original: In contrast to previous work which focuses on using intermediaterepresentations in a sequential manner, ECLAIRE exploits these representations in parallel to buildan ensemble of classiﬁers that can then be efﬁciently combined into a single rule set.
Revised: In contrast to previous work which focuses on using intermediate representations in a sequential manner, ECLAIRE exploits these representations in parallel to build an ensemble of classiﬁers that can then be efﬁciently combined into a single rule set.

Original: We evaluate
Revised: We evaluate

Original: ECLAIRE on six different tasks and show that it consistently outperforms the previous state-of-the-artdecompositional methods and vanilla rule induction methods in terms of the accuracy and ﬁdelity ofits output rule set.
Revised: ECLAIRE on six different tasks and show that it consistently outperforms the previous state-of-the-art decompositional methods and vanilla rule induction methods in terms of the accuracy and ﬁdelity of its output rule set.

Original: At the same time, ECLAIRE uses orders of magnitude less resources and producesfewer rules than current state-of-the-art decompositional methods.
Revised: At the same time, ECLAIRE uses orders of magnitude less resources and produces fewer rules than current state-of-the-art decompositional methods.

Original: Excited with the prospects that canarise from this work, and to encourage the use of our algorithms in both industry and research, wemake all of our methods, together with an extensive set of visualisation and inspection tools, publiclyavailable as part of the REMIX library.
Revised: Excited with the prospects that can arise from this work, and to encourage the use of our algorithms in both industry and research, we make all of our methods, together with an extensive set of visualisation and inspection tools, publicly available as part of the REMIX library.

Original: Follow up research could explore generalising ECLAIRE toregression tasks, as well as to RNN architectures, and could include work on alleviating the effects ofclass imbalance in multiclass settings. Consideration for Publication
Revised: Follow up research could explore generalising ECLAIRE to regression tasks, as well as to RNN architectures and could include work on alleviating the effects of class imbalance in multiclass settings

Original: If accepted, we would like this submission to be considered for publication in a journal special issue.

Original: Checklist
Revised: Acknowledgements

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: [Yes] See Section 4 for evidence for the contributions claimed

Original: in our abstract and introduction.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5 and Appendix E for

Original: a discussion on the limitations of our method’s runtime as well as some possible mech-

Original: anisms to cope with these limitations.

Original: Furthermore, see Section 4.4 for a discussion on

Original: some limitations concerning multiclass tasks with a large number of labels.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We are

Original: not aware of any negative societal impacts that our work may have.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] If you are including theoretical results...
Revised: MEZ acknowledges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] All assumptions

Original: [-of Theorem 
-]
Revised: [-of Theorem 
-]

Original: are explicitly mentioned in its formulation.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] A complete proof for

Original: Theorem 1 can be found in Appendix A

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] All

Original: the code used for our experiments, including experiment conﬁguration ﬁles allow-

Original: ing reproducibility of our results, can be found in our REMIX library repository at

Original: https://anonymous.4open.science/r/remix-15FB/.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] Details concerning all training and model hyperparameters can

Original: be found in Appendix C.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] This can be found at the beginningof Section 4 (for compute used for rule extraction) and in Appendix C (for compute

Original: used to train models).

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4 for

Original: details.

Original: (b) Did you mention the license of the assets?

Original: [No] Assets used from other authors come

Original: from unlicensed code releases that are supplemental to their publication.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: See REMIX library at https://anonymous.4open.science/r/remix-15FB/

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No] However, code used from other authors was publicly released as

Original: part of their publications and is acknowledged in both our repository and in this paper.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [Yes] See Appendix B for details on every dataset

Original: we use.

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]

Original: A ECLAIRE’s Asymptotic BoundsA.1 Notation
Revised: A ECLAIRE’s Asymptotic Bounds A.1 Notation

Original: Throughout this section, we let | R | be the number of rules in a rule set R, T ( R ) be the set of allunique terms in rules in R, and T max ( R ) be the set of terms in the longest rule in R .
Revised: Throughout this section we let | R | be the number of rules in a rule set R T ( R ) be the set of all unique terms in rules in R and T max ( R ) be the set of terms in the longest rule in R .

Original: A.2 Lemmas
Revised: A.2 Lemmas

Original: Before showing Theorem1, we introduce two useful lemmas related to rule induction:
Revised: Before showing Theorem we introduce two useful lemmas related to rule induction:

Original: Lemma 1.
Revised: Lemma 1.

Original: If R is a rule set induced from a decision tree trained with N samples using a recursivetop-down impurity-based algorithm (e.g., C5.0), then the following must hold:
Revised: If R is a rule set induced from a decision tree trained with N samples using a recursive top-down impurity-based algorithm (e g., C5.0), then the following must hold:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, notice that a tree generated by any top-down tree induction algorithm cannot have morethan N leaf nodes.
Revised: First, notice that a tree generated by any top-down tree induction algorithm cannot have more than N leaf nodes.

Original: This is because, in the worst-case scenario, any top-down algorithm may reachits base case N times with each leaf having one training sample in it.
Revised: This is because, in the worst-case scenario, any top-down algorithm may reach its base case N times with each leaf having one training sample in it.

Original: Therefore, this implies that atree cannot have more than ( N − 1) split nodes in it.
Revised: Therefore, this implies that a tree cannot have more than ( N − 1) split nodes in it.

Original: If this would not be true, then it is trivial tosee that the binary tree would have to have at least N + 1 leaf nodes, a contradiction to our previousobservation.
Revised: If this would not be true, then it is trivial to see that the binary tree would have to have at least N + 1 leaf nodes, a contradiction to our previous observation.

Original: Furthermore, notice that during rule induction each split node can contribute with at most two uniqueterms to the rule set (one per branch) and with at most one term to a single rule (one per root-to-leafpath).
Revised: Furthermore, notice that during rule induction each split node can contribute with at most two unique terms to the rule set (one per branch) and with at most one term to a single rule (one per root-to-leaf path).

Original: Because of this, a single rule cannot have more than ( N − 1) unique terms in it and the rule setcannot have more than 2( N − 1) unique total terms in it.
Revised: Because of this, a single rule cannot have more than ( N − 1) unique terms in it and the rule set cannot have more than 2( N − 1) unique total terms in it.

Original: Finally, note that these bounds can be tight.
Revised: Finally, note that these bounds can be tight.

Original: This can be seen in the decision tree shown in Figure 4 where each split node must be unique as nogains can be obtained from partitioning the training set twice in the same way in any given path.
Revised: This can be seen in the decision tree shown in Figure 4 where each split node must be unique as no gains can be obtained from partitioning the training set twice in the same way in any given path.

Original: This lemma can be extended to provide an asymptotic bound on the number of rules induced from adecision tree:
Revised: This lemma can be extended to provide an asymptotic bound on the number of rules induced from a decision tree:

Original: Lemma 2.
Revised: Lemma 2.

Original: If R is a rule set induced from an impurity-based top-down decision tree trained on Ndata samples, then it must be the case that | R | =
Revised: If R is a rule set induced from an impurity-based top-down decision tree trained on N data samples, then it must be the case that | R | =

Original: O ( N ) .
Revised: O ( N ) .

Original: Proof.
Revised: Proof.

Original: An immediate result of our proof of lemma 1 is that the number of leaf nodes in a decisiontree grows as O ( N ) .
Revised: An immediate result of our proof of lemma 1 is that the number of leaf nodes in a decision tree grows as O ( N ) .

Original: Therefore, because after rule induction there is a one-to-one correspondencebetween leaves in the decision tree and rules in the output rule set, the number of rules induced froma decision tree must also grow as O ( N ) .
Revised: Therefore, because after rule induction there is a one-to-one correspondence between leaves in the decision tree and rules in the output rule set, the number of rules induced from a decision tree must also grow as O ( N ) .

Original: Having built an understanding of how rule induction is affected by its training data, we now use theseresults to analyse ECLAIRE’s asymptotic behaviour.
Revised: Having built an understanding of how rule induction is affected by its training data, we now use these results to analyse ECLAIRE’s asymptotic behaviour.

Original: Theorem 1 (ECLAIRE Runtime Complexity) .
Revised: Theorem 1 (ECLAIRE Runtime Complexity) .

Revised: Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).

Revised: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .

Original: Given a  neural network with d hidden layers, suchthat there are at most m activations in any of its layers,and N training samples, ECLAIRE’s runtimegrows as a function of O (cid:16) dN 3 m (cid:17) .
Revised: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max [Equation] m p m (cid:17) .

Original: Proof.
Revised: Proof.

Original: We begin our proof with a high-level summary of ECLAIRE’s approach: for each layerh i in our network, ECLAIRE extracts an intermediate rule set R h
Revised: We begin our proof with a high-level summary of ECLAIRE’s approach: for each layer i in our network, ECLAIRE extracts an intermediate rule set R h

Original: i (cid:55)→ ˆ y using C5.0 on intermediateactivations
Revised: i (cid:55)→ ˆ y using rule extractor ψ ( · ) on intermediate activations h

Original: h i and the labels predicted by the input DNN.
Revised: i and the labels predicted by the input DNN.

Original: It then iterates over all premises in R h
Revised: It then iterates over all premises in R h

Original: i (cid:55)→ ˆ y ,extracts a rule set mapping input activations x to each premise’s truth value, and adds the relevantrules into the output set.
Revised: i (cid:55)→ ˆ y , extracts a rule set mapping input activations x to each premise’s truth value, and adds the relevant rules into the output set.

Original: This procedure implies that the total amount of work done by ECLAIRE toprocess hidden layerh i grows asymptotically as
Revised: This procedure implies that the total amount of work done by ECLAIRE to process hidden layer i grows asymptotically as

Original: O (cid:32) ( complexity of C5.0 with N samples and m features )
Revised: O (cid:32) ( complexity of ψ ( · ) when trained with N samples and m features )

Original: + ( number of rules in R h
Revised: + ( number of rules in R h

Original: i (cid:55)→ ˆ y ) × ( complexity of evaluating a rule in R h
Revised: i (cid:55)→ ˆ y ) × ( complexity of evaluating a rule in R h

Original: i (cid:55)→ ˆ y N times )
Revised: i (cid:55)→ ˆ y N times )

Revised: + ( number of rules in R h

Revised: i (cid:55)→ ˆ y ) × ( complexity of ψ ( · ) when trained with N samples and m features )

Original: + ( number of rules in R h
Revised: + ( number of rules in R h

Original: i (cid:55)→ˆ y
Revised: i (cid:55)→

Original: [Equation]
Revised: [Equation]

Original: × ( complexity of C5.0 with N samples and m features ) + ( number of rules in R h i (cid:55)→ ˆ y ) × ( complexity of post-processing and adding new rules )
Revised: ( complexity of post-processing and adding new rules )

Original: [Equation]
Revised: [Equation]

Original: Which can be rewritten as:
Revised: Which can be rewritten as:

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne I ( t ) x
Revised: where we deﬁne I ( t ) x

Original: (cid:55)→ p to be the temporary rule set (approximating premise p ) with the longest ruleand we deﬁne I ( r )x
Revised: (cid:55)→ p to be the temporary rule set (approximating premise p ) with the longest rule and we deﬁne I ( r )

Original:  (cid:55)→ p (cid:48)
Revised: x (cid:55)→ p (cid:48)

Original: as the temporary extracted rule set with the largest number of rules in it.
Revised: as the temporary extracted rule set with the largest number of rules in it.

Original: Note that we used the fact that C5.0’s worst case runtime grows as O ( N 2 m ) [45].
Revised: Note that we used the fact that, by assumption, ψ ( · ) ’s worst case runtime grows as O ( N p n m p m ) .

Original: Applying lemmas 1 and2 to this expression gives us
Revised: Applying lemmas 1 and 2 to this expression gives us

Original: [Equation]
Revised: [Equation]

Original: Finally, using the fact that ( N − 1) =
Revised: Finally, using the fact that ( N − 1) =

Original: O ( N ) and ( N 2 m ) =
Revised: O ( N ) and ( N 3 ) =

Original: O ( N 3 m ) , and recalling that ECLAIREperforms this much work for each hidden layer, we get that ECLAIRE’s total runtime grows as afunction of
Revised: O ( N max (3 ,p n +1) m p m ) as both p n and p m are natural numbers, and recalling that ECLAIRE performs this much work for each hidden layer, we get that ECLAIRE’s total runtime grows as a function of

Original: [Equation]
Revised: [Equation]

Original: B Dataset Details
Revised: B Dataset Details

Original: In this section we include a brief description of all the non-synthetic datasets used for the tasksdescribed in Section 4.
Revised: In this section we include a brief description of all the non-synthetic datasets used for the tasks described in Section 4.

Original: METABRIC [53] This dataset consists of a collection of anonymized features extracted from breastcancer tumours in a cohort of 1 , 980 patients.
Revised: METABRIC [45] This dataset consists of a collection of anonymized features extracted from breast cancer tumours in a cohort of 1 , 980 patients.

Original: It includes clinical traits, gene expression patterns,tumour characteristics, and survival rates for a period of 4 years.
Revised: It includes clinical traits, gene expression patterns, tumour characteristics, and survival rates for a period of 4 years.

Revised: The speciﬁc tasks we consider in

Revised: METABRIC are:

Revised: • Immunohistochemical subtype prediction (MB-ER): for this task we predict immunohistochemical subtypes in breast cancer patients using 1000 mRNA expression patterns.

Original:  MAGIC [54, 55]
Revised: Each sample can be classiﬁed as one of two types, ER+ or ER-, which are crucial for determining a patient’s treatment. • Histological tumour subtype prediction (MB-HIST): for this task we predict histological subtypes of breast cancer tumours using 1004 mRNA expression proﬁles. Each sample can be classiﬁed as either Invasive Lobular Carcinoma (ILC) or Invasive Ductal Carcinoma (IDC), two most common breast cancer histological subtypes [23]. MAGIC [46, 47]

Original: This is a particle physics dataset in which a signal needs to be classiﬁed as being ahigh-energy gamma ray or some background hadron cosmic radiation.
Revised: This is a particle physics dataset in which a signal needs to be classiﬁed as being a high-energy gamma ray or some background hadron cosmic radiation.

Original: Each of the 19,020 trainingsamples consists of 10 real-valued features that are generated via a Monte Carlo program and a binarylabel indicating whether the observation corresponds as a high-energy gamma ray (signal) or somebackground hadron radiation.
Revised: Each of the 19,020 training samples consists of 10 real-valued features that are generated via a Monte Carlo program and a binary label indicating whether the observation corresponds as a high-energy gamma ray (signal) or some background hadron radiation.

Original: MiniBooNE [61, 55]
Revised: MiniBooNE [55, 47]

Original: This is a particle physics dataset in which one is interested in discriminatingelectron neutrino events from background events in interactions collected in the MiniBooNE experiment.
Revised: This is a particle physics dataset in which one is interested in discriminating electron neutrino events from background events in interactions collected in the MiniBooNE experiment.

Original: Each of the 130,065 training samples consists of 50 real-valued features empirically collectedin the MiniBooNE experiment and a binary label indicating whether the sample represents a electronneutrino event (signal) or a background event.
Revised: Each of the 130,065 training samples consists of 50 real-valued features empirically collected in the MiniBooNE experiment and a binary label indicating whether the sample represents a electron neutrino event (signal) or a background event.

Original: Letter Recognition [57, 55]
Revised: Letter Recognition [49, 47]

Original: This dataset consists of 20 , 000 representations of black-and-white
Revised: This dataset consists of 20 , 000 representations of black-and-white

Original: English capital letters labelled with one of 26 classes (A to Z).
Revised: English capital letters labelled with one of 26 classes (A to Z).

Original: Each sample is generated by extracting16 statistical features from the images of each letter.
Revised: Each sample is generated by extracting 16 statistical features from the images of each letter.

Original: A summary of the properties of all tasks spanning from these datasets can be found in Table 2.
Revised: A summary of the properties of all tasks spanning from these datasets can be found in Table [Table]

Original: [Table]
Revised: [Table]

Original: C Model & Hyperparameter Selection 
Revised: C Model & Hyperparameter Selection C.1 Model Selection Details

Original: C.1 Model Selection Details

Original: For each of our tasks, we select a DNN architecture by iterating over several possible architectures andselecting the one with the highest testing performance for rule extraction.
Revised: For each of our tasks, we select a DNN architecture by iterating over several possible architectures and selecting the one with the highest testing performance for rule extraction.

Original: For the sake of simplicity,this search process consists of ﬁrst ﬁxing the number of hidden layers in the network for each task,and then performing a grid-search over different hyperparameters of MLPs with that many hiddenlayers.
Revised: For the sake of simplicity, this search process consists of ﬁrst ﬁxing the number of hidden layers in the network for each task, and then performing a grid-search over different hyperparameters of MLPs with that many hidden layers.

Original: In our search, we allow intermediate activations sizes to be chosen from { 8 , 16 , 32 , 64 , 128 } and activation functions to be chosen from { tanh , relu , elu } .
Revised: In our search, we allow intermediate activations sizes to be chosen from { 8 , 16 , 32 , 64 , 128 } and activation functions to be chosen from { tanh , relu , elu } .

Original: Furthermore, to constrain the searchspace further, we force the number of activations in hidden layer ( i + i ) to be less than the number ofactivations in layer i .
Revised: Furthermore, to constrain the search space further, we force the number of activations in hidden layer ( i + i ) to be less than the number of activations in layer i .

Original: This goes in line with several traditional encoding architectures [34].
Revised: This goes in line with several traditional encoding architectures [26].

Original: Unless speciﬁed otherwise, we train each model to minimise its weighted classiﬁcation cross-entropyloss for epochs ∈ { 50 , 100 , 150 , 200 } using stochastic gradient descent [66] (batch size in { 16 , 32 } )and use an Adam optimiser [67] with its default parameters ( β 1 = 0 .
Revised: Unless speciﬁed otherwise, we train each model to minimise its weighted classiﬁcation cross-entropy loss for epochs ∈ { 50 , 100 , 150 , 200 } using stochastic gradient descent [60] (batch size in { 16 , 32 } ) and use an Adam optimiser [61] with its default parameters ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 7 , andlr = 10 − 3 ) for computing weight updates.
Revised: 10 − 7 , and lr = 10 − 3 ) for computing weight updates.

Original: To speed this process up, we made use of GPU clustersoffered by Google’s colab services.
Revised: To speed this process up, we made use of GPU clusters offered by Google’s colab services.

Original: C.2 Rule Extraction Fine Tuning
Revised: C.2 Rule Extraction Fine Tuning

Original: For all of the rule extraction methods we experiment with in our evaluation, we attempt several valuesfor the minimum number of samples per split (i.e., µ ) and report only on the rule set that performedthe best on the testing dataset.
Revised: For all of the rule extraction methods we experiment with in our evaluation, we attempt several values for the minimum number of samples per split (i.e., µ ) and report only on the rule set that performed the best on the testing dataset.

Original: Because this search is very task-speciﬁc (due to different datasetsizes and/or architectures), in all of our tasks we deﬁne three evaluation parameters for each ruleextraction algorithm, namely µ min , µ max , and δµ , and evaluate the performance of rules extracted viathat method using values of µ ∈ { µ min , ( µ min + δµ ) , ( µ min + 2 δµ ) , · · · , µ max } .
Revised: Because this search is very task-speciﬁc (due to different dataset sizes and/or architectures), in all of our tasks we deﬁne three evaluation parameters for each rule extraction algorithm, namely µ min , µ max , and δµ , and evaluate the performance of rules extracted via that method using values of µ ∈ { µ min , ( µ min + δµ ) , ( µ min + 2 δµ ) , · · · , µ max } .

Original: The values deﬁningthe search space for each algorithm in our different tasks are described in the following section.
Revised: The values deﬁning the search space for each algorithm in our different tasks are described in the following section.

Original: C.3 Task-speciﬁc Conﬁgurations
Revised: C.3 Task-speciﬁc Conﬁgurations

Original: In this section we describe the conﬁguration that our grid search produces for each of the tasks wediscuss in Section 4.
Revised: In this section we describe the conﬁguration that our grid search produces for each of the tasks we discuss in Section 4.

Original: We also include a description of the search space used for µ when evaluatingdifferent rule extraction methods.
Revised: We also include a description of the search space used for µ when evaluating different rule extraction methods.

Original: C.3.1 XOR
Revised: C.3.1 XOR

Original: Given the simplicity of the XOR task, we constrain the architecture search space to be only over
Revised: Given the simplicity of the XOR task, we constrain the architecture search space to be only over

Original: MLPs using 3 hidden layers.
Revised: MLPs using 3 hidden layers.

Original: This results in the best performing model having hidden layers withsizes {64, 32, 16} and tanh activation between them.
Revised: This results in the best performing model having hidden layers with sizes {64, 32, 16} and tanh activation between them.

Original: This model is then trained for 150 epochsusing a batch size of 16.
Revised: This model is then trained for 150 epochs using a batch size of 16.

Original: When ﬁne-tuning the different rule extractors, we use µ min = 2 , µ max = 15 , and δµ = 1 for
Revised: When ﬁne-tuning the different rule extractors, we use µ min = 2 , µ max = 15 , and δµ = 1 for

Original: ECLAIRE, C5.0, and PedC5.0 as they all terminate relatively quickly and obtain good performancewithout much pruning.
Revised: ECLAIRE, C5.0, and PedC5.0 as they all terminate relatively quickly and obtain good performance without much pruning.

Original: REM-D and DeepRED * , however, fail to terminate before their allotted timeswhen using values of µ below 25 .
Revised: REM-D and DeepRED * , however, fail to terminate before their allotted times when using values of µ below 25 .

Original: Because of this, we evaluate them using values of µ in the rangedeﬁned by µ min = 25 , µ max = 35 , and δµ = 1 .
Revised: Because of this, we evaluate them using values of µ in the range deﬁned by µ min = 25 , µ max = 35 , and δµ = 1 .

Original: C.3.2 METABRIC
Revised: C.3.2 METABRIC

Original: For both METABRIC tasks described in Section 4.2, we use the exact same architecture and trainingprocess used by Shams et al.
Revised: For both METABRIC tasks described in Section 4.2, we use the exact same architecture and training process used by Shams et al.

Original: in [31], which was determined through a similar grid search process.
Revised: in [23], which was determined through a similar grid search process.

Original: This architecture consists of an MLP with hidden layers with sizes { 128 , 16 } and tanh activations inbetween them.
Revised: This architecture consists of an MLP with hidden layers with sizes { 128 , 16 } and tanh activations in between them.

Original: For both tasks we train the model for 150 epochs using a batch size of 16.
Revised: For both tasks we train the model for 150 epochs using a batch size of 16.

Original: In our rule extraction ﬁne-tuning, we use µ min = 2 , µ max = 15 , and δµ = 1 for C5.0, PedC5.0, and
Revised: In our rule extraction ﬁne-tuning, we use µ min = 2 , µ max = 15 , and δµ = 1 for C5.0, PedC5.0, and

Original: ECLAIRE.
Revised: ECLAIRE.

Original: As suggested by [31], and due to their longer run times, we use µ min = 5 , µ max = 15 , andδµ = 5 for REM-D and DeepRED * .
Revised: As suggested by [23], and due to their longer run times, we use µ min = 5 , µ max = 15 , and δµ = 5 for REM-D and DeepRED * .

Original: C.3.3 MAGIC
Revised: C.3.3 MAGIC

Original: In the MAGIC dataset results reported in Section 4.3, we search over architectures with 3 hiddenlayers and our grid-search results in the best model having layers of size {64, 32, 16} with ReLUactivations in between them.
Revised: In the MAGIC dataset results reported in Section 4.3, we search over architectures with 3 hidden layers and our grid-search results in the best model having layers of size {64, 32, 16} with ReLU activations in between them.

Original: The best training conﬁguration found is then trained for 200 epochswith a batch size of 32.
Revised: The best training conﬁguration found is then trained for 200 epochs with a batch size of 32.

Original: In our rule extraction ﬁne-tuning, for ECLAIRE we use µ min = 50 , µ max = 200 , and δµ = 25 whilefor both REM-D and DeepRED * we use µ min = 500 , µ max = 1000 , and δµ = 50 .
Revised: In our rule extraction ﬁne-tuning, for ECLAIRE we use µ min = 50 , µ max = 200 , and δµ = 25 while for both REM-D and DeepRED * we use µ min = 500 , µ max = 1000 , and δµ = 50 .

Original: Note that weuse large values of µ for REM-D and DeepRED * as we found it extremely hard to get reasonableextraction times when using less than 500 samples for µ .
Revised: Note that we use large values of µ for REM-D and DeepRED * as we found it extremely hard to get reasonable extraction times when using less than 500 samples for µ .

Original: Finally, for C5.0 and PedC5.0 we searchover all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .
Revised: Finally, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .

Original: C.3.4 MiniBooNE
Revised: C.3.4 MiniBooNE

Original: We found the MiniBooNE task to require the most capacity to obtain good results compared toothers experiments in this paper.
Revised: We found the MiniBooNE task to require the most capacity to obtain good results compared to others experiments in this paper.

Original: We force our model architecture to use 5 hidden layers and searchfor models trained with epochs ∈ { 20 , 30 , 40 } given the large training size.
Revised: We force our model architecture to use 5 hidden layers and search for models trained with epochs ∈ { 20 , 30 , 40 } given the large training size.

Original: This gives us a bestperforming architecture that uses hidden units {128, 64, 32, 16, 8} with an ELU activation in betweenthem which is trained for 30 epochs with a batch size of 16.
Revised: This gives us a best performing architecture that uses hidden units {128, 64, 32, 16, 8} with an ELU activation in between them which is trained for 30 epochs with a batch size of 16.

Original: In our rule extraction ﬁne-tuning, this task proved to be more complicated than the rest given itstraining size.
Revised: In our rule extraction ﬁne-tuning, this task proved to be more complicated than the rest given its training size.

Original: For ECLAIRE we use µ min = 0 .
Revised: For ECLAIRE we use µ min = 0 .

Original: 0005 N , µ max = 0 .
Revised: 0005 N , µ max = 0 .

Original: 0015 N , and δµ = 0 .
Revised: 0015 N , and δµ = 0 .

Original: 0001 N (where
Revised: 0001 N (where

Original: N is the number of training samples) as values below 0 .
Revised: N is the number of training samples) as values below 0 .

Original: N result extraction times longer than 6hours.
Revised: N result extraction times longer thanhours.

Original: For both REM-D and DeepRED * , we increase the minimum value of µ signiﬁcantly to getruns that terminate in their allotted times and use µ min = 0 .
Revised: For both REM-D and DeepRED * , we increase the minimum value of µ signiﬁcantly to get runs that terminate in their allotted times and use µ min = 0 .

Original: 02 N , µ max = 0 .
Revised: 02 N , µ max = 0 .

Original: 1 N , and δµ =
Revised: 1 N , and δµ =

Original: 005 N .
Revised: 005 N .

Original: Finally, given their fast extraction times, for C5.0 and PedC5.0 we search over all µ values deﬁned byµ min = 5 , µ max = 50 , and δµ = 5 .
Revised: Finally, given their fast extraction times, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .

Original: C.3.5 Letter Recognition
Revised: C.3.5 Letter Recognition

Original: For the results we report on the Letter Recognition dataset in Section 4.4, we search over architectureswith 2 hidden layers and obtain a best performing model that has layers of size {128, 64} with ELUactivations in between them.
Revised: For the results we report on the Letter Recognition dataset in Section 4.4, we search over architectures with 2 hidden layers and obtain a best performing model that has layers of size {128, 64} with ELU activations in between them.

Original: The best training conﬁguration is trained for 150 epochs with a batchsize of 32.
Revised: The best training conﬁguration is trained for 150 epochs with a batch size of 32.

Original: In our rule extraction ﬁne-tuning, we use µ min = 5 , µ max = 15 , and δµ = 1 for ECLAIRE, C5.0,and PedC5.0.
Revised: In our rule extraction ﬁne-tuning, we use µ min = 5 , µ max = 15 , and δµ = 1 for ECLAIRE, C5.0, and PedC5.0.

Original: As in the physics datasets, both DeepRED * and REM-D struggle to terminate in areasonable amount of time unless µ is signiﬁcantly high.
Revised: As in the physics datasets, both DeepRED * and REM-D struggle to terminate in a reasonable amount of time unless µ is signiﬁcantly high.

Original: This is exacerbated by the large number ofclasses in this task.
Revised: This is exacerbated by the large number of classes in this task.

Original: Because of this, we use µ min = 0 .
Revised: Because of this, we use µ min = 0 .

Original: 25 N , µ max = 0 .
Revised: 25 N , µ max = 0 .

Original: 5 N , and δµ =
Revised: 5 N , and δµ =

Original: 05 N duringtheir ﬁne-tuning process.
Revised: 05 N during their ﬁne-tuning process.

Revised: CART as Intermediate Rule Extractor in Letters

Revised: While in all of our binary tasks we are able to construct high-performing rule sets with C5.0, we fail to observe this same trend in the Letters dataset, our task of choice for multi-class evaluation.

Revised: More speciﬁcally, Table 1 shows that end-to-end

Revised: C5.0 rule sets are unable to achieve a high performance compared to that previously reported for other induction algorithms [62].

Revised: Therefore, in this section we explore the use of CART [63] trees for intermediate rule induction and show these results in Table 3.

Original: D DeepRED’s and REM-D’s Substitution Step
Revised: In these experiments, we compare the performance of rule sets induced from end-to-end CART trees, as well as rule sets induced from CART trees learnt from data that was labelled using the DNN’s predictions (which we refer to as PedCART ), against that of rule sets extracted with ECLAIRE when CART is used as its intermediate rule extractor. Furthermore, for the sake of obtaining a fair comparison between our baselines that is unbiased with respect to the choice of rule extraction algorithm, we also compare ECLAIRE against versions of both REM-D and DeepRED that use CART as an intermediate rule extractor.

Revised: For clarity, we refer to these versions of our baselines as ECLAIRE CART , REM-D CART , and DeepRED CART , respectively.

Revised: In all of these experiments, we control the growth of CART-generated trees using Cost Complexity

Revised: Pruning (CCP) [64] and by varying the number of minimum samples per split µ as in our previous tasks.

Revised: For CART, PedCART, and ECLAIRE CART we search over the spectrum deﬁned by µ min = 0 .

Revised: 0001 N , µ max = 0 .

Revised: 0051 N , and δµ =

Revised: 0005 N .

Revised: As it was the case when using C5.0 as an intermediate rule extractor, for both DeepRED CART and REM-D CART we require signiﬁcantly higher values of µ to terminate in the allotted time.

Revised: Because of this, we limit our search over µ to be over the spectrum deﬁned by µ min = 0 .

Revised: 1 N , µ max = 0 .

Revised: 3 N , and δµ =

Revised: 05 N .

Revised: This results in overpruned rule sets in DeepRED CART which, although small in size, take orders of magnitude more time to extract than those generated by ECLAIRE.

Original: [Figure]
Revised: [Table]

Original: example,
Revised: Our results show a signiﬁcant increase in accuracy when using end-to-end CART trees for rule induction over C5.0 trees (86% ± 0.4% vs 63.1% ± 2.5%). However, we note that this increase comes with a signiﬁcant rise in the number of rules extracted from CART trees compared to rule sets extracted from C5.0 trees. Regardless, our results also show that the same relative ranking we observe across our baselines in Table 1 holds when using CART as intermediate rule extractor: both ECLAIRE CART and ECLAIRE ∗ CART are able to extract rule sets with higher ﬁdelity than those extracted by PedCART, while ECLAIRE ∗ CART extracts rule sets that achieve a higher predictive accuracy than those extracted by both end-to-end CART trees and PedCART. Similarly, we observe that DeepRED CART and REM-D CART are unable to extract rule sets that perform better than random.

Original: -D DeepRED is substituting all terms inside an rule
Revised: D DeepRED’s and REM-D’s Substitution Step

Original: (cid:55)→ l which contains a single rule ( t 1 ∧ t 2 ) ⇒ l whose premise has two different terms t 1 and t

Original: The algorithm produces temporary rule sets
Revised: The algorithm produces temporary rule sets

Original: I h i (
Revised: I h i (

Original: cid:55)→ t k to map activations in layer h
Revised: cid:55)→ t k to map activations in layer h

Original: i to the binary truth value of term t k .
Revised: i to the binary truth value of term t k .

Original: It then replaces each term t k in R
Revised: It then replaces each term t k in R

Original: h
Revised: h

Original: i +1 (cid:55)→ l with the set of rules in I
Revised: i +1 (cid:55)→ l with the set of rules in I

Original: h i (cid:55)→
Revised: h i (cid:55)→

Original: t
Revised: t

Original: k that have
Revised: k that have

Original: TRUE as a conclusion.
Revised: TRUE as a conclusion.

Original: Similarly to Figure 2, we let p ( k ) j be the premise of the j -th rule in I
Revised: Similarly to Figure 2, we let p ( k ) j be the premise of the j -th rule in I

Original: h i (cid:55)→ t k whose consequence is TRUE .
Revised: h i (cid:55)→ t k whose consequence is TRUE .

Original: For this example, we assume that the number of premises with 
Revised: For this example, we assume that the number of premises with TRUE as a conclusion in I h i (cid:55)→ t 1 and I h i (cid:55)→ t 2 is N 1 and N 2 , respectively. This results in rule set R h i (cid:55)→ l having N 1 N 2 rules after substitution.

Original: E Growth Coping Mechanisms
Revised: E Growth Coping Mechanisms

Original: Although ECLAIRE can scale to large datasets and models, its cubic growth factor implies that itcan suffer from slow extraction times.
Revised: Although ECLAIRE can scale to large datasets and models, its cubic growth factor implies that it can suffer from slow extraction times.

Original: In this section, we include results of experiments in whichwe explore whether it is possible to alleviate these scaling issues without signiﬁcantly sacriﬁcingperformance.
Revised: In this section, we include results of experiments in which we explore whether it is possible to alleviate these scaling issues without signiﬁcantly sacriﬁcing performance.

Original: Speciﬁcally, we explore four different mechanisms:
Revised: Speciﬁcally, we explore four different mechanisms:

Original: Intermediate Rule Pruning
Revised: Intermediate Rule Pruning

Original: [Figure]

Original: The simplest mechanism for controlling both runtime and comprehensibility is to constrain the sizeof intermediate rule sets by increasing the value of µ .
Revised: The simplest mechanism for controlling both runtime and comprehensibility is to constrain the size of intermediate rule sets by increasing the value of µ .

Original: As seen in Figure 6, varying this parameter canresult in a drastic drop in the number of output rules.
Revised: As seen in Figure 6, varying this parameter can result in a drastic drop in the number of output rules.

Original: However, this may come at the cost of ﬁdelity.
Revised: However, this may come at the cost of ﬁdelity.

Original: This behaviour underlines one of the biggest limitations of ECLAIRE: its sensitivity to µ makes
Revised: This behaviour underlines one of the biggest limitations of ECLAIRE: its sensitivity to µ makes

Original: ECLAIRE hard to use in large tasks without the need for extensive ﬁne-tuning.
Revised: ECLAIRE hard to use in large tasks without the need for extensive ﬁne-tuning.

Original: Nevertheless, inmost tasks we observe that values of µ in the [10 − 5 N, 10 − 4 N ] range tend to give good performance.
Revised: Nevertheless, in most tasks we observe that values of µ in the [10 − 5 N, 10 − 4 N ] range tend to give good performance.

Original: Moreover, these experiments also suggest that if one can afford a small drop in performance in favourof a more comprehensible rule set, then increasing µ to be in the range [2 × 10 − 4 N, 5 × 10 − 4 N ] canlead to a very comprehensible yet still high-performing rule set.
Revised: Moreover, these experiments also suggest that if one can afford a small drop in performance in favour of a more comprehensible rule set, then increasing µ to be in the range [2 × 10 − 4 N, 5 × 10 − 4 N ] can lead to a very comprehensible yet still high-performing rule set.

Original: Hidden Representation Subsampling
Revised: Hidden Representation Subsampling

Original: When dealing with networks with multiple layers in them, one can alleviate ECLAIRE’s scalabilityby extracting intermediate rule sets
Revised: When dealing with networks with multiple layers in them, one can alleviate ECLAIRE’s scalability by extracting intermediate rule sets

Original: R h i (cid:55)→
Revised: R h i (cid:55)→

Original: ˆ y only for a subset of hidden layers i ∈ S ⊂ { 1 , 2 , . . .
Revised: ˆ y only for a subset of hidden layers i ∈ S ⊂ { 1 , 2 , . . .

Original: , d } .
Revised: , d } .

Original: This forces ECLAIRE to use fewer intermediate representations to build its rule set and results in thealgorithm effectively operating on a DNN with |S| < d hidden layers in it.
Revised: This forces ECLAIRE to use fewer intermediate representations to build its rule set and results in the algorithm effectively operating on a DNN with |S| < d hidden layers in it.

Original: We explore this possibilityin our MiniBooNE network by sampling intermediate layers with different frequencies.
Revised: We explore this possibility in our MiniBooNE network by sampling intermediate layers with different frequencies.

Original: Our results,shown in Figure 7, suggest that one can sample hidden layers at very high frequencies (i.e., every 3hidden layers) at a very small cost to ﬁdelity; all while halving both the runtime and rule set size.
Revised: Our results, shown in Figure 7, suggest that one can sample hidden layers at very high frequencies (i.e., everyhidden layers) at a very small cost to ﬁdelity; all while halving both the runtime and rule set size.

Original: One may therefore beneﬁt from subsampling hidden layers if the input network is deep enough tosupport this mechanism.
Revised: One may therefore beneﬁt from subsampling hidden layers if the input network is deep enough to support this mechanism.

Original: Training Set Subsampling
Revised: Training Set Subsampling

Original: If the scalability issue comes from the number of training samples, one can use only a fraction ofthe available training data for ECLAIRE’s rule construction.
Revised: If the scalability issue comes from the number of training samples, one can use only a fraction of the available training data for ECLAIRE’s rule construction.

Original: Our experiments in MiniBooNE showthat ECLAIRE is very data efﬁcient , maintaining its performance in low-data regimes.
Revised: Our experiments in MiniBooNE show that ECLAIRE is very data efﬁcient , maintaining its performance in low-data regimes.

Original: These results,shown in Figure 8, indicate that one can subsample up to 50% of the training data while incurring avery small drop in ﬁdelity and reducing the extraction time in half.
Revised: These results, shown in Figure 8, indicate that one can subsample up to 50% of the training data while incurring a very small drop in ﬁdelity and reducing the extraction time in half.

Original: This behaviour is in line withempirical evidence by Zilke et al.
Revised: This behaviour is in line with empirical evidence by Zilke et al.

Original: suggesting that other decompositional methods can maintain theirperformance while using a fraction of the DNN’s training set [33].
Revised: suggesting that other decompositional methods can maintain their performance while using a fraction of the DNN’s training set [25].

Original: Intermediate Rule Ranking
Revised: Intermediate Rule Ranking

Original: [Figure] It has been previously observed that one can drop signiﬁcant portions of a rule set while incurringonly in small performance costs [47, 68].
Revised: It has been previously observed that one can drop signiﬁcant portions of a rule set while incurring only in small performance costs [40, 65].

Original: Inspired by this observation, we explore whether ECLAIREcan beneﬁt from dropping the lowest p % of rules in intermediate rule sets R h
Revised: Inspired by this observation, we explore whether ECLAIRE can beneﬁt from dropping the lowest p % of rules in intermediate rule sets R h

Original: i
Revised: i

Original: (cid:55)→ ˆ y as ranked by theirconﬁdence level 5 .
Revised: (cid:55)→ ˆ y as ranked by their conﬁdence level 3 .

Original: Our results, shown in Figure 9, suggest that ECLAIRE is robust to substantialpruning of its intermediate rules without loosing much of its performance.
Revised: Our results, shown in Figure 9, suggest that ECLAIRE is robust to substantial pruning of its intermediate rules without loosing much of its performance.

Original: For example, one can dropabout 25% of all intermediate rules while experiencing a very small cost in ﬁdelity; all while usingless resources and generating fewer rules.
Revised: For example, one can drop about 25% of all intermediate rules while experiencing a very small cost in ﬁdelity; all while using less resources and generating fewer rules.

Original: Both of these results suggest that dropping a small fractionof rules can help scaling ECLAIRE to large tasks.
Revised: Both of these results suggest that dropping a small fraction of rules can help scaling ECLAIRE to large tasks.

Original: [Figure]