Original: Efﬁcient Decompositional Rule Extraction for Deep Neural Networks
Revised: Efﬁcient Decompositional Rule Extraction for Deep Neural Networks

Original: Abstract
Revised: Abstract

Original: In recent years, there has been signiﬁcant work on increasing both interpretability and debuggability of a Deep Neural Network (DNN) by extracting a rule-based model that approximates its decision boundary.
Revised: In recent years, there has been signiﬁcant work on increasing both interpretability and debuggability of a Deep Neural Network (DNN) by extracting a rule-based model that approximates its decision boundary.

Original: Nevertheless, current DNN rule extraction methods that consider a DNN’s latent space when extracting rules, known as decompositional algorithms, are either restricted to single-layer DNNs or intractable as the size of the DNN or data grows.
Revised: Nevertheless, current DNN rule extraction methods that consider a DNN’s latent space when extracting rules, known as decompositional algorithms, are either restricted to single-layer DNNs or intractable as the size of the DNN or data grows.

Original: In this paper, we address these limitations by introducing ECLAIRE, a novel polynomial-time rule extraction algorithm capable of scaling to both large DNN architectures and large training datasets.
Revised: In this paper, we address these limitations by introducing ECLAIRE, a novel polynomial-time rule extraction algorithm capable of scaling to both large DNN architectures and large training datasets.

Original: We evaluate ECLAIRE on a wide variety of tasks, ranging from breast cancer prognosis to particle detection, and show that it consistently extracts more accurate and comprehensible rule sets than the current state-of-the-art methods while using orders of magnitude less computational resources.
Revised: We evaluate ECLAIRE on a wide variety of tasks, ranging from breast cancer prognosis to particle detection, and show that it consistently extracts more accurate and comprehensible rule sets than the current state-of-the-art methods while using orders of magnitude less computational resources.

Original: We make all of our methods available, including a rule set visualisation interface, through the open-source REMIX library (https://github.com/mateoespinosa/remix).
Revised: We make all of our methods available, including a rule set visualisation interface, through the open-source REMIX library (https://github.com/mateoespinosa/remix).

Original:  Introduction
Revised:  Introduction

Original: As the ﬁeld of Artiﬁcial Intelligence (AI) transitions from traditional symbolic methods into opaque models like Deep Neural Networks (DNNs) [1], there has been a great amount of concern over the potential consequences of using these methods in safety-critical environments [2, 3].
Revised: As the ﬁeld of Artiﬁcial Intelligence (AI) transitions from traditional symbolic methods into opaque models like Deep Neural Networks (DNNs) [1], there has been a great amount of concern over the potential consequences of using these methods in safety-critical environments [2, 3].

Original: The use of such “black-box” models in practice can not only lead to uncertainty and delays when debugging and root-causing mistakes in a model’s behaviour (as in Uber’s infamous accident [4, 5]), but it also goes against the long-held conventions of using transparent, interpretable models for life threatening decisions [6, 7].
Revised: The use of such “black-box” models in practice can not only lead to uncertainty and delays when debugging and root-causing mistakes in a model’s behaviour (as in Uber’s infamous accident [4, 5]), but it also goes against the long-held conventions of using transparent, interpretable models for life threatening decisions [6, 7].

Original: Furthermore, the lack of transparency in DNNs has been shown to lead to user mistrust [8] and to models that misleadingly show promising results for the wrong reasons [9].
Revised: Furthermore, the lack of transparency in DNNs has been shown to lead to user mistrust [8] and to models that misleadingly show promising results for the wrong reasons [9].

Original: Such opacity perpetuates a barrier for the practical use of DNNs despite their celebrated success in tasks ranging from computer vision [10] and language modelling [11] to diagnosis [12].
Revised: Such opacity perpetuates a barrier for the practical use of DNNs despite their celebrated success in tasks ranging from computer vision [10] and language modelling [11] to diagnosis [12].

Original: As a reaction to these concerns, a new line of work in the ﬁeld of eXplainable AI (XAI) has been developed to explore ways of demystifying a DNN’s decision boundary and to facilitate debugging and deployment of these models.
Revised: As a reaction to these concerns, a new line of work in the ﬁeld of eXplainable AI (XAI) has been developed to explore ways of demystifying a DNN’s decision boundary and to facilitate debugging and deployment of these models.

Original: A promising proposal in this direction is to make use of the latent space learnt by a DNN to construct a rule-based model (a set or list of IF-THEN rules) that approximates its decision boundary [13, 14].
Revised: A promising proposal in this direction is to make use of the latent space learnt by a DNN to construct a rule-based model (a set or list of IF-THEN rules) that approximates its decision boundary [13, 14].

Original: Such approaches complement a DNN’s high performance by offering transparency and debuggability in the form of both “ local ” interpretability (i.e., understanding how a single prediction was made) as well as “ global ” interpretability (i.e., discovering global patterns in the data that explain how the DNN acts).
Revised: Such approaches complement a DNN’s high performance by offering transparency and debuggability in the form of both “ local ” interpretability (i.e., understanding how a single prediction was made) as well as “ global ” interpretability (i.e., discovering global patterns in the data that explain how the DNN acts).

Original: This is in contrast to existing XAI alternatives, such as feature importance methods [15, 16], sample importance methods [17, 18] and counterfactual explanations [19, 20], where a method only provides either local or global interpretability in their basic form.
Revised: This is in contrast to existing XAI alternatives, such as feature importance methods [15, 16], sample importance methods [17, 18] and counterfactual explanations [19, 20], where a method only provides either local or global interpretability in their basic form.

Original: Moreover, aided by publicly available rule set visualisation tools [21, 22], as well as the possibility of injecting expert hand-crafted rules into rule sets (e.g., as suggested in [23]), existing work and open-source frameworks for deploying rule-based models can signiﬁcantly assist model developers, legal moderators, and model users with debugging and deploying a DNN by analysing its approximating rule-based model.
Revised: Moreover, aided by publicly available rule set visualisation tools [21, 22], as well as the possibility of injecting expert hand-crafted rules into rule sets (e.g., as suggested in [23]), existing work and open-source frameworks for deploying rule-based models can signiﬁcantly assist model developers, legal moderators, and model users with debugging and deploying a DNN by analysing its approximating rule-based model.

Original: Nevertheless, previous rule extraction methods that consider the inner workings of a DNN to construct their rules, commonly referred to as “decompositional” methods [24], tend to have severe limitations.
Revised: Nevertheless, previous rule extraction methods that consider the inner workings of a DNN to construct their rules, commonly referred to as “decompositional” methods [24], tend to have severe limitations.

Original: First, with very few exceptions, most decompositional algorithms have been designed to work only on shallow architectures [14, 13].
Revised: First, with very few exceptions, most decompositional algorithms have been designed to work only on shallow architectures [14, 13].

Original: Second, to the best of our knowledge, all of the approaches that are able to operate on non-trivial DNNs suffer from intractability issues and, therefore, are impractical for tasks that require very deep architectures or have a large training dataset [25, 23].
Revised: Second, to the best of our knowledge, all of the approaches that are able to operate on non-trivial DNNs suffer from intractability issues and, therefore, are impractical for tasks that require very deep architectures or have a large training dataset [25, 23].

Original: Both of these are in stark contrast with the current direction in DNN architecture design where deep models trained on large datasets dominate over shallow models trained with small datasets [26].
Revised: Both of these are in stark contrast with the current direction in DNN architecture design where deep models trained on large datasets dominate over shallow models trained with small datasets [26].

Original: In this paper we address these limitations by proposing ECLAIRE, a novel decompositional rule extraction algorithm for DNNs.
Revised: In this paper we address these limitations by proposing ECLAIRE, a novel decompositional rule extraction algorithm for DNNs.

Original: Our method, depicted in Figure 1, avoids the intractability common to other decompositional methods by making use of a two-step process that efﬁciently uses a DNN’s latent space to guide its rule extraction.
Revised: Our method, depicted in Figure 1, avoids the intractability common to other decompositional methods by making use of a two-step process that efﬁciently uses a DNN’s latent space to guide its rule extraction.

Original: Our main contributions can be summarised as follows:
Revised: Our main contributions can be summarised as follows:

Original: • We introduce ECLAIRE, a novel decompositional rule extraction algorithm which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional method applicable to arbitrary DNNs.
Revised: • We introduce ECLAIRE, a novel decompositional rule extraction algorithm which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional method applicable to arbitrary DNNs.

Original: • We show that ECLAIRE’s rule sets outperform those extracted by current state-of-the-art decompositional methods in both predictive performance and comprehensibility (measured as the number and length of extracted rules).
Revised: • We show that ECLAIRE’s rule sets outperform those extracted by current state-of-the-art decompositional methods in both predictive performance and comprehensibility (measured as the number and length of extracted rules).

Original: Furthermore, we show that ECLAIRE is able to achieve this while using signiﬁcantly less computational resources than competing methods.
Revised: Furthermore, we show that ECLAIRE is able to achieve this while using signiﬁcantly less computational resources than competing methods.

Original: • We open-source our methods, together with a rule visualisation
Revised: • We open-source our methods, together with a rule visualisation

Original: and interaction library, as part of the REMIX package (https://github.com/mateoespinosa/remix).
Revised: and interaction library, as part of the REMIX package (https://github.com/mateoespinosa/remix).

Original:  Related Work
Revised:  Related Work

Original: Previous work on explaining a DNN’s prediction using rule-based explanations can be divided into two lines of work: (i) rule-based model-agnostic local explainability methods and (2) DNN rule extraction methods.
Revised: Previous work on explaining a DNN’s prediction using rule-based explanations can be divided into two lines of work: (i) rule-based model-agnostic local explainability methods and (2) DNN rule extraction methods.

Original: While members of the former family of methods, which include algorithms such as LORE [27] and Anchors [28], are able to generate rule-based explanations for the behaviour of a
Revised: While members of the former family of methods, which include algorithms such as LORE [27] and Anchors [28], are able to generate rule-based explanations for the behaviour of a

Original: DNN in a point’s local neighbourhood, they are unable to provide insights on any global patterns captured by the DNN’s decision boundary.
Revised: DNN in a point’s local neighbourhood, they are unable to provide insights on any global patterns captured by the DNN’s decision boundary.

Original: Hence, given that we are interested in extracting rule-based models that provide both local and global interpretability in a DNN, in this section we focus on DNN rule extraction methods.
Revised: Hence, given that we are interested in extracting rule-based models that provide both local and global interpretability in a DNN, in this section we focus on DNN rule extraction methods.

Original: The design of DNN rule extraction methods can be divided following a taxonomy proposed by
Revised: The design of DNN rule extraction methods can be divided following a taxonomy proposed by

Original: Andrews et al.
Revised: Andrews et al.

Original: [24] in which a method is classiﬁed as being in one of three possible categories: “ pedagogical ”, “ decompositional ”, and “ eclectic ”.
Revised: [24] in which a method is classiﬁed as being in one of three possible categories: “ pedagogical ”, “ decompositional ”, and “ eclectic ”.

Original: Pedagogical methods, such as HYPINV [29],
Revised: Pedagogical methods, such as HYPINV [29],

Original: TREPAN [30], and Validity Interval Analysis [31] extract rules from a DNN by treating it as a black box.
Revised: TREPAN [30], and Validity Interval Analysis [31] extract rules from a DNN by treating it as a black box.

Original: A limitation of these methods, however, is that by ignoring a DNN’s latent space, they do not take advantage of different hierarchical representations that a DNN was able to learn from its training set.
Revised: A limitation of these methods, however, is that by ignoring a DNN’s latent space, they do not take advantage of different hierarchical representations that a DNN was able to learn from its training set.

Original: In this work we are concerned with extracting rules from generic multi-layered architectures, and thus in this section we focus on discussing only those decompositional methods that are able to extract rules from arbitrary DNNs.
Revised: In this work we are concerned with extracting rules from generic multi-layered architectures, and thus in this section we focus on discussing only those decompositional methods that are able to extract rules from arbitrary DNNs.

Original: This focus renders the majority of decompositional approaches, such as those in [32, 33, 34, 35, 36], out of scope for our current work due to their restrictive application to only a subset of architectures.
Revised: This focus renders the majority of decompositional approaches, such as those in [32, 33, 34, 35, 36], out of scope for our current work due to their restrictive application to only a subset of architectures.

Original: Instead, we bring attention to the only two decompositional algorithms applicable to arbitrary DNNs: DeepRED [25] and, an optimised version of it, REM-D [23].
Revised: Instead, we bring attention to the only two decompositional algorithms applicable to arbitrary DNNs: DeepRED [25] and, an optimised version of it, REM-D [23].

Original: Borrowing inspiration from CRED [37], an earlier single-layer decompositional method, both
Revised: Borrowing inspiration from CRED [37], an earlier single-layer decompositional method, both

Original: DeepRED and REM-D use decision trees to induce rule sets that map input features to the truth value of each of the DNN’s output classes.
Revised: DeepRED and REM-D use decision trees to induce rule sets that map input features to the truth value of each of the DNN’s output classes.

Original: The crux of both methods lies in their substitution step
Revised: The crux of both methods lies in their substitution step

Original: in which a set of rules R h
Revised: in which a set of rules R h

Original: i (cid:55)→ l mapping activations in the i th layer to the truth value of class l is rewritten to be a function of activations in the ( i − 1) th layer.
Revised: i (cid:55)→ l mapping activations in the i th layer to the truth value of class l is rewritten to be a function of activations in the ( i − 1) th layer.

Original: This substitution is performed in a term-wise manner: each term of a rule’s premise in R h
Revised: This substitution is performed in a term-wise manner: each term of a rule’s premise in R h

Original: i (cid:55)→ l is replaced by a set of new rules which depend on the hidden activations of layer ( i − 1) .
Revised: i (cid:55)→ l is replaced by a set of new rules which depend on the hidden activations of layer ( i − 1) .

Original: This substitution, however, requires an exponential post-processing step in which rules substituted for a given term need to be distributed in a cartesian-product fashion with rules that were substituted for all terms in the same premise.
Revised: This substitution, however, requires an exponential post-processing step in which rules substituted for a given term need to be distributed in a cartesian-product fashion with rules that were substituted for all terms in the same premise.

Original: This results in both REM-D and
Revised: This results in both REM-D and

Original: DeepRED having an exponential asymptotic runtime [23].
Revised: DeepRED having an exponential asymptotic runtime [23].

Original: While algorithmically similar, these methods differ in two implementation-wise optimisations: (1)
Revised: While algorithmically similar, these methods differ in two implementation-wise optimisations: (1)

Original: REM-D uses C5.0 [38] for intermediate decision tree construction rather than C4.5 [39] (a less efﬁcient iteration of C5.0 with typically worse performance) and (2) REM-D reduces memory consumption by substituting intermediate rules as soon as they have been extracted.
Revised: REM-D uses C5.0 [38] for intermediate decision tree construction rather than C4.5 [39] (a less efﬁcient iteration of C5.0 with typically worse performance) and (2) REM-D reduces memory consumption by substituting intermediate rules as soon as they have been extracted.

Original: REM-D’s use of
Revised: REM-D’s use of

Original: C5.0 for intermediate rule extraction, and its early substitution, allow it to empirically outperform
Revised: C5.0 for intermediate rule extraction, and its early substitution, allow it to empirically outperform

Original: DeepRED in terms of its scalability as well as in the size and ﬁdelity of its rule sets [40].
Revised: DeepRED in terms of its scalability as well as in the size and ﬁdelity of its rule sets [40].

Original: The last family of rule extraction methods are eclectic algorithms, a hybrid between pedagogical and decompositional methods.
Revised: The last family of rule extraction methods are eclectic algorithms, a hybrid between pedagogical and decompositional methods.

Original: While eclectic methods such as RX [41], MofN [42], and ERENN_MHL [43] have been shown to extract competitive approximating rule sets, these algorithms either assume that the underlying DNN has only one hidden layer or they require full control over the DNN’s training pipeline (e.g., they require a speciﬁc training procedure to be used for the DNN).
Revised: While eclectic methods such as RX [41], MofN [42], and ERENN_MHL [43] have been shown to extract competitive approximating rule sets, these algorithms either assume that the underlying DNN has only one hidden layer or they require full control over the DNN’s training pipeline (e.g., they require a speciﬁc training procedure to be used for the DNN).

Original:  ECLAIRE
Revised:  ECLAIRE

Original: Problem Setup Assume we are given a set of unlabelled training samples
Revised: Problem Setup Assume we are given a set of unlabelled training samples

Original: X
Revised: X

Original: = { x ( i ) ∈
Revised: = { x ( i ) ∈

Original: R m } Ni = and a pre-trained DNN f θ : R m (cid:55)→ [0 , 1]
Revised: R m } Ni = and a pre-trained DNN f θ : R m (cid:55)→ [0 , 1]

Original: L such that for all x ∈ X , f θ ( x ) outputs a probability distribution over labels in set Y =
Revised: L such that for all x ∈ X , f θ ( x ) outputs a probability distribution over labels in set Y =

Original: { l 1 , l 2 , · · · , l L } .
Revised: { l 1 , l 2 , · · · , l L } .

Original: Furthermore, assume f θ has d hidden layers and let h i ( x ) be the output of the i -th layer of f θ when fed with input x .
Revised: Furthermore, assume f θ has d hidden layers and let h i ( x ) be the output of the i -th layer of f θ when fed with input x .

Original: In this setup, we let h 

Revised: In this setup, we let h

Original: ( x ) = x be the input activation layer and h d +1 ( x ) =
Revised: ( x ) = x be the input activation layer and h d +1 ( x ) =

Original: f θ ( x ) be f θ ’s output probability distribution.
Revised: f θ ( x ) be f θ ’s output probability distribution.

Original: Our goal is to use samples in X to construct a rule set R x
Revised: Our goal is to use samples in X to construct a rule set R x

Original: (cid:55)→ ˆ y of IF-THEN rules of the form
Revised: (cid:55)→ ˆ y of IF-THEN rules of the form

Original: IF
Revised: IF

Original: [Equation]
Revised: [Equation]

Original: (cid:122)
Revised: (cid:122)

Original: (cid:125)(cid:124) (cid:123) ( x i > v i ) ∧ ( x j ≤ v j ) ∧ · · · ∧ ( x n > v n ) (cid:124) (cid:123)(cid:122) (cid:125) premise/antecedent (cid:17)
Revised: (cid:125)(cid:124) (cid:123) ( x i > v i ) ∧ ( x j ≤ v j ) ∧ · · · ∧ ( x n > v n ) (cid:124) (cid:123)(cid:122) (cid:125) premise/antecedent (cid:17)

Original: THEN l k (cid:124)(cid:123)(cid:122)(cid:125)
Revised: THEN l k (cid:124)(cid:123)(cid:122)(cid:125)

Original: [Table]
Revised: [Table]

Original: such that, when subjected to a majority voting 1 with input x , it accurately predicts the result of evaluating arg max i f θ ( x ) i .
Revised: such that, when subjected to a majority voting 1 with input x , it accurately predicts the result of evaluating arg max i f θ ( x ) i .

Original: In alignment with existing decompositional methods (e.g., REM-D and
Revised: In alignment with existing decompositional methods (e.g., REM-D and

Original: DeepRED), in our work we constrain each rule’s premise to be a conjunction of terms of the form ( x i > v i ) or ( x i ≤ v i ) , where x i is the i -th input feature of sample x and v i ∈ R is a learnt threshold.
Revised: DeepRED), in our work we constrain each rule’s premise to be a conjunction of terms of the form ( x i > v i ) or ( x i ≤ v i ) , where x i is the i -th input feature of sample x and v i ∈ R is a learnt threshold.

Original: Efﬁcient Clause-wise Rule Extraction In order to introduce our method, we begin with the following observation: a neural network f θ with d hidden layers provides access to d distinct representations for samples x ∈ X .
Revised: Efﬁcient Clause-wise Rule Extraction In order to introduce our method, we begin with the following observation: a neural network f θ with d hidden layers provides access to d distinct representations for samples x ∈ X .

Original: This allows us to extend our support training set X by creating d new labelled training sets (cid:8) { (cid:0)
Revised: This allows us to extend our support training set X by creating d new labelled training sets (cid:8) { (cid:0)

Original: h i ( x ) , arg max i f θ ( x ) i (cid:1) | x ∈ X } (cid:9) d i =1 from which one can induce d rule sets
Revised: h i ( x ) , arg max i f θ ( x ) i (cid:1) | x ∈ X } (cid:9) d i =1 from which one can induce d rule sets

Original: R h i (cid:55)→ ˆ y that map each hidden layer’s output to f θ ’s label predictions ˆ y = arg max i f θ ( x ) i .
Revised: R h i (cid:55)→ ˆ y that map each hidden layer’s output to f θ ’s label predictions ˆ y = arg max i f θ ( x ) i .

Original: The crux of our method consists of efﬁciently unifying all d rule sets { R h 1 (cid:55)→ ˆ y , · · · , R h d (cid:55)→ ˆ y } into a single rule set
Revised: The crux of our method consists of efﬁciently unifying all d rule sets { R h 1 (cid:55)→ ˆ y , · · · , R h d (cid:55)→ ˆ y } into a single rule set

Original: R x
Revised: R x

Original: (cid:55)→ ˆ y that maps input samples x ∈ X to f θ ’s predicted labels.
Revised: (cid:55)→ ˆ y that maps input samples x ∈ X to f θ ’s predicted labels.

Original: In the same way that bagging and feature subsampling are key to the ability of random forests to reduce variance and overﬁtting [44], making use of d different representations of the same dataset to construct R x
Revised: In the same way that bagging and feature subsampling are key to the ability of random forests to reduce variance and overﬁtting [44], making use of d different representations of the same dataset to construct R x

Original: (cid:55)→ ˆ y can signiﬁcantly help reducing the variance and overﬁtting that would otherwise be present in vanilla rule induction algorithms.
Revised: (cid:55)→ ˆ y can signiﬁcantly help reducing the variance and overﬁtting that would otherwise be present in vanilla rule induction algorithms.

Original: Motivated by these observations, we introduce E fﬁcient CLA use-w I se R ule E xtraction (ECLAIRE).
Revised: Motivated by these observations, we introduce E fﬁcient CLA use-w I se R ule E xtraction (ECLAIRE).

Original: Our method, summarised in Algorithm 1, uses an input DNN f θ and a set of training samples X to incrementally construct a rule set R x
Revised: Our method, summarised in Algorithm 1, uses an input DNN f θ and a set of training samples X to incrementally construct a rule set R x

Original: (cid:55)→ ˆ y which approximates f θ (line 1 ).
Revised: (cid:55)→ ˆ y which approximates f θ (line 1 ).

Original: More speciﬁcally, ECLAIRE iterates over each hidden layer
Revised: More speciﬁcally, ECLAIRE iterates over each hidden layer

Original: h i and uses a general rule induction algorithm ψ ( · ) to induce a rule set R h i
Revised: h i and uses a general rule induction algorithm ψ ( · ) to induce a rule set R h i

Original: (cid:55)→ ˆ y that maps h
Revised: (cid:55)→ ˆ y that maps h

Original: i ’s activations to the DNN’s output predictions (lines 2 to 5 ).
Revised: i ’s activations to the DNN’s output predictions (lines 2 to 5 ).

Original: ECLAIRE then proceeds to do a clause-wise substitution of rules in R h
Revised: ECLAIRE then proceeds to do a clause-wise substitution of rules in R h

Original: i (cid:55)→ ˆ y .
Revised: i (cid:55)→ ˆ y .

Original: This substitution, depicted in Figure 2, iterates over all of the rules in R h
Revised: This substitution, depicted in Figure 2, iterates over all of the rules in R h

Original: i (cid:55)→ ˆ y (lines 6-7) and constructs a new temporary rule set I x (cid:55)→ p mapping input feature activations x to the truth value of each rule’s premise p (lines 8-9).
Revised: i (cid:55)→ ˆ y (lines 6-7) and constructs a new temporary rule set I x (cid:55)→ p mapping input feature activations x to the truth value of each rule’s premise p (lines 8-9).

Original: Note that in Algorithm 1 we use p ( x ( i ) ) to represent the result of evaluating premise p (which is a conjunction of terms) on sample x
Revised: Note that in Algorithm 1 we use p ( x ( i ) ) to represent the result of evaluating premise p (which is a conjunction of terms) on sample x

Original: ( i ) .
Revised: ( i ) .

Original: Finally, we add to R x
Revised: Finally, we add to R x

Original: (cid:55)→ ˆ
Revised: (cid:55)→ ˆ

Original: y
Revised: y

Original: all of the premises in I x
Revised: all of the premises in I x

Original: (cid:55)→ p that have TRUE as a conclusion (lines 10-12).
Revised: (cid:55)→ p that have TRUE as a conclusion (lines 10-12).

Original: In other words, we add all the rules that approximate premise p as being true and map those rules to the class which p originally predicted.
Revised: In other words, we add all the rules that approximate premise p as being true and map those rules to the class which p originally predicted.

Original: This process is repeated for all intermediate hidden layers and terminates by outputting R x
Revised: This process is repeated for all intermediate hidden layers and terminates by outputting R x

Original: (cid:55)→ ˆ y as its result.
Revised: (cid:55)→ ˆ y as its result.

Original: [Table]
Revised: [Table]

Original: ) 8: ˆ y (1) p , ˆ y (2) p , . . .
Revised: ) 8: ˆ y (1) p , ˆ y (2) p , . . .

Original: , ˆ y ( N ) p ← p (cid:0) x (cid:48) (1) (cid:1) , p (cid:0) x (cid:48) (2) (cid:1) , . . .
Revised: , ˆ y ( N ) p ← p (cid:0) x (cid:48) (1) (cid:1) , p (cid:0) x (cid:48) (2) (cid:1) , . . .

Original: , p (cid:0) x (cid:48)
Revised: , p (cid:0) x (cid:48)

Original: ( N ) (cid:1) 9:
Revised: ( N ) (cid:1) 9:

Original: I h 

Revised: I h

Original: → p ← ψ (cid:0) { ( x(cid:48) (1) , ˆ y (1) p ) , ( x(cid:48) (2) , ˆ y (2) p ) , ..., ( x(cid:48) ( N ) , ˆ y ( N ) p ) } (cid:1) 10: for clause c ∈ getPremisesByConclusion (cid:0) I h 

Revised: → p ← ψ (cid:0) { ( x (1) , ˆ y (1) p ) , ( x (2) , ˆ y (2) p ) , ..., ( x ( N ) , ˆ y ( N ) p ) } (cid:1) 10: for clause c ∈ getPremisesByConclusion (cid:0) I h

Original: → p , TRUE (cid:1) do 11:
Revised: → p , TRUE (cid:1) do 11:

Original: R x
Revised: R x

Original: (cid:55)→ ˆ y ← R x
Revised: (cid:55)→ ˆ y ← R x

Original: (cid:55)→ ˆ y ∪ (cid:110) IF c THEN getConclusion ( r ) (cid:111) 12: end for 13: end for 14: end for 15: return R x
Revised: (cid:55)→ ˆ y ∪ (cid:110) IF c THEN getConclusion ( r ) (cid:111) 12: end for 13: end for 14: end for 15: return R x

Original: (cid:55)→ ˆ y
Revised: (cid:55)→ ˆ y

Original: When using a polynomial-time rule extractor ψ ( · ) , ECLAIRE’s clause-wise substitution allows it to avoid the expensive and exponential term redistribution that is required in REM-D and DeepRED.
Revised: When using a polynomial-time rule extractor ψ ( · ) , ECLAIRE’s clause-wise substitution allows it to avoid the expensive and exponential term redistribution that is required in REM-D and DeepRED.

Original: This manifests itself in ECLAIRE’s runtime which can be shown to be as follows:
Revised: This manifests itself in ECLAIRE’s runtime which can be shown to be as follows:

Original: Theorem 1 (ECLAIRE Runtime Complexity) .
Revised: Theorem 1 (ECLAIRE Runtime Complexity) .

Original: Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).
Revised: Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).

Original: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .
Revised: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .

Original: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max (3 ,p n +1) m p m (cid:17) .
Revised: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max (3 ,p n +1) m p m (cid:17) .

Original: Theorem 1, whose proof is in Appendix A, highlights a signiﬁcant computational advantage in
Revised: Theorem 1, whose proof is in Appendix A, highlights a signiﬁcant computational advantage in

Original: ECLAIRE over previous decompositional methods: while both REM-D and DeepRED are known to have an exponential runtime with respect to the training set and the depth of the network [23],
Revised: ECLAIRE over previous decompositional methods: while both REM-D and DeepRED are known to have an exponential runtime with respect to the training set and the depth of the network [23],

Original: ECLAIRE exhibits a polynomial-time growth with respect to its input size when using a polynomial- time rule induction method for ψ ( · ) .
Revised: ECLAIRE exhibits a polynomial-time growth with respect to its input size when using a polynomial- time rule induction method for ψ ( · ) .

Original: Furthermore, ECLAIRE’s use of a DNN’s intermediate representations without regards of their topological order in the network implies that, in stark contrast to
Revised: Furthermore, ECLAIRE’s use of a DNN’s intermediate representations independently of their topological order in the network implies that, in stark contrast to

Original: REM-D and DeepRED, it is agnostic to the DNN’s network topology and it can be easily parallelised , something we take advantage of in practice by distributing the work in the main loop (lines 3-15) across n threads threads.
Revised: REM-D and DeepRED, it is agnostic to the DNN’s network topology and it can be easily parallelised , something we take advantage of in practice by distributing the work in the main loop (lines 3-15) across n threads threads.

Original: Notice that this is not possible in both REM-D and DeepRED as they extract rules in a sequential manner, processing the DNN’s layers in reverse topological order.
Revised: Notice that this is not possible in both REM-D and DeepRED as they extract rules in a sequential manner, processing the DNN’s layers in reverse topological order.

Original:  Experiments
Revised:  Experiments

Original: Setup For all experiments we perform a 5-fold stratiﬁed cross-validation and, for each metric of interest, we show both the mean and standard deviation errors across all 5 folds.
Revised: Setup For all experiments we perform a 5-fold stratiﬁed cross-validation and, for each metric of interest, we show both the mean and standard deviation errors across all 5 folds.

Original: Furthermore, all experiments are carried on a 2016 MacBook Pro with a 2.7 GHz Quad-Core Intel i7 processor and
Revised: Furthermore, all experiments are carried on a 2016 MacBook Pro with a 2.7 GHz Quad-Core Intel i7 processor and

Original: GB of RAM.
Revised: GB of RAM.

Original: Finally, we limit each individual experiment to span at most n threads = 6 threads when using ECLAIRE and we terminate an experiment if it takes longer than six hours to complete.
Revised: Finally, we limit each individual experiment to span at most n threads = 6 threads when using ECLAIRE and we terminate an experiment if it takes longer than six hours to complete.

Original: Datasets and Tasks We evaluate our methods on six tasks.
Revised: Datasets and Tasks We evaluate our methods on six tasks.

Original: We begin by using a synthetic task to contrast ECLAIRE’s rule sets to those extracted by the current state-of-the-art methods in a controlled setting.
Revised: We begin by using a synthetic task to contrast ECLAIRE’s rule sets to those extracted by the current state-of-the-art methods in a controlled setting.

Original: We then evaluate ECLAIRE on two breast cancer prognosis tasks based on the METRABRIC [45] dataset to highlight ECLAIRE’s applicability to real-world tasks in which transparency is crucial.
Revised: We then evaluate ECLAIRE on two breast cancer prognosis tasks based on the METRABRIC [45] dataset to highlight ECLAIRE’s applicability to real-world tasks in which transparency is crucial.

Original: We proceed to evaluate ECLAIRE’s scalability to large datasets and architectures by extracting rules from models trained on two large particle physics datasets (MAGIC [46, 47] and MiniBooNE [48, 47]).
Revised: We proceed to evaluate ECLAIRE’s scalability to large datasets and architectures by extracting rules from models trained on two large particle physics datasets (MAGIC [46, 47] and MiniBooNE [48, 47]).

Original: Finally, we use the Letter Recognition [49, 47] dataset to explore ECLAIRE’s ability to extend to non-binary classiﬁcation tasks.
Revised: Finally, we use the Letter Recognition [49, 47] dataset to explore ECLAIRE’s ability to extend to non-binary classiﬁcation tasks.

Original: A summary of each dataset, including its size and label distribution, can be found in Appendix B.
Revised: A summary of each dataset, including its size and label distribution, can be found in Appendix B.

Original: Baselines and Metrics In our experiments, we compare ECLAIRE against the only other two decompositional methods that share its applicability: REM-D and DeepRED.
Revised: Baselines and Metrics In our experiments, we compare ECLAIRE against the only other two decompositional methods that share its applicability: REM-D and DeepRED.

Original: Due to a lack of an up-to-date implementation of DeepRED, we follow the work in [40] and use a variation of DeepRED where C5.0 is used instead of C4.5.
Revised: Due to a lack of an up-to-date implementation of DeepRED, we follow the work in [40] and use a variation of DeepRED where C5.0 is used instead of C4.5.

Original: This results in a variant of DeepRED that outperforms its C4.
Revised: This results in a variant of DeepRED that outperforms its C4.

Original: counterpart.
Revised: counterpart.

Original: For both this implementation of DeepRED, which we refer to as DeepRED * , and the implementation of REM-D, we use as our starting point the code published by Shams et al.
Revised: For both this implementation of DeepRED, which we refer to as DeepRED * , and the implementation of REM-D, we use as our starting point the code published by Shams et al.

Original: When evaluating ECLAIRE, we use rule sets induced from C5.0 trees as our intermediate rule extractor ψ ( · ) unless speciﬁed otherwise.
Revised: When evaluating ECLAIRE, we use rule sets induced from C5.0 trees as our intermediate rule extractor ψ ( · ) unless speciﬁed otherwise.

Original: Although this constraints our evaluation to a single rule extractor (rather than using more recent rule extractors such as Bayesian rule sets [50] or Column
Revised: Although this constraints our evaluation to a single rule extractor (rather than using more recent rule extractors such as Bayesian rule sets [50] or Column

Original: Generation [51]), this allows us to fairly evaluate the algorithmic design of ECLAIRE against that of DeepRED * and REM-D without strongly biasing our results to our choice of intermediate rule extractor.
Revised: Generation [51]), this allows us to fairly evaluate the algorithmic design of ECLAIRE against that of DeepRED * and REM-D without strongly biasing our results to our choice of intermediate rule extractor.

Original: Nevertheless, notice that from Theorem 1, and the fact that C5.0’s runtime complexity grows as O ( N 2 m ) [38], it follows that the runtime complexity of this instantiation of ECLAIRE grows as
Revised: Nevertheless, notice that from Theorem 1, and the fact that C5.0’s runtime complexity grows as O ( N 2 m ) [38], it follows that the runtime complexity of this instantiation of ECLAIRE grows as

Original: O ( dN 3 m ) .
Revised: O ( dN 3 m ) .

Original: Finally, to highlight the beneﬁts of using a DNN’s latent space for rule extraction, we include two non-decompositional baselines: C5.0 as a baseline end-to-end rule induction method and
Revised: Finally, to highlight the beneﬁts of using a DNN’s latent space for rule extraction, we include two non-decompositional baselines: C5.0 as a baseline end-to-end rule induction method and

Original: PedC5.0 [23, 40] as a baseline pedagogical algorithm.
Revised: PedC5.0 [23, 40] as a baseline pedagogical algorithm.

Original: The latter method uses C5.0 to induce a rule set from a training set where each sample is labelled using the DNN’s prediction for that sample.
Revised: The latter method uses C5.0 to induce a rule set from a training set where each sample is labelled using the DNN’s prediction for that sample.

Original: When comparing any two methods, we evaluate the ﬁdelity of the extracted rule sets with respect to the input DNN (deﬁned as the predictive accuracy when sample x ( i ) is assigned label ˆ y ( i ) =
Revised: When comparing any two methods, we evaluate the ﬁdelity of the extracted rule sets with respect to the input DNN (deﬁned as the predictive accuracy when sample x ( i ) is assigned label ˆ y ( i ) =

Original: arg max j f θ ( x ( i ) )
Revised: arg max j f θ ( x ( i ) )

Original: j ) and their comprehensibility as measured by their number of rules and their average rule length.
Revised: j ) and their comprehensibility as measured by their number of rules and their average rule length.

Original: Furthermore, as a way to evaluate the tractability of our methods, we keep track of the resources utilised by each method in terms of both wall-clock time and memory.
Revised: Furthermore, as a way to evaluate the tractability of our methods, we keep track of the resources utilised by each method in terms of both wall-clock time and memory.

Original: Finally, given that there may be a direct beneﬁt from using the rule sets extracted from the DNN as standalone interpretable models (e.g., as a form of model distillation that generates stronger rule-based models), we include each rule set’s accuracy and AUC as part of our evaluation.
Revised: Finally, given that there may be a direct beneﬁt from using the rule sets extracted from the DNN as standalone interpretable models (e.g., as a form of model distillation that generates stronger rule-based models), we include each rule set’s accuracy and AUC as part of our evaluation.

Original: Model and Hyperparameter Selection
Revised: Model and Hyperparameter Selection

Original: Although ECLAIRE is applicable to arbitrary DNN architectures, in order for us to simplify architecture selection during evaluation, in our experiments we focus on extracting rule sets from multilayer perceptrons (MLPs).
Revised: Although ECLAIRE is applicable to arbitrary DNN architectures, in order for us to simplify architecture selection during evaluation, in our experiments we focus on extracting rule sets from multilayer perceptrons (MLPs).

Original: Moreover, in order to simulate
Revised: Moreover, in order to simulate

Original: ECLAIRE’s use case, we maintain a strict separation between ﬁne-tuning the DNN we train for a task and ﬁne-tuning the rule extraction algorithm we apply to that model.
Revised: ECLAIRE’s use case, we maintain a strict separation between ﬁne-tuning the DNN we train for a task and ﬁne-tuning the rule extraction algorithm we apply to that model.

Original: More speciﬁcally, we train several MLP architectures for each task and extract rules only from the best performing model.
Revised: More speciﬁcally, we train several MLP architectures for each task and extract rules only from the best performing model.

Original: During rule extraction, we control the growth of intermediate rule sets induced via C5.0 by varying only the minimum number of samples µ that we require C5.0 to have before splitting a node.
Revised: During rule extraction, we control the growth of intermediate rule sets induced via C5.0 by varying only the minimum number of samples µ that we require C5.0 to have before splitting a node.

Original: Because µ requires different ﬁne-tuning strategies across different tasks 2 , in this section we report only the highest performing rule set we obtain after varying µ across a task-and-method-speciﬁc spectrum.
Revised: Because µ requires different ﬁne-tuning strategies across different tasks 2 , in this section we report only the highest performing rule set we obtain after varying µ across a task-and-method-speciﬁc spectrum.

Original: Unless speciﬁed otherwise, all other C5.0’s hyperparameters, with the exception of winnowing which is always set to true, are left as their default values.
Revised: Unless speciﬁed otherwise, all other C5.0’s hyperparameters, with the exception of winnowing which is always set to true, are left as their default values.

Original: We include all details of each task’s architecture and hyperparameter selection, including the search spectrum used for µ for each task, in Appendix C.
Revised: We include all details of each task’s architecture and hyperparameter selection, including the search spectrum used for µ for each task, in Appendix C.

Original: 4.1 XOR: Evaluating Rule Set Interpretability in Controlled Setup
Revised: 4.1 XOR: Evaluating Rule Set Interpretability in Controlled Setup

Original: In this section, we use a synthetic dataset to evaluate the performance of ECLAIRE and our baselines in a controlled environment known to be challenging for vanilla rule induction algorithms and pedagogical rule extraction methods.
Revised: In this section, we use a synthetic dataset to evaluate the performance of ECLAIRE and our baselines in a controlled environment known to be challenging for vanilla rule induction algorithms and pedagogical rule extraction methods.

Original: For this, we make use of a variant of the XOR classiﬁcation task, in a form commonly used for studying feature selection methods [7, 52, 53], in which we generate a dataset { ( x ( i ) , y i ) } 1000 i =1 with 1000 10-dimensional samples such that every data point x ( i ) ∈ [0 , 1] 10 is constructed by independently sampling each dimension from a uniform distribution in [0 , 1] .
Revised: For this, we make use of a variant of the XOR classiﬁcation task, in a form commonly used for studying feature selection methods [7, 52, 53], in which we generate a dataset { ( x ( i ) , y i ) } 1000 i =1 with 1000 10-dimensional samples such that every data point x ( i ) ∈ [0 , 1] 10 is constructed by independently sampling each dimension from a uniform distribution in [0 , 1] .

Original: In this dataset, we assign a binary label y i ∈ { 0 , 1 } to each point x
Revised: In this dataset, we assign a binary label y i ∈ { 0 , 1 } to each point x

Original: ( i ) by XOR-ing the result of rounding its ﬁrst two dimensions as y i = round (cid:0) x
Revised: ( i ) by XOR-ing the result of rounding its ﬁrst two dimensions as y i = round (cid:0) x

Original: ( i ) 1 (cid:1)
Revised: ( i ) 1 (cid:1)

Original: ⊕ round (cid:0) x
Revised: ⊕ round (cid:0) x

Original: ( i ) 2 (cid:1) .
Revised: ( i ) 2 (cid:1) .

Original: In order for a classiﬁer to be able to achieve a high performance in this task, it must learn that only the ﬁrst two dimensions are meaningful for predicting the output label.
Revised: In order for a classiﬁer to be able to achieve a high performance in this task, it must learn that only the ﬁrst two dimensions are meaningful for predicting the output label.

Original: This property makes this dataset an illustrative example of a task in which decision trees are unable to obtain a high performance without a large training dataset [54, 25].
Revised: This property makes this dataset an illustrative example of a task in which decision trees are unable to obtain a high performance without a large training dataset [54, 25].

Original: In this section, we hypothesise that decompositional methods will outperform both vanilla rule induction algorithms and pedagogical rule extraction methods as their use of a DNN’s latent representations during training results in an implicit data augmentation process.
Revised: In this section, we hypothesise that decompositional methods will outperform both vanilla rule induction algorithms and pedagogical rule extraction methods as their use of a DNN’s latent representations during training results in an implicit data augmentation process.

Original: Furthermore, we hypothesise that within our decompositional baselines,
Revised: Furthermore, we hypothesise that within our decompositional baselines,

Original: ECLAIRE will outperform other methods given that its two-step process avoids the need of nesting multiple sequential approximations when merging rule sets as done in both REM-D and DeepRED * .
Revised: ECLAIRE will outperform other methods given that its two-step process avoids the need of nesting multiple sequential approximations when merging rule sets as done in both REM-D and DeepRED * .

Original: To test both hypothesises, we extract rules from an MLP with hidden layer sizes { 64 , 32 , 16 } trained on this dataset and report our ﬁndings in the XOR rows of Table 1.
Revised: To test both hypothesises, we extract rules from an MLP with hidden layer sizes { 64 , 32 , 16 } trained on this dataset and report our ﬁndings in the XOR rows of Table 1.

Original: The MLP’s test accuracy is 96 .
Revised: The MLP’s test accuracy is 96 .

Original: 6% ± 1 . 9% and its AUC is 96 .
Revised: 6% ± 1 . 9% and its AUC is 96 .

Original: 6% ± 1 . 9% .
Revised: 6% ± 1 . 9% .

Original: As hypothesised, Table 1 shows that ECLAIRE outperforms all other decompositional methods in the
Revised: As hypothesised, Table 1 shows that ECLAIRE outperforms all other decompositional methods in the

Original: XOR task in terms of both accuracy and ﬁdelity.
Revised: XOR task in terms of both accuracy and ﬁdelity.

Original: We further observe that ECLAIRE uses signiﬁcantly less resources than REM-D and DeepRED * while generating signiﬁcantly fewer rules than both.
Revised: We further observe that ECLAIRE uses signiﬁcantly less resources than REM-D and DeepRED * while generating signiﬁcantly fewer rules than both.

Original: Moreover, note that, as expected, we also observe that both C5.0 and PedC5.0 are unable to capture any meaningful rules and instead generate an empty rule set that acts as a random classiﬁer.
Revised: Moreover, note that, as expected, we also observe that both C5.0 and PedC5.0 are unable to capture any meaningful rules and instead generate an empty rule set that acts as a random classiﬁer.

Original: Finally, we observe that the standard deviation of ECLAIRE’s rule set sizes is relatively small compared to its mean.
Revised: Finally, we observe that the standard deviation of ECLAIRE’s rule set sizes is relatively small compared to its mean.

Original: This is in stark contrast with the observed standard deviations of rule set sizes in both REM-D and DeepRED * , which tend to be even greater than their means.
Revised: This is in stark contrast with the observed standard deviations of rule set sizes in both REM-D and DeepRED * , which tend to be even greater than their means.

Original: Such behaviour indicates that
Revised: Such behaviour indicates that

Original: [Table]
Revised: [Table]

Original: ECLAIRE is more robust to changes in its training data and input DNN than REM-D and DeepRED * .
Revised: ECLAIRE is more robust to changes in its training data and input DNN than REM-D and DeepRED * .

Original: We conjecture a hypothesis to explain these results in Section 5.
Revised: We conjecture a hypothesis to explain these results in Section 5.

Original: Our results also indicate that ECLAIRE is better at extracting rules that correctly capture the underlying generating process of the XOR dataset, therefore providing better global insights on what a DNN learnt from its training data.
Revised: Our results also indicate that ECLAIRE is better at extracting rules that correctly capture the underlying generating process of the XOR dataset, therefore providing better global insights on what a DNN learnt from its training data.

Original: This can be seen when analysing which features were considered important in the rules extracted by ECLAIRE.
Revised: This can be seen when analysing which features were considered important in the rules extracted by ECLAIRE.

Original: Such analysis, shown in Figure 3, indicates that rules in
Revised: Such analysis, shown in Figure 3, indicates that rules in

Original: ECLAIRE are able to identify the importance that the ﬁrst two features (i.e., x 1 and x 2 ) have on the resulting output class by very rarely conditioning on other features.
Revised: ECLAIRE are able to identify the importance that the ﬁrst two features (i.e., x 1 and x 2 ) have on the resulting output class by very rarely conditioning on other features.

Original: This is at odds with what we see in rules extracted by REM-D and
Revised: This is at odds with what we see in rules extracted by REM-D and

Original: DeepRED * , where a signiﬁcant fraction of rule premises heavily rely on features other than x 1 and x 2 .
Revised: DeepRED * , where a signiﬁcant fraction of rule premises heavily rely on features other than x 1 and x 2 .

Original: 4.2 METABRIC: Medical Applications
Revised: 4.2 METABRIC: Medical Applications

Original: In this section we evaluate ECLAIRE in real and safety-critical tasks in which interpretable models are favoured.
Revised: In this section we evaluate ECLAIRE in real and safety-critical tasks in which interpretable models are favoured.

Original: To achieve this, we evaluate our methods on two medical genomics tasks deﬁned by
Revised: To achieve this, we evaluate our methods on two medical genomics tasks deﬁned by

Original: Shams et al.
Revised: Shams et al.

Original: in [23] based on the METABRIC dataset.
Revised: in [23] based on the METABRIC dataset.

Original: More speciﬁcally, we extract rules from models trained to predict immunohistochenical subtypes (MB-ER) and histological tumour subtypes (MB-HIST) from a sequence of mRNA expression proﬁles.
Revised: More speciﬁcally, we extract rules from models trained to predict immunohistochenical subtypes (MB-ER) and histological tumour subtypes (MB-HIST) from a sequence of mRNA expression proﬁles.

Original: For further details, refer to Appendix B.
Revised: For further details, refer to Appendix B.

Original: Using the same DNN architecture and hyperparameters as in [23], we train two MLPs with hidden layers { 128 , 16 } , one trained on MB-ER and one trained on MB-HIST, and extract rules from these models.
Revised: Using the same DNN architecture and hyperparameters as in [23], we train two MLPs with hidden layers { 128 , 16 } , one trained on MB-ER and one trained on MB-HIST, and extract rules from these models.

Original: In MB-ER, our MLP achieves a test accuracy of 97 .
Revised: In MB-ER, our MLP achieves a test accuracy of 97 .

Original: 9% ± 0 .
Revised: 9% ± 0 .

Original: 3% and test AUC of 95 .
Revised: 3% and test AUC of 95 .

Original: 6% ± 1 . 1% while in MB-HIST our MLP achieves a test accuracy of 91 .
Revised: 6% ± 1 . 1% while in MB-HIST our MLP achieves a test accuracy of 91 .

Original: 2% ± 3 .
Revised: 2% ± 3 .

Original: 5% and a test
Revised: 5% and a test

Original: AUC of 88 .
Revised: AUC of 88 .

Original: 4% ± 3% .
Revised: 4% ± 3% .

Original: We summarise these results in the MB-ER and MB-HIST rows of Table 1.
Revised: We summarise these results in the MB-ER and MB-HIST rows of Table 1.

Original: As in XOR, in both of these tasks we observe that ECLAIRE is able to efﬁciently extract rule sets that are not only signiﬁcantly smaller than those produced by DeepRED * and REM-D (up to 32x fewer rules on MB-ER) but also higher-performing in their predictive accuracy.
Revised: As in XOR, in both of these tasks we observe that ECLAIRE is able to efﬁciently extract rule sets that are not only signiﬁcantly smaller than those produced by DeepRED * and REM-D (up to 32x fewer rules on MB-ER) but also higher-performing in their predictive accuracy.

Original: Only in MB-HIST we see ECLAIRE underperforming in predictive accuracy when compared to REM-D.
Revised: Only in MB-HIST we see ECLAIRE underperforming in predictive accuracy when compared to REM-D.

Original: However, given the high class imbalance in this dataset (91.3% of samples are of one class), AUC is a more meaningful metric than accuracy.
Revised: However, given the high class imbalance in this dataset (91.3% of samples are of one class), AUC is a more meaningful metric than accuracy.

Original: ECLAIRE’s superior AUC in MB-HIST indicates that it is able to extract higher-ﬁdelity rule sets than our baselines in binary tasks with a high class imbalance, a signiﬁcant advantage for several real-world tasks.
Revised: ECLAIRE’s superior AUC in MB-HIST indicates that it is able to extract higher-ﬁdelity rule sets than our baselines in binary tasks with a high class imbalance, a signiﬁcant advantage for several real-world tasks.

Original: Finally, note that although we see similar beneﬁts in ECLAIRE’s runtime as in XOR, in these tasks ECLAIRE’s parallelism overhead leads to a higher memory consumption than REM-D and DeepRED * .
Revised: Finally, note that although we see similar beneﬁts in ECLAIRE’s runtime as in XOR, in these tasks ECLAIRE’s parallelism overhead leads to a higher memory consumption than REM-D and DeepRED * .

Original: Nevertheless, although ECALIRE’s rule sets are not as small as those produced by PedC5.0 and C5.0, they have fewer than 50 rules, making them easy to inspect and debug for practitioners that may require such a model for prognosis purposes.
Revised: Nevertheless, although ECALIRE’s rule sets are not as small as those produced by PedC5.0 and C5.0, they have fewer than 50 rules, making them easy to inspect and debug for practitioners that may require such a model for prognosis purposes.

Original: 4.3 MiniBooNE and MAGIC: Scalability Study
Revised: 4.3 MiniBooNE and MAGIC: Scalability Study

Original: In this section we discuss a series of experiments that evaluate how ECLAIRE and our baselines scale when the number of samples and the depth of the architecture increase.
Revised: In this section we discuss a series of experiments that evaluate how ECLAIRE and our baselines scale when the number of samples and the depth of the architecture increase.

Original: For this, we use two particle physics binary classiﬁcation tasks that offer a large training set: MAGIC [46, 47], with 19,020 training samples, and MiniBooNE [55, 47], with 130,065 training samples.
Revised: For this, we use two particle physics binary classiﬁcation tasks that offer a large training set: MAGIC [46, 47], with 19,020 training samples, and MiniBooNE [55, 47], with 130,065 training samples.

Original: We train an MLP with hidden layers { 64 , 32 , 16 } on MAGIC and an MLP with hidden layers { 128 , 64 , 32 , 16 , 8 } on
Revised: We train an MLP with hidden layers { 64 , 32 , 16 } on MAGIC and an MLP with hidden layers { 128 , 64 , 32 , 16 , 8 } on

Original: MiniBooNE and report those results in Table 1.
Revised: MiniBooNE and report those results in Table 1.

Original: The MLP trained on MAGIC achieves a test accuracy of 86 .
Revised: The MLP trained on MAGIC achieves a test accuracy of 86 .

Original: 3% ± 1 . 8% and a test AUC of 86 .
Revised: 3% ± 1 . 8% and a test AUC of 86 .

Original: 3% ± 1 . 8% while the MLP trained on MiniBooNE achieves a test accuracy of 97.9% ± 0.3% and a test AUC of 93% ± 0.2%.
Revised: 3% ± 1 . 8% while the MLP trained on MiniBooNE achieves a test accuracy of 97.9% ± 0.3% and a test AUC of 93% ± 0.2%.

Original: In these experiments, we came across signiﬁcant difﬁculties when using REM-D and DeepRED * .
Revised: In these experiments, we came across signiﬁcant difﬁculties when using REM-D and DeepRED * .

Original: This was because most of their conﬁgurations led to runs that took more than 6 hours in total, either due to intractable rule extraction or intractable inference caused by large extracted rule sets.
Revised: This was because most of their conﬁgurations led to runs that took more than 6 hours in total, either due to intractable rule extraction or intractable inference caused by large extracted rule sets.

Original: Thus, we increased the value of µ to be a signiﬁcant fraction of N , the number of points in the dataset, and we show the best value we obtain by varying µ as indicated in Appendix C.3.4.
Revised: Thus, we increased the value of µ to be a signiﬁcant fraction of N , the number of points in the dataset, and we show the best value we obtain by varying µ as indicated in Appendix C.3.4.

Original: Nevertheless, notice that although such amounts of pruning lead to smaller rule sets than ECLAIRE’s in MiniBooNE, these are overpruned rule sets which underperform in the test set.
Revised: Nevertheless, notice that although such amounts of pruning lead to smaller rule sets than ECLAIRE’s in MiniBooNE, these are overpruned rule sets which underperform in the test set.

Original: Our experiments show that ECLAIRE can scale to both large architectures and large datasets without the need for excessively large values of µ (i.e., no need for excessive intermediate rule set pruning).
Revised: Our experiments show that ECLAIRE can scale to both large architectures and large datasets without the need for excessively large values of µ (i.e., no need for excessive intermediate rule set pruning).

Original: Furthermore, we can see that ECLAIRE outperforms all of the baselines in both tasks in terms of its AUC and ﬁdelity.
Revised: Furthermore, we can see that ECLAIRE outperforms all of the baselines in both tasks in terms of its AUC and ﬁdelity.

Original: Only in its accuracy it is marginally outperformed by C5.0 and PedC5.0 in the
Revised: Only in its accuracy it is marginally outperformed by C5.0 and PedC5.0 in the

Original: MiniBooNE task.
Revised: MiniBooNE task.

Original: Nevertheless, given the class imbalance in this task, the beneﬁt from ECLAIRE’s
Revised: Nevertheless, given the class imbalance in this task, the beneﬁt from ECLAIRE’s

Original: AUC improvements out-weights its marginally worse accuracy.
Revised: AUC improvements out-weights its marginally worse accuracy.

Original: 4.4 Letter Recognition: Multiclass Classiﬁcation
Revised: 4.4 Letter Recognition: Multiclass Classiﬁcation

Original: Although both REM-D and DeepRED were developed to be applicable to multiclass tasks, they have never been formally evaluated in such tasks.
Revised: Although both REM-D and DeepRED were developed to be applicable to multiclass tasks, they have never been formally evaluated in such tasks.

Original: In this section we ﬁll this gap by extracting rules from a model trained on the Letter Recognition Dataset [49, 47], a classiﬁcation task with 26 target labels.
Revised: In this section we ﬁll this gap by extracting rules from a model trained on the Letter Recognition Dataset [49, 47], a classiﬁcation task with 26 target labels.

Original: We proceed by extracting rules from a trained MLP with hidden layers { 128 , 64 } which achieves a test accuracy of 95% ± 0 .
Revised: We proceed by extracting rules from a trained MLP with hidden layers { 128 , 64 } which achieves a test accuracy of 95% ± 0 .

Original: We summarise our results in Table 1 and include further experiments using an alternative tree induction algorithm in Appendix C.3.5.
Revised: We summarise our results in Table 1 and include further experiments using an alternative tree induction algorithm in Appendix C.3.5.

Original: As in other data-intensive tasks, we ﬁrst observe a difﬁcultly in tractably extracting rules using REM-D and DeepRED * without large values of µ .
Revised: As in other data-intensive tasks, we ﬁrst observe a difﬁcultly in tractably extracting rules using REM-D and DeepRED * without large values of µ .

Original: Even when we ﬁnd a conﬁguration for these methods that terminates in the allotted time, we are unable to observe results that are signiﬁcantly better than random.
Revised: Even when we ﬁnd a conﬁguration for these methods that terminates in the allotted time, we are unable to observe results that are signiﬁcantly better than random.

Original: Second, although ECLAIRE is able to outperform both DeepRED * and REM-D, it struggles to outperform PedC5.0 and C5.0.
Revised: Second, although ECLAIRE is able to outperform both DeepRED * and REM-D, it struggles to outperform PedC5.0 and C5.0.

Original: We believe this to be the case due to possible class imbalance in intermediate rule sets: when performing a clause-wise substitution in intermediate rules, and as the number of classes increases, the number of samples that satisfy the premise of an intermediate rule becomes smaller.
Revised: We believe this to be the case due to possible class imbalance in intermediate rule sets: when performing a clause-wise substitution in intermediate rules, and as the number of classes increases, the number of samples that satisfy the premise of an intermediate rule becomes smaller.

Original: This implies that when using C5.0 to substitute an intermediate clause, it has to face a highly imbalanced binary classiﬁcation task, something known to be problematic for decision trees [56].
Revised: This implies that when using C5.0 to substitute an intermediate clause, it has to face a highly imbalanced binary classiﬁcation task, something known to be problematic for decision trees [56].

Original: While we are able to slightly alleviate this in our results through C5.0’s support for independent class weights, it remains as future work to explore better mechanisms for handling this imbalance.
Revised: While we are able to slightly alleviate this in our results through C5.0’s support for independent class weights, it remains as future work to explore better mechanisms for handling this imbalance.

Original: An interesting observation is that when we include the input features as one of the representations which ECLAIRE can use to extract intermediate rule sets, it results in a rule set that outperforms
Revised: An interesting observation is that when we include the input features as one of the representations which ECLAIRE can use to extract intermediate rule sets, it results in a rule set that outperforms

Original: C5.0 and PedC5.0.
Revised: C5.0 and PedC5.0.

Original: This result, shown at the bottom of Table 1 as ECLAIRE * , suggests that using a
Revised: This result, shown at the bottom of Table 1 as ECLAIRE * , suggests that using a

Original: DNN’s input features as one of the representations that ECLAIRE has access to can be beneﬁcial for the performance of its end rule set.
Revised: DNN’s input features as one of the representations that ECLAIRE has access to can be beneﬁcial for the performance of its end rule set.

Original: This can be thought of as an eclectic variation of ECLAIRE and we leave it as future research to explore whether this improvement generalises across domains.
Revised: This can be thought of as an eclectic variation of ECLAIRE and we leave it as future research to explore whether this improvement generalises across domains.

Original:  Discussion
Revised:  Discussion

Original: Variance Reduction in ECLAIRE In [23], Shams et al.
Revised: Variance Reduction in ECLAIRE In [23], Shams et al.

Original: observe a very high variance in the size of rule sets extracted with REM-D and postulate that this is due to different neural network initialisations leading to latent spaces that vary in interpretability properties.
Revised: observe a very high variance in the size of rule sets extracted with REM-D and postulate that this is due to different neural network initialisations leading to latent spaces that vary in interpretability properties.

Original: Nevertheless, our results in Table 1 suggest that this may not be the case, as ECLAIRE’s variance is signiﬁcantly smaller than that observed in both REM-D and DeepRED * .
Revised: Nevertheless, our results in Table 1 suggest that this may not be the case, as ECLAIRE’s variance is signiﬁcantly smaller than that observed in both REM-D and DeepRED * .

Original: Inspired by results on convergence learning [57, 58], in this paper we challenge this hypothesis by arguing that the variance observed in REM-D, and therefore in DeepRED, is not the result of different initialisations leading to different levels of interpretability in a DNN’s latent space, but it is rather the result of a crucial design factor in REM-D:
Revised: Inspired by results on convergence learning [57, 58], in this paper we challenge this hypothesis by arguing that the variance observed in REM-D, and therefore in DeepRED, is not the result of different initialisations leading to different levels of interpretability in a DNN’s latent space, but it is rather the result of a crucial design factor in REM-D:

Original: REM-D uses rule sets induced from decision trees to substitute terms in its intermediate rule sets, forcing the need of a cartesian-product redistribution of terms afterwards.
Revised: REM-D uses rule sets induced from decision trees to substitute terms in its intermediate rule sets, forcing the need of a cartesian-product redistribution of terms afterwards.

Original: Decision trees, being highvariance classiﬁers [59], are highly sensitive to their training sets.
Revised: Decision trees, being highvariance classiﬁers [59], are highly sensitive to their training sets.

Original: This implies that when REM-D substitutes the same term in an intermediate rule using two different datasets (as it would happen across two different training folds), it can generate vastly different rule sets.
Revised: This implies that when REM-D substitutes the same term in an intermediate rule using two different datasets (as it would happen across two different training folds), it can generate vastly different rule sets.

Original: Hence, because during its term-wise substitution REM-D replaces each term in an intermediate rule’s premise with a set of rules extracted from a decision tree, the variance in the number of rules after one substitution grows in a multiplicative fashion.
Revised: Hence, because during its term-wise substitution REM-D replaces each term in an intermediate rule’s premise with a set of rules extracted from a decision tree, the variance in the number of rules after one substitution grows in a multiplicative fashion.

Original: This is because all newly substituted terms in a clause need to be redistributed in a cartesian-product (a visual representation of this process can be found in Appendix D).
Revised: This is because all newly substituted terms in a clause need to be redistributed in a cartesian-product (a visual representation of this process can be found in Appendix D).

Original: Therefore, we argue that the signiﬁcant reduction in variance we observe in our evaluation of ECLAIRE is due to the fact that ECLAIRE performs a clause-wise substitution rather than a term-wise substitution, hence avoiding a multiplicative growth in the number of rules.
Revised: Therefore, we argue that the signiﬁcant reduction in variance we observe in our evaluation of ECLAIRE is due to the fact that ECLAIRE performs a clause-wise substitution rather than a term-wise substitution, hence avoiding a multiplicative growth in the number of rules.

Original: This hypothesis explains the results in
Revised: This hypothesis explains the results in

Original: Table 1 and highlights a signiﬁcant advantage of ECLAIRE over other decompositional methods.
Revised: Table 1 and highlights a signiﬁcant advantage of ECLAIRE over other decompositional methods.

Original: Growth Coping Mechanisms Despite ECLAIRE’s superior performance compared to our baselines,
Revised: Growth Coping Mechanisms Despite ECLAIRE’s superior performance compared to our baselines,

Original: Table 1 shows that ECLAIRE can still take a considerable amount of time to extract high-performing rule sets in large datasets and architectures.
Revised: Table 1 shows that ECLAIRE can still take a considerable amount of time to extract high-performing rule sets in large datasets and architectures.

Original: Furthermore, we observe a growth in the number of rules extracted by ECLAIRE as the amount of data and the model size increase.
Revised: Furthermore, we observe a growth in the number of rules extracted by ECLAIRE as the amount of data and the model size increase.

Original: As a way to alleviate this growth, we explore four different coping mechanisms: (1) intermediate rule pruning , where, as in [23], we increase the value of µ to generate smaller intermediate rule sets, (2) hidden representation subsampling , where we generate intermediate rule sets only from a subset of the DNN’s intermediate representations, (3) training set subsampling , where, as in [25], only a fraction of the DNN’s training set is used for rule extraction, and (4) intermediate rule ranking , where the lowest p % of rules in intermediate rule sets, as ranked by their conﬁdence levels, are dropped.
Revised: As a way to alleviate this growth, we explore four different coping mechanisms: (1) intermediate rule pruning , where, as in [23], we increase the value of µ to generate smaller intermediate rule sets, (2) hidden representation subsampling , where we generate intermediate rule sets only from a subset of the DNN’s intermediate representations, (3) training set subsampling , where, as in [25], only a fraction of the DNN’s training set is used for rule extraction, and (4) intermediate rule ranking , where the lowest p % of rules in intermediate rule sets, as ranked by their conﬁdence levels, are dropped.

Original: We summarise these experiments in Appendix E.
Revised: We summarise these experiments in Appendix E.

Original: Our results indicate that the performance of ECLAIRE’s rule sets is robust to heavy training data subsampling as well as signiﬁcant intermediate representation subsampling and intermediate rule pruning via conﬁdence ranking.
Revised: Our results indicate that the performance of ECLAIRE’s rule sets is robust to heavy training data subsampling as well as signiﬁcant intermediate representation subsampling and intermediate rule pruning via conﬁdence ranking.

Original: Speciﬁcally, we observe that ECLAIRE is very data efﬁcient : one can subsample up to 50% of the training set during rule extraction, causing a ∼ 50% drop in extraction time and rule set size, without observing an absolute loss of more than 0 .
Revised: Speciﬁcally, we observe that ECLAIRE is very data efﬁcient : one can subsample up to 50% of the training set during rule extraction, causing a ∼ 50% drop in extraction time and rule set size, without observing an absolute loss of more than 0 .

Original: 5% in the resulting rule set ﬁdelity.
Revised: 5% in the resulting rule set ﬁdelity.

Original: Similar results are observed when one subsamples every other two hidden representations when generating intermediate rule sets or drops the lowest 25% of rules in intermediate rule sets.
Revised: Similar results are observed when one subsamples every other two hidden representations when generating intermediate rule sets or drops the lowest 25% of rules in intermediate rule sets.

Original: Nevertheless, our experiments also highlight a clear limitation in ECLAIRE: for achieving optimal performance, ECLAIRE requires signiﬁcant task-speciﬁc ﬁne tuning of its µ hyperparameter.
Revised: Nevertheless, our experiments also highlight a clear limitation in ECLAIRE: for achieving optimal performance, ECLAIRE requires signiﬁcant task-speciﬁc ﬁne tuning of its µ hyperparameter.

Original: While in practice we observe that selecting µ from [10 − 5 N, 10 − 4 N ] yields good results, we leave the exploration of more systematic strategies for selecting µ to future work.
Revised: While in practice we observe that selecting µ from [10 − 5 N, 10 − 4 N ] yields good results, we leave the exploration of more systematic strategies for selecting µ to future work.

Original:  Conclusion
Revised:  Conclusion

Original: With the ubiquitous use of DNNs, it has become increasingly important to be able to explain, as well as debug, the decisions made by production-sized DNNs.
Revised: With the ubiquitous use of DNNs, it has become increasingly important to be able to explain, as well as debug, the decisions made by production-sized DNNs.

Original: In this work we aim to ﬁll this gap by proposing ECLAIRE, a novel decompositional rule extraction algorithm, which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional algorithm able to scale to both large training sets and deep architectures.
Revised: In this work we aim to ﬁll this gap by proposing ECLAIRE, a novel decompositional rule extraction algorithm, which, to the best of our knowledge, is the ﬁrst polynomial-time decompositional algorithm able to scale to both large training sets and deep architectures.

Original: In contrast to previous work which focuses on using intermediate representations in a sequential manner, ECLAIRE exploits these representations in parallel to build an ensemble of classiﬁers that can then be efﬁciently combined into a single rule set.
Revised: In contrast to previous work which focuses on using intermediate representations in a sequential manner, ECLAIRE exploits these representations in parallel to build an ensemble of classiﬁers that can then be efﬁciently combined into a single rule set.

Original: We evaluate
Revised: We evaluate

Original: ECLAIRE on six different tasks and show that it consistently outperforms the previous state-of-the-art decompositional methods and vanilla rule induction methods in terms of the accuracy and ﬁdelity of its output rule set.
Revised: ECLAIRE on six different tasks and show that it consistently outperforms the previous state-of-the-art decompositional methods and vanilla rule induction methods in terms of the accuracy and ﬁdelity of its output rule set.

Original: At the same time, ECLAIRE uses orders of magnitude less resources and produces fewer rules than current state-of-the-art decompositional methods.
Revised: At the same time, ECLAIRE uses orders of magnitude less resources and produces fewer rules than current state-of-the-art decompositional methods.

Original: Excited with the prospects that can arise from this work, and to encourage the use of our algorithms in both industry and research, we make all of our methods, together with an extensive set of visualisation and inspection tools, publicly available as part of the REMIX library.
Revised: Excited with the prospects that can arise from this work, and to encourage the use of our algorithms in both industry and research, we make all of our methods, together with an extensive set of visualisation and inspection tools, publicly available as part of the REMIX library.

Original: Follow up research could explore generalising ECLAIRE to regression tasks, as well as to RNN architectures and could include work on alleviating the effects of class imbalance in multiclass settings
Revised: Follow up research could explore generalising ECLAIRE to regression tasks, as well as to RNN architectures and could include work on alleviating the effects of class imbalance in multiclass settings

Original: Acknowledgements
Revised: Acknowledgements

Original: MEZ acknowledges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship
Revised: MEZ acknowledges support from the Gates Cambridge Trust via the Gates Cambridge Scholarship

Original: A ECLAIRE’s Asymptotic Bounds A.1 Notation
Revised: A ECLAIRE’s Asymptotic Bounds A.1 Notation

Original: Throughout this section we let | R | be the number of rules in a rule set R
Revised: Throughout this section we let | R | be the number of rules in a rule set R

Original: T ( R ) be the set of all unique terms in rules in R and T max ( R ) be the set of terms in the longest rule in R .
Revised: T ( R ) be the set of all unique terms in rules in R and T max ( R ) be the set of terms in the longest rule in R .

Original: A.2 Lemmas
Revised: A.2 Lemmas

Original: Before showing Theorem we introduce two useful lemmas related to rule induction:
Revised: Before showing Theorem we introduce two useful lemmas related to rule induction:

Original: Lemma 1.
Revised: Lemma 1.

Original: If R is a rule set induced from a decision tree trained with N samples using a recursive top-down impurity-based algorithm (e g., C5.0), then the following must hold:
Revised: If R is a rule set induced from a decision tree trained with N samples using a recursive top-down impurity-based algorithm (e g., C5.0), then the following must hold:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, notice that a tree generated by any top-down tree induction algorithm cannot have more than N leaf nodes.
Revised: First, notice that a tree generated by any top-down tree induction algorithm cannot have more than N leaf nodes.

Original: This is because, in the worst-case scenario, any top-down algorithm may reach its base case N times with each leaf having one training sample in it.
Revised: This is because, in the worst-case scenario, any top-down algorithm may reach its base case N times with each leaf having one training sample in it.

Original: Therefore, this implies that a tree cannot have more than ( N − 1) split nodes in it.
Revised: Therefore, this implies that a tree cannot have more than ( N − 1) split nodes in it.

Original: If this would not be true, then it is trivial to see that the binary tree would have to have at least N + 1 leaf nodes, a contradiction to our previous observation.
Revised: If this would not be true, then it is trivial to see that the binary tree would have to have at least N + 1 leaf nodes, a contradiction to our previous observation.

Original: Furthermore, notice that during rule induction each split node can contribute with at most two unique terms to the rule set (one per branch) and with at most one term to a single rule (one per root-to-leaf path).
Revised: Furthermore, notice that during rule induction each split node can contribute with at most two unique terms to the rule set (one per branch) and with at most one term to a single rule (one per root-to-leaf path).

Original: Because of this, a single rule cannot have more than ( N − 1) unique terms in it and the rule set cannot have more than 2( N − 1) unique total terms in it.
Revised: Because of this, a single rule cannot have more than ( N − 1) unique terms in it and the rule set cannot have more than 2( N − 1) unique total terms in it.

Original: Finally, note that these bounds can be tight.
Revised: Finally, note that these bounds can be tight.

Original: This can be seen in the decision tree shown in Figure 4 where each split node must be unique as no gains can be obtained from partitioning the training set twice in the same way in any given path.
Revised: This can be seen in the decision tree shown in Figure 4 where each split node must be unique as no gains can be obtained from partitioning the training set twice in the same way in any given path.

Original: This lemma can be extended to provide an asymptotic bound on the number of rules induced from a decision tree:
Revised: This lemma can be extended to provide an asymptotic bound on the number of rules induced from a decision tree:

Original: Lemma 2.
Revised: Lemma 2.

Original: If R is a rule set induced from an impurity-based top-down decision tree trained on N data samples, then it must be the case that | R | =
Revised: If R is a rule set induced from an impurity-based top-down decision tree trained on N data samples, then it must be the case that | R | =

Original: O ( N ) .
Revised: O ( N ) .

Original: Proof.
Revised: Proof.

Original: An immediate result of our proof of lemma 1 is that the number of leaf nodes in a decision tree grows as O ( N ) .
Revised: An immediate result of our proof of lemma 1 is that the number of leaf nodes in a decision tree grows as O ( N ) .

Original: Therefore, because after rule induction there is a one-to-one correspondence between leaves in the decision tree and rules in the output rule set, the number of rules induced from a decision tree must also grow as O ( N ) .
Revised: Therefore, because after rule induction there is a one-to-one correspondence between leaves in the decision tree and rules in the output rule set, the number of rules induced from a decision tree must also grow as O ( N ) .

Original: Having built an understanding of how rule induction is affected by its training data, we now use these results to analyse ECLAIRE’s asymptotic behaviour.
Revised: Having built an understanding of how rule induction is affected by its training data, we now use these results to analyse ECLAIRE’s asymptotic behaviour.

Original: Theorem 1 (ECLAIRE Runtime Complexity) .
Revised: Theorem 1 (ECLAIRE Runtime Complexity) .

Original: Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).
Revised: Assume ECLAIRE’s intermediate rule extraction algorithm ψ ( · ) operates by inducing rules from a decision tree that was learnt using a top-down impuritybased algorithm (e.g., C5.0).

Original: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .
Revised: Furthermore, assume that, when trained with N m -dimensional samples, ψ ( · ) ’s runtime grows as O ( N p n m p m ) , for some p n , p m ∈ N .

Original: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max
Revised: Given a training set with N samples and a neural network with d hidden layers, such that there are at most m activations in any of its layers, ECLAIRE’s runtime will grow as a function of O (cid:16) dN max

Original: [Equation]
Revised: [Equation]

Original: m p m (cid:17) .
Revised: m p m (cid:17) .

Original: Proof.
Revised: Proof.

Original: We begin our proof with a high-level summary of ECLAIRE’s approach: for each layer i in our network, ECLAIRE extracts an intermediate rule set R h
Revised: We begin our proof with a high-level summary of ECLAIRE’s approach: for each layer i in our network, ECLAIRE extracts an intermediate rule set R h

Original: i (cid:55)→ ˆ y using rule extractor ψ ( · ) on intermediate activations h
Revised: i (cid:55)→ ˆ y using rule extractor ψ ( · ) on intermediate activations h

Original: i and the labels predicted by the input DNN.
Revised: i and the labels predicted by the input DNN.

Original: It then iterates over all premises in R h
Revised: It then iterates over all premises in R h

Original: i (cid:55)→ ˆ y , extracts a rule set mapping input activations x to each premise’s truth value, and adds the relevant rules into the output set.
Revised: i (cid:55)→ ˆ y , extracts a rule set mapping input activations x to each premise’s truth value, and adds the relevant rules into the output set.

Original: This procedure implies that the total amount of work done by ECLAIRE to process hidden layer i grows asymptotically as
Revised: This procedure implies that the total amount of work done by ECLAIRE to process hidden layer i grows asymptotically as

Original: O (cid:32) ( complexity of ψ ( · ) when trained with N samples and m features )
Revised: O (cid:32) ( complexity of ψ ( · ) when trained with N samples and m features )

Original: + ( number of rules in R h
Revised: + ( number of rules in R h

Original: i (cid:55)→ ˆ y ) × ( complexity of evaluating a rule in R h
Revised: i (cid:55)→ ˆ y ) × ( complexity of evaluating a rule in R h

Original: i (cid:55)→ ˆ y N times )
Revised: i (cid:55)→ ˆ y N times )

Original: + ( number of rules in R h
Revised: + ( number of rules in R h

Original: i (cid:55)→ ˆ y ) × ( complexity of ψ ( · ) when trained with N samples and m features )
Revised: i (cid:55)→ ˆ y ) × ( complexity of ψ ( · ) when trained with N samples and m features )

Original: + ( number of rules in R h
Revised: + ( number of rules in R h

Original: i (cid:55)→
Revised: i (cid:55)→

Original: [Equation]
Revised: [Equation]

Original: ( complexity of post-processing and adding new rules )
Revised: ( complexity of post-processing and adding new rules )

Original: [Equation]
Revised: [Equation]

Original: Which can be rewritten as:
Revised: Which can be rewritten as:

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne I ( t ) x
Revised: where we deﬁne I ( t ) x

Original: (cid:55)→ p to be the temporary rule set (approximating premise p ) with the longest rule and we deﬁne I ( r )
Revised: (cid:55)→ p to be the temporary rule set (approximating premise p ) with the longest rule and we deﬁne I ( r )

Original: x (cid:55)→ p (cid:48)
Revised: x (cid:55)→ p (cid:48)

Original: as the temporary extracted rule set with the largest number of rules in it.
Revised: as the temporary extracted rule set with the largest number of rules in it.

Original: Note that we used the fact that, by assumption, ψ ( · ) ’s worst case runtime grows as O ( N p n m p m ) .
Revised: Note that we used the fact that, by assumption, ψ ( · ) ’s worst case runtime grows as O ( N p n m p m ) .

Original: Applying lemmas 1 and 2 to this expression gives us
Revised: Applying lemmas 1 and 2 to this expression gives us

Original: [Equation]
Revised: [Equation]

Original: Finally, using the fact that ( N − 1) =
Revised: Finally, using the fact that ( N − 1) =

Original: O ( N ) and ( N 3 ) =
Revised: O ( N ) and ( N 3 ) =

Original: O ( N max (3 ,p n +1) m p m ) as both p n and p m are natural numbers, and recalling that ECLAIRE performs this much work for each hidden layer, we get that ECLAIRE’s total runtime grows as a function of
Revised: O ( N max (3 ,p n +1) m p m ) as both p n and p m are natural numbers, and recalling that ECLAIRE performs this much work for each hidden layer, we get that ECLAIRE’s total runtime grows as a function of

Original: [Equation]
Revised: [Equation]

Original: B Dataset Details
Revised: B Dataset Details

Original: In this section we include a brief description of all the non-synthetic datasets used for the tasks described in Section 4.
Revised: In this section we include a brief description of all the non-synthetic datasets used for the tasks described in Section 4.

Original: METABRIC [45] This dataset consists of a collection of anonymized features extracted from breast cancer tumours in a cohort of 1 , 980 patients.
Revised: METABRIC [45] This dataset consists of a collection of anonymized features extracted from breast cancer tumours in a cohort of 1 , 980 patients.

Original: It includes clinical traits, gene expression patterns, tumour characteristics, and survival rates for a period of 4 years.
Revised: It includes clinical traits, gene expression patterns, tumour characteristics, and survival rates for a period of 4 years.

Original: The speciﬁc tasks we consider in
Revised: The speciﬁc tasks we consider in

Original: METABRIC are:
Revised: METABRIC, taken from [23], are:

Original: • Immunohistochemical subtype prediction (MB-ER): for this task we predict immunohistochemical subtypes in breast cancer patients using 1000 mRNA expression patterns.
Revised: • Immunohistochemical subtype prediction (MB-ER): for this task we predict immunohistochemical subtypes in breast cancer patients using 1000 mRNA expression patterns.

Original: Each sample can be classiﬁed as one of two types, ER+ or ER-, which are crucial for determining a patient’s treatment.
Revised: Each sample can be classiﬁed as one of two types, ER+ or ER-, which are crucial for determining a patient’s treatment.

Original: • Histological tumour subtype prediction (MB-HIST): for this task we predict histological subtypes of breast cancer tumours using 1004 mRNA expression proﬁles.
Revised: • Histological tumour subtype prediction (MB-HIST): for this task we predict histological subtypes of breast cancer tumours using 1004 mRNA expression proﬁles.

Original: Each sample can be classiﬁed as either Invasive Lobular Carcinoma (ILC) or Invasive Ductal Carcinoma (IDC), two most common breast cancer histological subtypes [23].
Revised: Each sample can be classiﬁed as either Invasive Lobular Carcinoma (ILC) or Invasive Ductal Carcinoma (IDC), two most common breast cancer histological subtypes [23].

Original: MAGIC [46, 47]
Revised: MAGIC [46, 47]

Original: This is a particle physics dataset in which a signal needs to be classiﬁed as being a high-energy gamma ray or some background hadron cosmic radiation.
Revised: This is a particle physics dataset in which a signal needs to be classiﬁed as being a high-energy gamma ray or some background hadron cosmic radiation.

Original: Each of the 19,020 training samples consists of 10 real-valued features that are generated via a Monte Carlo program and a binary label indicating whether the observation corresponds as a high-energy gamma ray (signal) or some background hadron radiation.
Revised: Each of the 19,020 training samples consists of 10 real-valued features that are generated via a Monte Carlo program and a binary label indicating whether the observation corresponds as a high-energy gamma ray (signal) or some background hadron radiation.

Original: MiniBooNE [55, 47]
Revised: MiniBooNE [55, 47]

Original: This is a particle physics dataset in which one is interested in discriminating electron neutrino events from background events in interactions collected in the MiniBooNE experiment.
Revised: This is a particle physics dataset in which one is interested in discriminating electron neutrino events from background events in interactions collected in the MiniBooNE experiment.

Original: Each of the 130,065 training samples consists of 50 real-valued features empirically collected in the MiniBooNE experiment and a binary label indicating whether the sample represents a electron neutrino event (signal) or a background event.
Revised: Each of the 130,065 training samples consists of 50 real-valued features empirically collected in the MiniBooNE experiment and a binary label indicating whether the sample represents a electron neutrino event (signal) or a background event.

Original: Letter Recognition [49, 47]
Revised: Letter Recognition [49, 47]

Original: This dataset consists of 20 , 000 representations of black-and-white
Revised: This dataset consists of 20 , 000 representations of black-and-white

Original: English capital letters labelled with one of 26 classes (A to Z).
Revised: English capital letters labelled with one of 26 classes (A to Z).

Original: Each sample is generated by extracting 16 statistical features from the images of each letter.
Revised: Each sample is generated by extracting 16 statistical features from the images of each letter.

Original: A summary of the properties of all tasks spanning from these datasets can be found in Table [Table]
Revised: A summary of the properties of all tasks spanning from these datasets can be found in Table 2.

Original: [Table]
Revised: [Table]

Original: C Model & Hyperparameter Selection C.1 Model Selection Details
Revised: C Model & Hyperparameter Selection C.1 Model Selection Details

Original: For each of our tasks, we select a DNN architecture by iterating over several possible architectures and selecting the one with the highest testing performance for rule extraction.
Revised: For each of our tasks, we select a DNN architecture by iterating over several possible architectures and selecting the one with the highest testing performance for rule extraction.

Original: For the sake of simplicity, this search process consists of ﬁrst ﬁxing the number of hidden layers in the network for each task, and then performing a grid-search over different hyperparameters of MLPs with that many hidden layers.
Revised: For the sake of simplicity, this search process consists of ﬁrst ﬁxing the number of hidden layers in the network for each task, and then performing a grid-search over different hyperparameters of MLPs with that many hidden layers.

Original: In our search, we allow intermediate activations sizes to be chosen from { 8 , 16 , 32 , 64 , 128 } and activation functions to be chosen from { tanh , relu , elu } .
Revised: In our search, we allow intermediate activations sizes to be chosen from { 8 , 16 , 32 , 64 , 128 } and activation functions to be chosen from { tanh , relu , elu } .

Original: Furthermore, to constrain the search space further, we force the number of activations in hidden layer ( i + i ) to be less than the number of activations in layer i .
Revised: Furthermore, to constrain the search space further, we force the number of activations in hidden layer ( i + i ) to be less than the number of activations in layer i .

Original: This goes in line with several traditional encoding architectures [26].
Revised: This goes in line with several traditional encoding architectures [26].

Original: Unless speciﬁed otherwise, we train each model to minimise its weighted classiﬁcation cross-entropy loss for epochs ∈ { 50 , 100 , 150 , 200 } using stochastic gradient descent [60] (batch size in { 16 , 32 } ) and use an Adam optimiser [61] with its default parameters ( β 1 = 0 .
Revised: Unless speciﬁed otherwise, we train each model to minimise its weighted classiﬁcation cross-entropy loss for epochs ∈ { 50 , 100 , 150 , 200 } using stochastic gradient descent [60] (batch size in { 16 , 32 } ) and use an Adam optimiser [61] with its default parameters ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 7 , and lr = 10 − 3 ) for computing weight updates.
Revised: 10 − 7 , and lr = 10 − 3 ) for computing weight updates.

Original: To speed this process up, we made use of GPU clusters offered by Google’s colab services.
Revised: To speed this process up, we made use of GPU clusters offered by Google’s colab services.

Original: C.2 Rule Extraction Fine Tuning
Revised: C.2 Rule Extraction Fine Tuning

Original: For all of the rule extraction methods we experiment with in our evaluation, we attempt several values for the minimum number of samples per split (i.e., µ ) and report only on the rule set that performed the best on the testing dataset.
Revised: For all of the rule extraction methods we experiment with in our evaluation, we attempt several values for the minimum number of samples per split (i.e., µ ) and report only on the rule set that performed the best on the testing dataset.

Original: Because this search is very task-speciﬁc (due to different dataset sizes and/or architectures), in all of our tasks we deﬁne three evaluation parameters for each rule extraction algorithm, namely µ min , µ max , and δµ , and evaluate the performance of rules extracted via that method using values of µ ∈ { µ min , ( µ min + δµ ) , ( µ min + 2 δµ ) , · · · , µ max } .
Revised: Because this search is very task-speciﬁc (due to different dataset sizes and/or architectures), in all of our tasks we deﬁne three evaluation parameters for each rule extraction algorithm, namely µ min , µ max , and δµ , and evaluate the performance of rules extracted via that method using values of µ ∈ { µ min , ( µ min + δµ ) , ( µ min + 2 δµ ) , · · · , µ max } .

Original: The values deﬁning the search space for each algorithm in our different tasks are described in the following section.
Revised: The values deﬁning the search space for each algorithm in our different tasks are described in the following section.

Original: C.3 Task-speciﬁc Conﬁgurations
Revised: C.3 Task-speciﬁc Conﬁgurations

Original: In this section we describe the conﬁguration that our grid search produces for each of the tasks we discuss in Section 4.
Revised: In this section we describe the conﬁguration that our grid search produces for each of the tasks we discuss in Section 4.

Original: We also include a description of the search space used for µ when evaluating different rule extraction methods.
Revised: We also include a description of the search space used for µ when evaluating different rule extraction methods.

Original: C.3.1 XOR
Revised: C.3.1 XOR

Original: Given the simplicity of the XOR task, we constrain the architecture search space to be only over
Revised: Given the simplicity of the XOR task, we constrain the architecture search space to be only over

Original: MLPs using 3 hidden layers.
Revised: MLPs using 3 hidden layers.

Original: This results in the best performing model having hidden layers with sizes {64, 32, 16} and tanh activation between them.
Revised: This results in the best performing model having hidden layers with sizes {64, 32, 16} and tanh activation between them.

Original: This model is then trained for 150 epochs using a batch size of 16.
Revised: This model is then trained for 150 epochs using a batch size of 16.

Original: When ﬁne-tuning the different rule extractors, we use µ min = 2 , µ max = 15 , and δµ = 1 for
Revised: When ﬁne-tuning the different rule extractors, we use µ min = 2 , µ max = 15 , and δµ = 1 for

Original: ECLAIRE, C5.0, and PedC5.0 as they all terminate relatively quickly and obtain good performance without much pruning.
Revised: ECLAIRE, C5.0, and PedC5.0 as they all terminate relatively quickly and obtain good performance without much pruning.

Original: REM-D and DeepRED * , however, fail to terminate before their allotted times when using values of µ below 25 .
Revised: REM-D and DeepRED * , however, fail to terminate before their allotted times when using values of µ below 25 .

Original: Because of this, we evaluate them using values of µ in the range deﬁned by µ min = 25 , µ max = 35 , and δµ = 1 .
Revised: Because of this, we evaluate them using values of µ in the range deﬁned by µ min = 25 , µ max = 35 , and δµ = 1 .

Original: C.3.2 METABRIC
Revised: C.3.2 METABRIC

Original: For both METABRIC tasks described in Section 4.2, we use the exact same architecture and training process used by Shams et al.
Revised: For both METABRIC tasks described in Section 4.2, we use the exact same architecture and training process used by Shams et al.

Original: in [23], which was determined through a similar grid search process.
Revised: in [23], which was determined through a similar grid search process.

Original: This architecture consists of an MLP with hidden layers with sizes { 128 , 16 } and tanh activations in between them.
Revised: This architecture consists of an MLP with hidden layers with sizes { 128 , 16 } and tanh activations in between them.

Original: For both tasks we train the model for 150 epochs using a batch size of 16.
Revised: For both tasks we train the model for 150 epochs using a batch size of 16.

Original: In our rule extraction ﬁne-tuning, we use µ min = 2 , µ max = 15 , and δµ = 1 for C5.0, PedC5.0, and
Revised: In our rule extraction ﬁne-tuning, we use µ min = 2 , µ max = 15 , and δµ = 1 for C5.0, PedC5.0, and

Original: ECLAIRE.
Revised: ECLAIRE.

Original: As suggested by [23], and due to their longer run times, we use µ min = 5 , µ max = 15 , and δµ = 5 for REM-D and DeepRED * .
Revised: As suggested by [23], and due to their longer run times, we use µ min = 5 , µ max = 15 , and δµ = 5 for REM-D and DeepRED * .

Original: C.3.3 MAGIC
Revised: C.3.3 MAGIC

Original: In the MAGIC dataset results reported in Section 4.3, we search over architectures with 3 hidden layers and our grid-search results in the best model having layers of size {64, 32, 16} with ReLU activations in between them.
Revised: In the MAGIC dataset results reported in Section 4.3, we search over architectures with 3 hidden layers and our grid-search results in the best model having layers of size {64, 32, 16} with ReLU activations in between them.

Original: The best training conﬁguration found is then trained for 200 epochs with a batch size of 32.
Revised: The best training conﬁguration found is then trained for 200 epochs with a batch size of 32.

Original: In our rule extraction ﬁne-tuning, for ECLAIRE we use µ min = 50 , µ max = 200 , and δµ = 25 while for both REM-D and DeepRED * we use µ min = 500 , µ max = 1000 , and δµ = 50 .
Revised: In our rule extraction ﬁne-tuning, for ECLAIRE we use µ min = 50 , µ max = 200 , and δµ = 25 while for both REM-D and DeepRED * we use µ min = 500 , µ max = 1000 , and δµ = 50 .

Original: Note that we use large values of µ for REM-D and DeepRED * as we found it extremely hard to get reasonable extraction times when using less than 500 samples for µ .
Revised: Note that we use large values of µ for REM-D and DeepRED * as we found it extremely hard to get reasonable extraction times when using less than 500 samples for µ .

Original: Finally, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .
Revised: Finally, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .

Original: C.3.4 MiniBooNE
Revised: C.3.4 MiniBooNE

Original: We found the MiniBooNE task to require the most capacity to obtain good results compared to others experiments in this paper.
Revised: We found the MiniBooNE task to require the most capacity to obtain good results compared to others experiments in this paper.

Original: We force our model architecture to use 5 hidden layers and search for models trained with epochs ∈ { 20 , 30 , 40 } given the large training size.
Revised: We force our model architecture to use 5 hidden layers and search for models trained with epochs ∈ { 20 , 30 , 40 } given the large training size.

Original: This gives us a best performing architecture that uses hidden units {128, 64, 32, 16, 8} with an ELU activation in between them which is trained for 30 epochs with a batch size of 16.
Revised: This gives us a best performing architecture that uses hidden units {128, 64, 32, 16, 8} with an ELU activation in between them which is trained for 30 epochs with a batch size of 16.

Original: In our rule extraction ﬁne-tuning, this task proved to be more complicated than the rest given its training size.
Revised: In our rule extraction ﬁne-tuning, this task proved to be more complicated than the rest given its training size.

Original: For ECLAIRE we use µ min = 0 .
Revised: For ECLAIRE we use µ min = 0 .

Original: 0005 N , µ max = 0 .
Revised: 0005 N , µ max = 0 .

Original: 0015 N , and δµ = 0 .
Revised: 0015 N , and δµ = 0 .

Original: 0001 N (where
Revised: 0001 N (where

Original: N is the number of training samples) as values below 0 .
Revised: N is the number of training samples) as values below 0 .

Original: N result extraction times longer thanhours.
Revised: N result extraction times longer thanhours.

Original: For both REM-D and DeepRED * , we increase the minimum value of µ signiﬁcantly to get runs that terminate in their allotted times and use µ min = 0 .
Revised: For both REM-D and DeepRED * , we increase the minimum value of µ signiﬁcantly to get runs that terminate in their allotted times and use µ min = 0 .

Original: 02 N , µ max = 0 .
Revised: 02 N , µ max = 0 .

Original: 1 N , and δµ =
Revised: 1 N , and δµ =

Original: 005 N .
Revised: 005 N .

Original: Finally, given their fast extraction times, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .
Revised: Finally, given their fast extraction times, for C5.0 and PedC5.0 we search over all µ values deﬁned by µ min = 5 , µ max = 50 , and δµ = 5 .

Original: C.3.5 Letter Recognition
Revised: C.3.5 Letter Recognition

Original: For the results we report on the Letter Recognition dataset in Section 4.4, we search over architectures with 2 hidden layers and obtain a best performing model that has layers of size {128, 64} with ELU activations in between them.
Revised: For the results we report on the Letter Recognition dataset in Section 4.4, we search over architectures with 2 hidden layers and obtain a best performing model that has layers of size {128, 64} with ELU activations in between them.

Original: The best training conﬁguration is trained for 150 epochs with a batch size of 32.
Revised: The best training conﬁguration is trained for 150 epochs with a batch size of 32.

Original: In our rule extraction ﬁne-tuning, we use µ min = 5 , µ max = 15 , and δµ = 1 for ECLAIRE, C5.0, and PedC5.0.
Revised: In our rule extraction ﬁne-tuning, we use µ min = 5 , µ max = 15 , and δµ = 1 for ECLAIRE, C5.0, and PedC5.0.

Original: As in the physics datasets, both DeepRED * and REM-D struggle to terminate in a reasonable amount of time unless µ is signiﬁcantly high.
Revised: As in the physics datasets, both DeepRED * and REM-D struggle to terminate in a reasonable amount of time unless µ is signiﬁcantly high.

Original: This is exacerbated by the large number of classes in this task.
Revised: This is exacerbated by the large number of classes in this task.

Original: Because of this, we use µ min = 0 .
Revised: Because of this, we use µ min = 0 .

Original: 25 N , µ max = 0 .
Revised: 25 N , µ max = 0 .

Original: 5 N , and δµ =
Revised: 5 N , and δµ =

Original: 05 N during their ﬁne-tuning process.
Revised: 05 N during their ﬁne-tuning process.

Original: CART as Intermediate Rule Extractor in Letters
Revised: CART as Intermediate Rule Extractor in Letters

Original: While in all of our binary tasks we are able to construct high-performing rule sets with C5.0, we fail to observe this same trend in the Letters dataset, our task of choice for multi-class evaluation.
Revised: While in all of our binary tasks we are able to construct high-performing rule sets with C5.0, we fail to observe this same trend in the Letters dataset, our task of choice for multi-class evaluation.

Original: More speciﬁcally, Table 1 shows that end-to-end
Revised: More speciﬁcally, Table 1 shows that end-to-end

Original: C5.0 rule sets are unable to achieve a high performance compared to that previously reported for other induction algorithms [62].
Revised: C5.0 rule sets are unable to achieve a high performance compared to that previously reported for other induction algorithms [62].

Original: Therefore, in this section we explore the use of CART [63] trees for intermediate rule induction and show these results in Table 3.
Revised: Therefore, in this section we explore the use of CART [63] trees for intermediate rule induction and show these results in Table 3.

Original: In these experiments, we compare the performance of rule sets induced from end-to-end CART trees, as well as rule sets induced from
Revised: In these experiments, we compare the performance of rule sets induced from end-to-end CART trees, as well as rule sets induced from

Original: CART trees learnt from data that was labelled using the DNN’s predictions (which we refer to as
Revised: CART trees learnt from data that was labelled using the DNN’s predictions (which we refer to as

Original: PedCART ), against that of rule sets extracted with ECLAIRE when CART is used as its intermediate rule extractor.
Revised: PedCART ), against that of rule sets extracted with ECLAIRE when CART is used as its intermediate rule extractor.

Original: Furthermore, for the sake of obtaining a fair comparison between our baselines that is unbiased with respect to the choice of rule extraction algorithm, we also compare ECLAIRE against versions of both REM-D and DeepRED that use CART as an intermediate rule extractor.
Revised: Furthermore, for the sake of obtaining a fair comparison between our baselines that is unbiased with respect to the choice of rule extraction algorithm, we also compare ECLAIRE against versions of both REM-D and DeepRED that use CART as an intermediate rule extractor.

Original: For clarity, we refer to these versions of our baselines as ECLAIRE CART , REM-D CART , and DeepRED CART , respectively.
Revised: For clarity, we refer to these versions of our baselines as ECLAIRE CART , REM-D CART , and DeepRED CART , respectively.

Original: In all of these experiments, we control the growth of CART-generated trees using Cost Complexity
Revised: In all of these experiments, we control the growth of CART-generated trees using Cost Complexity

Original: Pruning (CCP) [64] and by varying the number of minimum samples per split µ as in our previous tasks.
Revised: Pruning (CCP) [64] and by varying the number of minimum samples per split µ as in our previous tasks.

Original: For CART, PedCART, and ECLAIRE CART we search over the spectrum deﬁned by µ min = 0 .
Revised: For CART, PedCART, and ECLAIRE CART we search over the spectrum deﬁned by µ min = 0 .

Original: 0001 N , µ max = 0 .
Revised: 0001 N , µ max = 0 .

Original: 0051 N , and δµ =
Revised: 0051 N , and δµ =

Original: 0005 N .
Revised: 0005 N .

Original: As it was the case when using C5.0 as an intermediate rule extractor, for both DeepRED CART and REM-D CART we require signiﬁcantly higher values of µ to terminate in the allotted time.
Revised: As it was the case when using C5.0 as an intermediate rule extractor, for both DeepRED CART and REM-D CART we require signiﬁcantly higher values of µ to terminate in the allotted time.

Original: Because of this, we limit our search over µ to be over the spectrum deﬁned by µ min = 0 .
Revised: Because of this, we limit our search over µ to be over the spectrum deﬁned by µ min = 0 .

Original: 1 N , µ max = 0 .
Revised: 1 N , µ max = 0 .

Original: 3 N , and δµ =
Revised: 3 N , and δµ =

Original: 05 N .
Revised: 05 N .

Original: This results in overpruned rule sets in DeepRED CART which, although small in size, take orders of magnitude more time to extract than those generated by ECLAIRE.
Revised: This results in overpruned rule sets in DeepRED CART which, although small in size, take orders of magnitude more time to extract than those generated by ECLAIRE.

Original: [Table]
Revised: [Table]

Original: Our results show a signiﬁcant increase in accuracy when using end-to-end CART trees for rule induction over C5.0 trees (86% ± 0.4% vs 63.1% ± 2.5%).
Revised: Our results show a signiﬁcant increase in accuracy when using end-to-end CART trees for rule induction over C5.0 trees (86% ± 0.4% vs 63.1% ± 2.5%).

Original: However, we note that this increase comes with a signiﬁcant rise in the number of rules extracted from CART trees compared to rule sets extracted from C5.0 trees.
Revised: However, we note that this increase comes with a signiﬁcant rise in the number of rules extracted from CART trees compared to rule sets extracted from C5.0 trees.

Original: Regardless, our results also show that the same relative ranking we observe across our baselines in Table 1 holds when using CART as intermediate rule extractor: both ECLAIRE CART and ECLAIRE ∗ CART are able to extract rule sets with higher ﬁdelity than those extracted by PedCART, while ECLAIRE ∗ CART extracts rule sets that achieve a higher predictive accuracy than those extracted by both end-to-end CART trees and PedCART.
Revised: Regardless, our results also show that the same relative ranking we observe across our baselines in Table 1 holds when using CART as intermediate rule extractor: both ECLAIRE CART and ECLAIRE ∗ CART are able to extract rule sets with higher ﬁdelity than those extracted by PedCART, while ECLAIRE ∗ CART extracts rule sets that achieve a higher predictive accuracy than those extracted by both end-to-end CART trees and PedCART.

Original: Similarly, we observe that DeepRED CART and REM-D CART are unable to extract rule sets that perform better than random.
Revised: Similarly, we observe that DeepRED CART and REM-D CART are unable to extract rule sets that perform better than random.

Original: D DeepRED’s and REM-D’s Substitution Step
Revised: D DeepRED’s and REM-D’s Substitution Step

Original: The algorithm produces temporary rule sets
Revised: The algorithm produces temporary rule sets

Original: I h i (
Revised: I h i (

Original: cid:55)→ t k to map activations in layer h
Revised: cid:55)→ t k to map activations in layer h

Original: i to the binary truth value of term t k .
Revised: i to the binary truth value of term t k .

Original: It then replaces each term t k in R
Revised: It then replaces each term t k in R

Original: h
Revised: h

Original: i +1 (cid:55)→ l with the set of rules in I
Revised: i +1 (cid:55)→ l with the set of rules in I

Original: h i (cid:55)→
Revised: h i (cid:55)→

Original: t
Revised: t

Original: k that have
Revised: k that have

Original: TRUE as a conclusion.
Revised: TRUE as a conclusion.

Original: Similarly to Figure 2, we let p ( k ) j be the premise of the j -th rule in I
Revised: Similarly to Figure 2, we let p ( k ) j be the premise of the j -th rule in I

Original: h i (cid:55)→ t k whose consequence is TRUE .
Revised: h i (cid:55)→ t k whose consequence is TRUE .

Original: For this example, we assume that the number of premises with TRUE as a conclusion in I h i
Revised: For this example, we assume that the number of premises with TRUE as a conclusion in I h i

Original: (cid:55)→ t 1 and I h i (cid:55)→ t 2 is N 1 and N 2 , respectively.
Revised: (cid:55)→ t 1 and I h i (cid:55)→ t 2 is N 1 and N 2 , respectively.

Original: This results in rule set R h
Revised: This results in rule set R h

Original: i (cid:55)→ l having N 1 N 2 rules after substitution.
Revised: i (cid:55)→ l having N 1 N 2 rules after substitution.

Original: E Growth Coping Mechanisms
Revised: E Growth Coping Mechanisms

Original: Although ECLAIRE can scale to large datasets and models, its cubic growth factor implies that it can suffer from slow extraction times.
Revised: Although ECLAIRE can scale to large datasets and models, its cubic growth factor implies that it can suffer from slow extraction times.

Original: In this section, we include results of experiments in which we explore whether it is possible to alleviate these scaling issues without signiﬁcantly sacriﬁcing performance.
Revised: In this section, we include results of experiments in which we explore whether it is possible to alleviate these scaling issues without signiﬁcantly sacriﬁcing performance.

Original: Speciﬁcally, we explore four different mechanisms:
Revised: Speciﬁcally, we explore four different mechanisms:

Original: Intermediate Rule Pruning
Revised: Intermediate Rule Pruning

Original: The simplest mechanism for controlling both runtime and comprehensibility is to constrain the size of intermediate rule sets by increasing the value of µ .
Revised: The simplest mechanism for controlling both runtime and comprehensibility is to constrain the size of intermediate rule sets by increasing the value of µ .

Original: As seen in Figure 6, varying this parameter can result in a drastic drop in the number of output rules.
Revised: As seen in Figure 6, varying this parameter can result in a drastic drop in the number of output rules.

Original: However, this may come at the cost of ﬁdelity.
Revised: However, this may come at the cost of ﬁdelity.

Original: This behaviour underlines one of the biggest limitations of ECLAIRE: its sensitivity to µ makes
Revised: This behaviour underlines one of the biggest limitations of ECLAIRE: its sensitivity to µ makes

Original: ECLAIRE hard to use in large tasks without the need for extensive ﬁne-tuning.
Revised: ECLAIRE hard to use in large tasks without the need for extensive ﬁne-tuning.

Original: Nevertheless, in most tasks we observe that values of µ in the [10 − 5 N, 10 − 4 N ] range tend to give good performance.
Revised: Nevertheless, in most tasks we observe that values of µ in the [10 − 5 N, 10 − 4 N ] range tend to give good performance.

Original: Moreover, these experiments also suggest that if one can afford a small drop in performance in favour of a more comprehensible rule set, then increasing µ to be in the range [2 × 10 − 4 N, 5 × 10 − 4 N ] can lead to a very comprehensible yet still high-performing rule set.
Revised: Moreover, these experiments also suggest that if one can afford a small drop in performance in favour of a more comprehensible rule set, then increasing µ to be in the range [2 × 10 − 4 N, 5 × 10 − 4 N ] can lead to a very comprehensible yet still high-performing rule set.

Original: Hidden Representation Subsampling
Revised: Hidden Representation Subsampling

Original: When dealing with networks with multiple layers in them, one can alleviate ECLAIRE’s scalability by extracting intermediate rule sets
Revised: When dealing with networks with multiple layers in them, one can alleviate ECLAIRE’s scalability by extracting intermediate rule sets

Original: R h i (cid:55)→
Revised: R h i (cid:55)→

Original: ˆ y only for a subset of hidden layers i ∈ S ⊂ { 1 , 2 , . . .
Revised: ˆ y only for a subset of hidden layers i ∈ S ⊂ { 1 , 2 , . . .

Original: , d } .
Revised: , d } .

Original: This forces ECLAIRE to use fewer intermediate representations to build its rule set and results in the algorithm effectively operating on a DNN with |S| < d hidden layers in it.
Revised: This forces ECLAIRE to use fewer intermediate representations to build its rule set and results in the algorithm effectively operating on a DNN with |S| < d hidden layers in it.

Original: We explore this possibility in our MiniBooNE network by sampling intermediate layers with different frequencies.
Revised: We explore this possibility in our MiniBooNE network by sampling intermediate layers with different frequencies.

Original: Our results, shown in Figure 7, suggest that one can sample hidden layers at very high frequencies (i.e., everyhidden layers) at a very small cost to ﬁdelity; all while halving both the runtime and rule set size.
Revised: Our results, shown in Figure 7, suggest that one can sample hidden layers at very high frequencies (i.e., everyhidden layers) at a very small cost to ﬁdelity; all while halving both the runtime and rule set size.

Original: One may therefore beneﬁt from subsampling hidden layers if the input network is deep enough to support this mechanism.
Revised: One may therefore beneﬁt from subsampling hidden layers if the input network is deep enough to support this mechanism.

Original: Training Set Subsampling
Revised: Training Set Subsampling

Original: If the scalability issue comes from the number of training samples, one can use only a fraction of the available training data for ECLAIRE’s rule construction.
Revised: If the scalability issue comes from the number of training samples, one can use only a fraction of the available training data for ECLAIRE’s rule construction.

Original: Our experiments in MiniBooNE show that ECLAIRE is very data efﬁcient , maintaining its performance in low-data regimes.
Revised: Our experiments in MiniBooNE show that ECLAIRE is very data efﬁcient , maintaining its performance in low-data regimes.

Original: These results, shown in Figure 8, indicate that one can subsample up to 50% of the training data while incurring a very small drop in ﬁdelity and reducing the extraction time in half.
Revised: These results, shown in Figure 8, indicate that one can subsample up to 50% of the training data while incurring a very small drop in ﬁdelity and reducing the extraction time in half.

Original: This behaviour is in line with empirical evidence by Zilke et al.
Revised: This behaviour is in line with empirical evidence by Zilke et al.

Original: suggesting that other decompositional methods can maintain their performance while using a fraction of the DNN’s training set [25].
Revised: suggesting that other decompositional methods can maintain their performance while using a fraction of the DNN’s training set [25].

Original: Intermediate Rule Ranking
Revised: Intermediate Rule Ranking

Original: It has been previously observed that one can drop signiﬁcant portions of a rule set while incurring only in small performance costs [40, 65].
Revised: It has been previously observed that one can drop signiﬁcant portions of a rule set while incurring only in small performance costs [40, 65].

Original: Inspired by this observation, we explore whether ECLAIRE can beneﬁt from dropping the lowest p % of rules in intermediate rule sets R h
Revised: Inspired by this observation, we explore whether ECLAIRE can beneﬁt from dropping the lowest p % of rules in intermediate rule sets R h

Original: i
Revised: i

Original: (cid:55)→ ˆ y as ranked by their conﬁdence level 3 .
Revised: (cid:55)→ ˆ y as ranked by their conﬁdence level 3 .

Original: Our results, shown in Figure 9, suggest that ECLAIRE is robust to substantial pruning of its intermediate rules without loosing much of its performance.
Revised: Our results, shown in Figure 9, suggest that ECLAIRE is robust to substantial pruning of its intermediate rules without loosing much of its performance.

Original: For example, one can drop about 25% of all intermediate rules while experiencing a very small cost in ﬁdelity; all while using less resources and generating fewer rules.
Revised: For example, one can drop about 25% of all intermediate rules while experiencing a very small cost in ﬁdelity; all while using less resources and generating fewer rules.

Original: Both of these results suggest that dropping a small fraction of rules can help scaling ECLAIRE to large tasks.
Revised: Both of these results suggest that dropping a small fraction of rules can help scaling ECLAIRE to large tasks.