Original: A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines
Revised: A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines

Original: Abstract
Revised: Abstract

Original: Understanding the results of deep neural networks is an essential step towardswider acceptance of deep learning algorithms.
Revised: Understanding the results of deep neural networks is an essential step towards wider acceptance of deep learning algorithms.

Original: Many approaches address the issueof interpreting artiﬁcial neural networks, but often provide divergent explanations.
Revised: Many approaches address the issue of interpreting artiﬁcial neural networks, but often provide divergent explanations.

Original: Moreover, different hyperparameters of an explanatory method can lead to conﬂicting interpretations.
Revised: Moreover, different hyperparameters of an explanatory method can lead to conﬂicting interpretations.

Original: In this paper, we propose a technique for aggregating thefeature attributions of different explanatory algorithms usinga Restricted Boltzmann machine (RBM) to achieve a more accurate and robust interpretation of deepneural networks.
Revised: In this paper, we propose a technique for aggregating the feature attributions of different explanatory algorithms using Restricted Boltzmann Machines (RBMs) to achieve a more accurate and robust interpretation of deep neural networks.

Original: Several challenging experiments on real-world datasets show thatthe proposed RBM method outperforms popular feature attribution methods andbasic ensemble techniques
Revised: Several challenging experiments on real-world datasets show that the proposed RBM method outperforms popular feature attribution methods and basic ensemble techniques

Original:  Introduction
Revised:  Introduction

Original: As the applications of deep neural networks (DNNs) continue to grow, the black-box nature of DNNscreates potential trust issues [1].
Revised: As the applications of deep neural networks (DNNs) continue to grow, the black-box nature of DNNs creates potential trust issues [1].

Original: Moreover, numerous life-critical (such as medical, automotive, orﬁnancial) applications utilize DNNs for various estimation tasks.
Revised: Moreover, numerous life-critical (such as medical, automotive, or ﬁnancial) applications utilize DNNs for various estimation tasks.

Original: In such applications, and especiallyfor the long-term acceptance of Artiﬁcial Intelligence (AI) solutions, a deeper understanding andtrust in the produced results is crucial.
Revised: In such applications, and especially for the long-term acceptance of artiﬁcial intelligence (AI) solutions, a deeper understanding and trust in the produced results is crucial.

Revised: Furthermore, feature attribution methods are important tools for deep model debugging and diagnosis [2].

Original: Explaining how the input inﬂuences the output for a given DNN is one form to interpret the black-boxnature of the DNN and bring trust to a system.
Revised: Explaining how the input inﬂuences the output for a given DNN is one form to interpret the black-box nature of the DNN and bring trust to a system.

Original: These so-called feature-based explanation methodsreceived a lot of attention in recent years [1,2, 3, 4].
Revised: These so-called feature-based explanation methods received a lot of attention in recent years [1, 3, 4, 5].

Original: They can be grouped into three broad categories,(1) approaches based on gradient information [5, 6], (2) perturbation-based approaches [7, 8], and(3) attribution-based approaches [2, 9].
Revised: They can be grouped into three broad categories, (1) approaches based on gradient information [6, 7], (2) perturbation-based approaches [8, 9], and (3) attribution-based approaches [3, 10].

Original: Interestingly, different feature-based explanation approachesregularly produce mixed views on the main attributes (areas of an image or variables), and in theabsence of the ground truth, it is still a challenge to verify which explanation method is the mosttrustworthy.
Revised: Interestingly, different feature-based explanation approaches regularly produce mixed views on the main attributes (areas of an image or variables), and in the absence of the ground truth, it is still a challenge to verify which explanation method is the most trustworthy.

Original: Moreover, in the AI community, there are no  accepted quality measures for feature-basedexplanations.
Revised: Moreover, in the AI community, there are no yet accepted quality measures for featurebased explanations.

Original: All these difﬁculties resulted in a large number of different explanation methods and ina lack of consensus on which techniques are most accurate.
Revised: All these difﬁculties resulted in a large number of different explanation methods and in a lack of consensus on which techniques are most reliable.

Original: Within the machine learning (ML) community, there is much work on the combination of methodsthat do not always agree with each other, i.e. ensemble learning [10, 11].
Revised: Within the machine learning (ML) community, there is much work on the combination of methods that do not always agree with each other, i.e. ensemble learning [11, 12].

Original: Normally ensemble modelsoutperform the non-ensemble models and turn out to be more robust to outliers.
Revised: Normally ensemble models outperform the non-ensemble models and turn out to be more robust to outliers.

Original: The main idea is thatif multiple methods make mistakes in different areas, combining them in an intelligent way improvesperformance and reduces the effect of outliers as compared to the single method.
Revised: The main idea is that if multiple methods make mistakes in different areas, combining them in an intelligent way improves performance and reduces the effect of outliers as compared to the single method.

Original: Moreover, fromstatistical learning theory and practical applications, it is well understood that ensemble learning isthe path of choice towards a more robust machine learning system [12], even in unsupervised learningscenarios where the target is not available [13, 14].
Revised: Moreover, from statistical learning theory and practical applications, it is well understood that ensemble learning is the path of choice towards a more robust machine learning system [13], even in unsupervised learning scenarios where the target is not available [14, 15].

Original: In this work, utilizing ideas from[14, 10] and [15, 16], we introduce a novel approach for theunsupervised ensemble learning of accurate and robust feature-based explanations for deep neuralnetworks.
Revised: In this work, utilizing ideas from [15, 11] and [16, 17], we introduce a novel approach for the unsupervised ensemble learning of reliable and robust feature-based explanations for deep neural networks.

Original: To this end, we propose using a  Restricted Boltzmann Machine (RBM), which achievesthis goal by aggregating the results (saliency maps) of different feature-based explanation methodsin a principled probabilistic fashion.
Revised: To this end, we propose using a model based on Restricted Boltzmann Machines (RBMs), which achieves this goal by aggregating the results (saliency maps) of different feature-based explanation methods in a principled probabilistic fashion.

Original: Also, it has been shown that an RBM can be used in the truthdiscovery setting [15, 16], which is analogous to our task of ﬁnding a reliable feature importancemap from different importance maps.
Revised: Also, it has been shown that an RBM can be used in the truth discovery setting [18, 19], which is analogous to our task of ﬁnding a reliable feature importance map from different importance maps.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • We introduce a novel method for a robust and accurate feature-based explanation usingensemble learning.
Revised: • We introduce a novel method for a robust and reliable feature-based explanation using ensemble learning.

Original: • We empirically and visually show the superior performance of the proposed method incomparison to state-of-the-art feature attribution 
Revised: • We empirically and visually show the superior performance of the proposed method in comparison to state-of-the-art feature attribution baselines.

Original: baselines.• We open-source our code and make it publicly available(https://anonymous.4open.science/r/AggregationOfLocalExplanations80D7/example.ipynb).
Revised: • We open-source our code and make it publicly available, as an RBM ensemble framework: (https://github.com/JohanvandenHeuvel/AggregationOfLocalExplanations), Besides, we also developed a single Python package with various evaluation metrics for feature attribution methods metrics: https://github.com/meier-johannes94/ExplainableAIImageMeasures

Original: The paper is organized as follows:
Revised: The paper is organized as follows:

Original: In Section 2 we discuss the related work and provide essentialbackground information.
Revised: In Section 2 we discuss the related work and provide essential background information.

Original: Section 3 presents the proposed ensemble method for local feature-Basedexplanations using a restricted Boltzmann machine.
Revised: Section 3 presents the proposed ensemble method for local feature-based explanations using an RBM.

Original: Inthe following Section 4, we present the resultsof variouschallenging experiments.
Revised: In Section 4, we present the results of various experiments.

Original: Section 5 discuss the limitation of our work and ways how toaddress in the future.
Revised: Sectiondiscusses limitations of our work and ways to address them in the future.

Original: Section 6 concludes our work with a short summary.
Revised: Section 6 concludes our work with a short summary.

Original:  Background and Related Work
Revised:  Background and Related Work

Original: This part of the manuscript provides the  background and  related work.
Revised: This part of the manuscript provides the needed background and discusses related approaches.

Original: First, we present the basicnotation used in this work and proceed by presenting two ensemble techniques for aggregating featureimportance maps.
Revised: First, we present the basic notation used in this work and proceed by presenting two ensemble techniques for aggregating feature importance maps.

Original: 2.1 Feature Attribution Function
Revised: 2.1 Feature Attribution Function

Original: Formally, a feature attribution function can be seen as φ ( f, x , c x ) , where f is a black box model andx is an input data point from a corresponding class c x .
Revised: Formally, a feature attribution function can be seen as φ ( f, x , c x ) , where f is a black box model and x is an input data point from a corresponding class c x .

Original: The output of φ is an explanation vector ormatrix
Revised: The output of φ is an explanation vector or matrix

Original: e f ( x ) . Where each element of e f ( x ) is an importance score for the corresponding feature valuein x .
Revised: e f ( x ) , where each element of e f ( x ) is an importance score for the corresponding feature value in x .

Original: A large positive or negative value in e f ( x ) indicates that the corresponding feature (pixel) has alarge inﬂuence on the outcome of the black-box model f .
Revised: A large positive or negative value in e f ( x ) indicates that the corresponding feature (pixel) has a large inﬂuence on the outcome of the black-box model f .

Original: Assumption 2.

Revised: Assumption 2.

Original: In the following, we assume that a true feature attribution ¯ e f ( x ) for a given model fand input x exists and can be constructed by adequately aggregating available attributions e f ( x ) ,i , i ∈{ 1 , ..., N } .
Revised: In the following, we assume that a true feature attribution ¯ e f ( x ) for a given model f and input x exists and can be constructed by adequately aggregating available attributions e f ( x ) ,i , i ∈ { 1 , ..., N } , where N is the number of base learners.

Original: For better readability and simplicity, from here we omit the index f ( x ) .
Revised: For better readability and simplicity, from here we omit the index f ( x ) .

Original: The goal of any explanation method φ is to obtain an attribution e that is as close as possible to¯ e .
Revised: The goal of any explanation method φ is to obtain an attribution e that is as close as possible to ¯ e .

Revised: Note that our method naturally generalizes to probabilistic local explanation methods [20].

Original: Given this assumption, we can say that there is a joint probability distribution of the pair ( e , ¯ e )parametrized by θ .
Revised: Given the before-mentioned assumption, we can say that there is a joint probability distribution of the pair ( e , ¯ e ) parametrized by θ .

Original: [Equation]
Revised: [Equation]

Original: The joint distribution p θ ( e , ¯ e ) is not known, and neither are the marginals p θ ( e ) , p θ (¯ e ) .
Revised: The joint distribution p θ ( e , ¯ e ) is not known, and neither are the marginals p θ ( e ) , p θ (¯ e ) .

Original: For the following theoretical results we require that the explanation methods give independentexplanations when conditioned on the true explanation.
Revised: For the following theoretical results we require that the explanation methods give independent explanations when conditioned on the true explanation.

Original: However,  forour practical implementation, itdoes not matter whether this assumption is violated [14].
Revised: However, as with Naive Bayes methods, for practical purposes, this assumption can be violated without negatively impacting the aggregation quality [15]. Also note that we do assume some consistency between the explanations, following the assumption that feature attributions reﬂect the underlying (but unknown) importance distributions of the feature values [21]. Assuming conditional independence between the provided base explanations given the (unknown) true explanation, we have

Original: [Equation]
Revised: [Equation]

Original: where e n is a base learner (sometimes a weak learner) in the ensemble involves N different learners.
Revised: where e n is a base explanation in the ensemble involving N different base explanations.

Original: 2.2 Ensemble Learning
Revised: 2.2 Ensemble Learning

Original: As we state in the introduction, ensemble learning is a well studied approach for improving theperformance of an ML system.
Revised: As we state in the introduction, ensemble learning is a well-studied approach for improving the performance of an ML system.

Original: One of the most basic ensemble methods employs the mean of resultsof base learners [12], where a base learner is a single algorithm from the ensemble.
Revised: One of the most basic ensemble methods employs the mean of results of base learners [13], where a base learner is a single algorithm from the ensemble.

Original: [Equation]
Revised: [Equation]

Original: A signiﬁcant drawback of the mean ensemble approach is that it still is sensitive to outliers or noisyestimations.
Revised: A signiﬁcant drawback of the mean ensemble approach is that it still is sensitive to outliers or noisy estimations.

Original: Furthermore, data scaling may strongly inﬂuence the aggregation.
Revised: Furthermore, data scaling may strongly inﬂuence the aggregation.

Original: In Section 4, theseweaknesses of the mean ensemble approach are also seen in the experimental evaluation.
Revised: In Section 4, these weaknesses of the mean ensemble approach are also seen in the experimental evaluation.

Original: To mitigate these weaknesses, the authors of [17] propose to take the local uncertainty into account.
Revised: To mitigate these weaknesses, the authors of [22] propose to take the local uncertainty into account.

Original: To this end they divide the mean by the local variance plus a constant (cid:15) for stability reasons, whichresults in the variance ensemble approach :
Revised: To this end, they divide the mean by the local variance plus a constant (cid:15) for stability reasons, which results in the variance ensemble approach :

Original: [Equation]
Revised: [Equation]

Original: where σ ∗ ( e i ∈{ 1 ,...,N } ) is the point-wise standard deviation over all the available explanations
Revised: where σ ∗ ( e i ∈{ 1 ,...,N } ) is the point-wise standard deviation over all the available explanations

Original: e i , i ∈{ 1 , ..., N } .
Revised: e i , i ∈ { 1 , ..., N } .

Original: This method assigns less relevance to explanations that have high disagreement with theremaining explanations.
Revised: This method assigns less relevance to explanations that have high disagreement with the remaining explanations.

Revised: Also, the authors of [23] proposed a novel method to aggregate Shapley values through an explanation function that minimizes sensitivity.

Original:  Ensemble Learning using Restricted Boltzmann Machines
Revised:  Ensemble Learning using Restricted Boltzmann Machines

Revised: [Equation]

Revised: three visible and two hidden units.

Revised: In our work, we use an RBM with a single hidden node.

Original: In this section, we present an unsupervised aggregation of feature attribution maps using a Restricted 
Revised: In this section, we present an unsupervised aggregation of feature attribution maps using a Restricted Boltzmann

Original: Boltzmann Machine (RBM).
Revised: Machine (RBM).

Original: Similar aggregation techniques have been proposed in other contexts,e.g., in [14, 15].
Revised: Similar aggregation techniques have been proposed in other contexts, e.g., in [15, 18].

Original: 3.1 The Restricted Boltzmann Machine
Revised: 3.1 The Restricted Boltzmann Machine

Original: An RBM is an undirected bipartite graph that can be parametrized by a neural network.
Revised: An RBM is an undirected bipartite graph that can be parametrized by a neural network.

Original: It is a variantof the Boltzmann Machine, with the additional property that there are no connections within boththe group of visible nodes or the group of hidden nodes.
Revised: It is a variant of the Boltzmann Machine, with the additional property that there are no connections within both the group of visible nodes or the group of hidden nodes.

Original: The advantage of this property is thatnodes in one group are conditionally independent of each other, given that we know the state of thenodes in the other group.
Revised: The advantage of this property is that nodes in one group are conditionally independent of each other, given that we know the state of the nodes in the other group.

Original: One of the main characteristics of an RBM is that it can learn a probabilitydistribution over its set of inputs.
Revised: One of the main characteristics of an RBM is that it can learn a probability distribution over its set of inputs.

Original: A graphical representation of an example RBM is shown in Figure1. [Equation]
Revised: A graphical representation of an example RBM is shown in Figure 1.

Original: two The formal deﬁnition of an RBM is as follows.
Revised: The formal deﬁnition of an RBM is as follows.

Original: There isa set X of n visible binary random variables and a set H of m hidden binary random variables.
Revised: There is a set X of n visible binary random variables and a set H of m hidden binary random variables.

Original: The RBM hasparameters λ =
Revised: The RBM has parameters λ =

Original: ( W, a, b ) .
Revised: ( W , a , b ) .

Original: W is the weight matrix of theconnections between the nodes, a is the bias of the visiblelayer and b is the bias of the hidden layer.
Revised: W is the weight matrix of the connections between the nodes, a is the bias of the visible layer and b is the bias of the hidden layer.

Original: Each possiblestate of the RBM, i.e. the particular values of ( X, H ) , isassociated with the following energy function (in matrixnotation):
Revised: Each possible state of the RBM, i.e. the particular values of ( X, H ) , is associated with the following energy function (in matrix notation):

Original: [Equation]
Revised: [Equation]

Original: which then can also be used to deﬁne the joint probability distribution for the visible and hiddenvectors is deﬁned in terms of the energy function:
Revised: which then can also be used to deﬁne the joint probability distribution for the visible and hidden vectors is deﬁned in terms of the energy function:

Original: [Equation]
Revised: [Equation]

Original: where Z is the sum over e − E ( x,h ) for all possible conﬁgurations x, h , which can be seen as anormalization constant to ensure that all probabilities sum to 1 , also known as the partition function.
Revised: where Z is the sum over e − E ( x , h ) for all possible conﬁgurations x , h , which can be seen as a normalization constant to ensure that all probabilities sum to 1 , also known as the partition function.

Original: The optimization objective of the RBM is to maximize the expected log probability of a trainingsample x :
Revised: The optimization objective of the RBM is to maximize the expected log probability of a training sample x :

Original: [Equation]
Revised: [Equation]

Original: To train an RBM, a gradient-based optimization can be applied using the contrastive divergencealgorithm [18, 19].
Revised: To train an RBM, a gradient-based optimization can be applied using the contrastive divergence algorithm [24, 25].

Original: 3.2 Aggregation of Local Explanations using an RBM
Revised: 3.2 Aggregation of Local Explanations using an RBM

Original: Given an RBM with N visible nodes and one hidden node, with input x from a conditional independent model, where N is the number of base learners in our ensemble, it can be shown that thetrue posterior probability of y can be efﬁciently estimated (Lemma 4.1, Lemma 4.2 from [14]).
Revised: Given an RBM with N visible nodes and one hidden node, with input x from a conditional independent model, where N is the number of base learners in our ensemble, it can be shown that the true posterior probability of y can be efﬁciently estimated (Lemma 4.1, Lemma 4.2 from [15]).

Original: Furthermore, given mild assumptions [14], the maximum likelihood estimate ¯ λ MLE for the parametersof the RBM, the RBM posterior probability P ¯ λ MLE ( H = 1 | X = x ) converges to true posterior {+P θ ( Y = 
+}
Revised: Furthermore, given mild assumptions [15], the maximum likelihood estimate ¯ λ MLE for the parameters of the RBM, the RBM posterior probability P ¯ λ MLE ( H = 1 | X = x ) converges to true posterior {+P θ ( Y = 
+}

Original: P θ ( Y = 1 | X = x ) .
Revised: | X = x ) .

Original: Hence, taking into account the assumption we made in Section 2 and [14], we are able to applythe RBM to the unsupervised aggregation of N available feature-based explanations.
Revised: Hence, taking into account the assumption we made in Section 2 and [15], we are able to apply the RBM to the unsupervised aggregation of N available feature-based explanations.

Original: We assume ajoint distribution p θ ( e , ¯ e ) , and that the e i ’s are conditionally independent from each other given ¯ e) .
Revised: We assume a joint distribution p θ ( e , ¯ e ) , and that the e i ’s are conditionally independent from each other given ¯ e .

Original: By ﬁtting the RBM we learn the parameters θ and thus obtain the relationship between our knownexplanations e i and the true explanation ¯ e .
Revised: By ﬁtting the RBM we learn the parameters θ and thus obtain the relationship between our known explanations e i and the true explanation ¯ e .

Original: The ensemble pipeline of the proposed method is depictedin Fig.
Revised: The ensemble pipeline of the proposed method is depicted in Fig.

Original: In order to preserve the spatial information for visual data using the RBM-based ensemble, we doa pixel-wise aggregation.
Revised: In order to preserve the spatial information for visual data using the RBM-based ensemble, we do a pixel-wise aggregation.

Original: Therefore, for each pixel we train a Bernoulli RBM with a single hiddennode.
Revised: Therefore, for each pixel we train a Bernoulli RBM with a single hidden node.

Original: A known limitation of an RBM is the so-called ﬂipping issue [14, 15, 16], which arises from theparametrization symmetry.
Revised: A known limitation of an RBM is the so-called ﬂipping issue [15, 18, 19], which arises from the RBM parametrization symmetry.

Original: That is, the weights of the RBM can be ﬂipped symmetrically withoutchanging the behavior of the RBM.
Revised: That is, the weights of the RBM can be ﬂipped symmetrically without changing the behavior of the RBM.

Original: In order to avoid this unwanted effect, we propose two approaches:ﬂip detection and metric optimization.
Revised: In order to avoid this unwanted effect, we propose two approaches: ﬂip detection and metric optimization.

Original: The ﬂip detection algorithm extends the idea from Remark 4.3in [14], by comparing the top 5 % of most important and 5 % of less important pixels to the meanbaseline.
Revised: The ﬂip detection algorithm extends the idea from Remark 4.3 in [15], by comparing the top 5 % of most important and 5 % of less important pixels to the mean baseline.

Original: The algorithm inverts the current important scores if there is a strong disagreement betweenthe proposed approach and the mean baseline.
Revised: The algorithm inverts the current important scores if there is a strong disagreement between the proposed approach and the mean baseline.

Original: The metric optimization method utilizes the chosenmetric to overcome the ﬂipping issue.
Revised: The metric optimization method utilizes the chosen metric to overcome the ﬂipping issue.

Original: It compares two versions of the RBM ensemble results andselects the one with a better performance according to the selected metric.
Revised: It compares two versions of the RBM ensemble results and selects the one with a better performance according to the selected metric.

Original:  Experiments
Revised:  Experiments

Original: To demonstrate the effectiveness of the proposed ensemble algorithm we conduct various visualand quantitative experiments.
Revised: To demonstrate the effectiveness of the proposed ensemble algorithm we conduct various visual and quantitative experiments.

Original: First, we present the visual inspection results on the MNIST [20] and
Revised: First, we present the visual inspection results on the MNIST [27] and

Original: ImageNet [21] datasets in two settings, with and without noisy explanation maps in our ensemble.
Revised: ImageNet [26] datasets in two settings, with and without noisy explanation maps in our ensemble.

Original: Despite a growing body of research focusing on explainable ML, the fair quantitative comparison ofsaliency algorithm benchmarks is still an open question, since the existing methods mostly utilize thepixel perturbation strategy (e.g., removing the most or least important pixels and reporting the changein recognition quality) [22, 23].
Revised: Despite a growing body of research focusing on explainable ML, the fair quantitative comparison of saliency algorithm benchmarks is still an open question, since the existing methods mostly utilize the pixel perturbation strategy (e.g., removing the most or least important pixels and reporting the change in recognition quality) [28, 29].

Original: However, these benchmarks have a signiﬁcant drawback, replacingimage pixels with black or "mean" or any other pixel values may lead to artifacts affecting the datadistribution [23].
Revised: Also, these benchmarks have a signiﬁcant drawback, replacing image pixels with black or "mean" or any other pixel values may lead to artifacts affecting the data distribution [29].

Original: Nevertheless, since pixel perturbation benchmarks are employed in many relatedworks, for our quantitative analysis we select the following approaches: the pixel perturbation forinsertion (IAUC) and deletion (DAUC) and IROF [24] benchmarks on the CIFAR10 dataset [25].
Revised: Nevertheless, since pixel perturbation benchmarks are employed in many related works, for our quantitative analysis we select the following approaches: the pixel perturbation for insertion (IAUC) and deletion (DAUC). Furthermore, we utilize the iterative removal of features (IROF) benchmark [30].

Original: Weexplain each benchmark method it details in the corresponding subsections.
Revised: We explain each benchmark method it details in the corresponding subsections.

Original: In our last experiment, we demonstrate that our ensemble approach can be  used within a singe featureattribution framework to achieve more robust and stable explanations.
Revised: In our last experiment, we demonstrate that our ensemble approach can be also used within a singe feature attribution framework to achieve more robust and stable explanations.

Original: Since, it has been shown thathyperparameters choice can signiﬁcantly affect the saliency maps [26].
Revised: Since, it has been shown that hyperparameters choice can signiﬁcantly affect the saliency maps [31].

Original: 4.1 Visual Inspection for Image Data
Revised: 4.1 Visual Inspection for Image Data

Original: In our ﬁrst experiment, a visual evaluation on images from ImageNet [21] and MNIST [20] isperformed for several baseline and ensemble methods.
Revised: In our ﬁrst experiment, a visual evaluation on images from ImageNet [26] and MNIST [27] is performed for several baseline and ensemble methods.

Original: We provide benchmark outcomes for twosettings, with and without ﬁfteen noisy base learners in an ensemble.
Revised: We provide benchmark outcomes for two settings, with and without ﬁfteen noisy base learners in an ensemble.

Original: The results are depicted in
Revised: The results are depicted in

Original: Table 1.
Revised: Table 1.

Original: Without artiﬁcial noise in the ensemble.
Revised: Without artiﬁcial noise in the ensemble.

Original: We select four samples from ImageNet dataset [21] andﬁve base learners for the ensemble models: LIME [7], Guided Backpropagation (GB) [27], Integrated
Revised: We select four samples from ImageNet dataset [26] and ﬁve base learners for the ensemble models: LIME [8], Guided Backpropagation (GB) [32], Integrated

Original: Gradients (IG) [28], Gradient SHAP (GS) [8], and SmoothGrad (SG) [29].
Revised: Gradients (IG) [33], Gradient SHAP (GS) [9], and SmoothGrad (SG) [34].

Original: We compare the proposed
Revised: We compare the proposed

Original: RBM ensemble strategy to simple mean and variance ensembles [17].
Revised: RBM ensemble strategy to simple mean and variance ensembles [22].

Original: The results in Table 1 show thatour approach produces sharp and visually appealing saliency maps in comparison to other ensemblebaselines.
Revised: The results in Table 1 show that our approach produces sharp and visually appealing saliency maps in comparison to other ensemble baselines.

Original: With artiﬁcial noise in the ensemble.
Revised: With artiﬁcial noise in the ensemble.

Original: We challenge the discussed approaches by adding ﬁfteenbaselines with random noise sampled from the standard normal distribution e random ∼ N (0 , 1)to the ensemble.
Revised: We challenge the discussed approaches by adding ﬁfteen baselines with random noise sampled from the standard normal distribution e random ∼ N (0 , 1) to the ensemble.

Original: The results in Table 1 reveal that the proposed RBM-based aggregation methodmitigates noise and hence results in more robust saliency maps in comparison to the other ensemblebaselines.
Revised: The results in Table 1 reveal that the proposed RBM-based aggregation method mitigates noise and hence results in more robust saliency maps in comparison to the other ensemble baselines. [Table] [Table]

Original: 4.2 Pixel Perturbation Experiment
Revised: 4.2 Pixel Perturbation Experiment

Original: In the ﬁrst quantitative experiments, we compare multiple baseline models and ensemble methods onthe CIFAR10 dataset [25] by removing the most important pixels (according to a scoring function)and reporting the area under a curve score (DAUC).
Revised: In the ﬁrst quantitative experiments [Table] we compare multiple baseline models and ensemble methods on the CIFAR10 dataset [35] by removing the most important pixels (according to a scoring function) and reporting the area under a curve score (DAUC).

Original: In addition, we also follow the approach ofinserting the most important pixels into an empty image and again report the area under a curve(IAUC).
Revised: In addition, we also follow the approach of inserting the most important pixels into an empty image and again report the area under a curve (IAUC).

Original: Thus, an ideal feature scoring function has a large IAUC and low DAUC.
Revised: Thus, an ideal feature scoring function has a large IAUC and low DAUC.

Original: These benchmarkmethods well accepted by the research community [30].
Revised: These benchmark methods well accepted by the research community [36].

Original: For this experiment, we select the followingalgorithms as base learners: Gradient SHAP [8], DeepLIFT [2], LIME [7], Saliency maps [31],
Revised: For this experiment, we select the following algorithms as base learners: Gradient SHAP [9], DeepLIFT [3], LIME [8], Saliency maps [37],

Original: SmoothGrad [29], Integrated Gradients [28], Guided Backpropagation [27].
Revised: SmoothGrad [34], Integrated Gradients [33], Guided Backpropagation [32].

Original: As suggested in [10], weadd the original image as a base learner to the ensemble.
Revised: As suggested in [11], we add the original image as a base learner to the ensemble.

Original: According to our conducted experiments,an original image to the ensemble does improve the overall ensemble performance.
Revised: According to our conducted experiments, an original image to the ensemble does improve the overall ensemble performance.

Original: We report allscores for base learners and ensemble methods in Table 2.
Revised: We report all scores for base learners and ensemble methods in Table 2.

Original: 4.3 IROF Experiment
Revised: 4.3 IROF Experiment

Original: In [17] the authors propose the IROF measure as an extension to the work [32].
Revised: In [22] the authors propose the IROF measure as an extension to the work [38].

Original: The main idea of the
Revised: The main idea of the

Original: IROF benchmark is as follows: the image is divided into superpixels using the SLIC algorithm 
Revised: IROF benchmark is as follows: the image is divided into superpixels using the SLIC algorithm [39].

Original: Superpixels are regional blocks of pixels within an image where the contained pixels share a highsimilarity measure among each other.
Revised: Superpixels are regional blocks of pixels within an image where the contained pixels share a high similarity measure among each other.

Original: The relevancy for each superpixel is calculated by averaging[Table] [Table] over the attribution scores over all contained pixels (inside the superpixel).
Revised: The relevancy for each superpixel is calculated by averaging over the attribution scores over all contained pixels (inside the superpixel).

Original: After, the superpixels aresorted descending by their relevancy.
Revised: After, the superpixels are sorted descending by their relevancy.

Original: The entire superpixels are gradually replaced by a baseline andsent through the network again to measure the new recognition quality for the modiﬁed image wrt.
Revised: The entire superpixels are gradually replaced by a baseline and sent through the network again to measure the new recognition quality for the modiﬁed image wrt.

Original: tothe target label.
Revised: to the target label.

Original: For more accurate attribution methods, the recognition quality decreases faster, andthus the area under the curve is lower.
Revised: For more accurate attribution methods, the recognition quality decreases faster, and thus the area under the curve is lower.

Original: The IROF score is deﬁned as the area over the curve (AOC):
Revised: The IROF score is deﬁned as the area over the curve (AOC):

Original: AOC = 1 − AUC .
Revised: AOC = 1 − AUC .

Original: Higher values are therefore indicating a better attribution.
Revised: Higher values are therefore indicating a better attribution.

Original: We use the same baselearners as in the pixel perturbation experiment (Sec. 4.2).
Revised: We use the same base learners as in the pixel perturbation experiment (Sec. 4.2).

Original: The results are listed in Table 2.
Revised: The results are listed in Table 2.

Original: 4.4 An RBM Ensemble within a Singe Explanation Framework
Revised: 4.4 An RBM Ensemble Within a Singe Explanation Framework

Original: In this experiment, we demonstrate that even for multiple base learners of the same type, theunsupervised ensemble with an RBM can be an improvement.
Revised: In this experiment, we demonstrate that even for multiple base learners of the same type, the unsupervised ensemble with an RBM can be an improvement.

Original: To this end, we select the LIME [7]method with a different hyperparameter - the number of superpixels in the image.
Revised: To this end, we select the LIME [8] method with a different hyperparameter - the number of superpixels in the image.

Original: For base learners(LIME-1, LIME-2, and LIME-3), we used 10 , 100 , and 1000 superpixels per image, respectively.
Revised: For base learners (LIME-1, LIME-2, and LIME-3), we used 10 , 100 , and 1000 superpixels per image, respectively.

Original: The results can be seen in 3.
Revised: The results can be seen in Fig.

Original: The main idea is that each lime method has a different granularity level,thus highlighting distinct detail levels, and the proposed method’s aggregation improves the accuracyof feature attributions. 
Revised: The main idea is that each lime method has a different granularity level, thus highlighting distinct detail levels, and the proposed method’s aggregation improves the accuracy of feature attributions. Original LIMELIME- LIME- # of superpixels:

Revised: [Figure]

Original: 4.5 Reproducibility
Revised: 4.5 Reproducibility

Original: For reproducibility reasons, we describe the data pre-processing step used for all the experimentsand provide information about packages used in this work.
Revised: For reproducibility reasons, we describe the data preprocessing step used for all the experiments and provide information about packages used in this work.

Original: Also, the code for every experiment ispublicly available online, the link  in Section 1.
Revised: Also, the code for every experiment is publicly available online (see the link provided in Section 1).

Original: To achieve a fair comparison, the image data from all datasets was preprocessed in the same way foreach baseline.
Revised: To achieve a fair comparison, the image data from all datasets was preprocessed in the same way for each baseline.

Original: We do per saliency map normalization before  aggregation.
Revised: We performed a per saliency map normalization before the aggregation.

Original: In every experiment, weuse the ResNet18 neural network architecture [34], except for the experiment on the MNIST datasetwhere we utilize a simple ﬁve layers convolutional neural network.
Revised: In every experiment, we used the ResNet18 neural network architecture [40], except for the experiment on the MNIST dataset where we utilized a simple ﬁve layers convolutional neural network.

Original: We use a pre-trained model for ImageNet dataset[21] from torchvision library [35].
Revised: We use a pre-trained model for ImageNet dataset[26] from torchvision library [41].

Original: In experiments we utilize the Bernoulli RBM implementation from the scikit-learn library [36] withfollowing hyperparameters for each experiment: for the MNIST dataset we set the batch size to 5 ,the learning rate to 0 .
Revised: For the experiments we used the Bernoulli RBM implementation from the scikit-learn library [42] with following hyperparameters for each experiment: for the MNIST dataset we set the batch size to 5 , the learning rate to 0 .

Original: 01 , and the number of iterations to 
Revised: 01 , and the number of iterations to 100 .

Original:  ImageNet datasets  of  of source
Revised: For CIFAR10 and ImageNet datasets we use the following hyperparameters: a batch size of 35 , a learning rate of 0 . 001 , and a number of iterations is 250 . The rest of hyperparameters are default to the scikit-learn package. For all baselines (base explanation techniques) we use the publicly available open-source implementations from the captum library [43] with their default hyperparameters.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: The results of multiple experiments with the proposed RBM ensemble show its competitive performance compared to base learners and other ensemble techniques.
Revised: The results of multiple experiments with the proposed RBM ensemble show its competitive performance compared to base explanation techniques and other ensemble approaches.

Original: We hypothesize moderateperformance of the RBM ensemble on the insertion (IAUC) benchmark is connected to our datapreparation step since we ﬁlter the negatives values for every saliency map in the ensemble.
Revised: We hypothesize moderate performance of the RBM ensemble on the insertion (IAUC) benchmark is connected to our data preparation step since we ﬁlter the negative values for every saliency map in the ensemble.

Original: The computational complexity of an ensemble method primarily depends on the base learner.
Revised: The computational complexity of an ensemble method primarily depends on the base learner.

Original: In ourcase, the base learners are relatively fast, especially on desiccated hardware (GPU orTPU),
Revised: In our case, the base explanation techniques are relatively fast, especially on specialized hardware (GPU or

Original:  where an RBM has low computational complexity.
Revised: TPU), where an RBM has low computational complexity.

Original: The saliency map generation is a fast process,

Original: The gradient-based methods frequently produce noisy explanations.
Revised: The gradient-based methods frequently produce noisy explanations.

Original: We empirically demonstrate thatour approach reduces the noisiness in the ﬁnal ensemble (Tab. 1).
Revised: We empirically demonstrated that our approach reduces the noise in the ﬁnal ensemble (Tab. 1).

Original: Therefore, we believe that the
Revised: Therefore, we believe that the

Original: RBM aggregation of multiple saliency maps from gradient-based feature attribution methods willreduce the noisiness, and as a consequence, improve the overall correctness.
Revised: RBM aggregation of multiple saliency maps from gradient-based feature attributions is a powerful tool for improving the overall reliability of local explanations.

Original: For future steps, we would want to make certain that ensemble techniques improve this domain. Besides that, we would like to repeat our experiments with a larger sample size.
Revised: As part of our future work, we aim to evaluate our aggregation approach on larger datasets.

Original: Future research onbase learner selection is needed for  a more efﬁcient ensemble is needed. In addition, several otherunsupervised ensemble methods can be applied to this problem.
Revised: Furthermore, methods for selecting a few quite reliable base explanations for aggregation might lead to efﬁcient explanations ensembles for larger datasets. Finally we expect that the proposed approach can be easily adapted to handle local explanations over structured tabular data, where the explanation of deep neural networks is an essential task for many crucial applications such as healthcare and ﬁnance [44]. [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we presented a novel approach to unsupervised aggregation of feature-based explanationsusing Restricted Boltzmann Machines with the aim of accurately interpreting the inﬂuence of inputson the output of deep neural networks.
Revised: In this work, we presented a novel approach to unsupervised aggregation of feature-based explanations using Restricted Boltzmann Machines with the aim of reliably interpreting the inﬂuence of inputs on the output of deep neural networks.

Original: In addition to explanatory reasons, the latter is also essentialfor debugging and diagnostic purposes and serves the long-term acceptance of deep learning in 
Revised: In addition to explanatory reasons, the latter is also essential for debugging and diagnostic purposes and serves the long-term acceptance of deep learning in real-world applications.

Original: Using the proposed approach, we demonstrate with visual and quantitative experiments its ability toobtain more robust and accurate explanations than other existing ensemble methods.
Revised: Using the proposed approach we demonstrated through visual and quantitative experiments its ability to obtain more robust and reliable explanations than other existing ensemble methods.

Original: In a setting withnoisy attribution maps in an ensemble, the proposed approach successfully selects only the valuableinformation, mitigating the randomness.
Revised: In a setting with noisy attribution maps in an ensemble, the proposed approach successfully selects only the valuable information, mitigating noise.

Original: Moreover, our work illuminates and mitigates the problem ofpossible contradictory results that may be obtained by different explanation and evaluation methods.
Revised: Moreover, our work illuminates and mitigates the problem of possible contradictory results that may be obtained by different explanation and evaluation methods.

Original: Finally, we show that our approach can also be applied within a single interpretability framework toreduce the sensitivity of a feature-based explanatory approach to its hyperparameters
Revised: Finally, we note that our approach can also be used within a single interpretability framework to reduce the sensitivity of a feature-based explanatory approach to its hyperparameters. A Additional Experiments

Revised: Table 3 presents extended experimental results for the compression with or without noisy feature attribution maps in the ensemble