Original: A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines
Revised: A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines

Original: Abstract
Revised: Abstract

Original: Understanding the results of deep neural networks is an essential step towards wider acceptance of deep learning algorithms.
Revised: Understanding the results of deep neural networks is an essential step towards wider acceptance of deep learning algorithms.

Original: Many approaches address the issue of interpreting artiﬁcial neural networks, but often provide divergent explanations.
Revised: Many approaches address the issue of interpreting artiﬁcial neural networks, but often provide divergent explanations.

Original: Moreover, different hyperparameters of an explanatory method can lead to conﬂicting interpretations.
Revised: Moreover, different hyperparameters of an explanatory method can lead to conﬂicting interpretations.

Original: In this paper, we propose a technique for aggregating the feature attributions of different explanatory algorithms using Restricted Boltzmann
Revised: In this paper, we propose a technique for aggregating the feature attributions of different explanatory algorithms using Restricted Boltzmann

Original: Machines (RBMs) to achieve a more reliable and robust interpretation of deep neural networks.
Revised: Machines (RBMs) to achieve a more reliable and robust interpretation of deep neural networks.

Original: Several challenging experiments on real-world datasets show that the proposed RBM method outperforms popular feature attribution methods and basic ensemble techniques
Revised: Several challenging experiments on real-world datasets show that the proposed RBM method outperforms popular feature attribution methods and basic ensemble techniques

Original:  Introduction
Revised:  Introduction

Original: As the applications of deep neural networks (DNNs) continue to grow, the black-box nature of DNNs creates potential trust issues [1].
Revised: As the applications of deep neural networks (DNNs) continue to grow, the black-box nature of DNNs creates potential trust issues [1].

Original: Moreover, numerous life-critical (such as medical, automotive, or ﬁnancial) applications utilize DNNs for various estimation tasks.
Revised: Moreover, numerous life-critical (such as medical, automotive, or ﬁnancial) applications utilize DNNs for various estimation tasks.

Original: In such applications, and especially for the long-term acceptance of artiﬁcial intelligence (AI) solutions, a deeper understanding and trust in the produced results is crucial.
Revised: In such applications, and especially for the long-term acceptance of artiﬁcial intelligence (AI) solutions, a deeper understanding and trust in the produced results is crucial.

Original: Furthermore, feature attribution methods are important tools for deep model debugging and diagnosis [2].
Revised: Furthermore, feature attribution methods are important tools for deep model debugging and diagnosis [2].

Original: Explaining how the input inﬂuences the output for a given DNN is one form to interpret the black-box nature of the DNN and bring trust to a system.
Revised: Explaining how the input inﬂuences the output for a given DNN is one form to interpret the black-box nature of the DNN and bring trust to a system.

Original: These so-called feature-based explanation methods received a lot of attention in recent years [1, 3–5].
Revised: These so-called feature-based explanation methods received a lot of attention in recent years [1, 3–5].

Original: They can be grouped into three broad categories, (1) approaches based on gradient information [6, 7], (2) perturbation-based approaches [8, 9], and (3) attribution-based approaches [3, 10].
Revised: They can be grouped into three broad categories, (1) approaches based on gradient information [6, 7], (2) perturbation-based approaches [8, 9], and (3) attribution-based approaches [3, 10].

Original: Interestingly, different feature-based explanation approaches regularly produce mixed views on the main attributes (areas of an image or variables), and in the absence of the ground truth, it is still a challenge to verify which explanation method is the most trustworthy.
Revised: Interestingly, different feature-based explanation approaches regularly produce mixed views on the main attributes (areas of an image or variables), and in the absence of the ground truth, it is still a challenge to verify which explanation method is the most trustworthy.

Original: Moreover, in the AI community, there are no yet accepted quality measures for featurebased explanations.
Revised: Moreover, in the AI community, there are no yet accepted quality measures for featurebased explanations.

Original: All these difﬁculties resulted in a large number of different explanation methods and in a lack of consensus on which techniques are most reliable.
Revised: All these difﬁculties resulted in a large number of different explanation methods and in a lack of consensus on which techniques are most reliable.

Original: Within the machine learning (ML) community, there is much work on the combination of methods that do not always agree with each other, i.e. ensemble learning [11, 12].
Revised: Within the machine learning (ML) community, there is much work on the combination of methods that do not always agree with each other, i.e. ensemble learning [11, 12].

Original: Normally ensemble models outperform the non-ensemble models and turn out to be more robust to outliers.
Revised: Normally ensemble models outperform the non-ensemble models and turn out to be more robust to outliers.

Original: The main idea is that if multiple methods make mistakes in different areas, combining them in an intelligent way improves performance and reduces the effect of outliers as compared to the single method.
Revised: The main idea is that if multiple methods make mistakes in different areas, combining them in an intelligent way improves performance and reduces the effect of outliers as compared to the single method.

Original: Moreover, from statistical learning theory and practical applications, it is well understood that ensemble learning is the path of choice towards a more robust machine learning system [13], even in unsupervised learning scenarios where the target is not available [14, 15].
Revised: Moreover, from statistical learning theory and practical applications, it is well understood that ensemble learning is the path of choice towards a more robust machine learning system [13], even in unsupervised learning scenarios where the target is not available [14, 15].

Original: In this work, utilizing ideas from [15, 11] and [16, 17], we introduce a novel approach for the unsupervised ensemble learning of reliable and robust feature-based explanations for deep neural networks.
Revised: In this work, utilizing ideas from [15, 11] and [16, 17], we introduce a novel approach for the unsupervised ensemble learning of reliable and robust feature-based explanations for deep neural networks.

Original: To this end, we propose using a model based on Restricted Boltzmann Machines (RBMs), which achieves this goal by aggregating the results (saliency maps) of different feature-based explanation methods in a principled probabilistic fashion.
Revised: To this end, we propose using a model based on Restricted Boltzmann Machines (RBMs), which achieves this goal by aggregating the results (saliency maps) of different feature-based explanation methods in a principled probabilistic fashion.

Original: Also, it has been shown that an RBM can be used in the truth discovery setting [18, 19], which is analogous to our task of ﬁnding a reliable feature importance map from different importance maps.
Revised: Also, it has been shown that an RBM can be used in the truth discovery setting [18, 19], which is analogous to our task of ﬁnding a reliable feature importance map from different importance maps.

Original: The main contributions of this work are:
Revised: The main contributions of this work are:

Original: • We introduce a novel method for a robust and reliable feature-based explanation using ensemble learning.
Revised: • We introduce a novel method for a robust and reliable feature-based explanation using ensemble learning.

Original: • We empirically and visually show the superior performance of the proposed method in comparison to state-of-the-art feature attribution baselines.
Revised: • We empirically and visually show the superior performance of the proposed method in comparison to state-of-the-art feature attribution baselines.

Original: • We open-source our code and make it publicly available, as an RBM ensemble framework: (https://github.com/JohanvandenHeuvel/AggregationOfLocalExplanations), Besides, we also developed a single Python package with various evaluation metrics for feature attribution methods metrics: https://github.com/meier-johannes94/ExplainableAIImageMeasures
Revised: • We open-source our code and make it publicly available, as an RBM ensemble framework: (https://github.com/JohanvandenHeuvel/AggregationOfLocalExplanations), Besides, we also developed a single Python package with various evaluation metrics for feature attribution methods metrics: https://github.com/meier-johannes94/ExplainableAIImageMeasures

Original: The paper is organized as follows:
Revised: The paper is organized as follows:

Original: In Section 2 we discuss the related work and provide essential background information.
Revised: In Section 2 we discuss the related work and provide essential background information.

Original: Section 3 presents the proposed ensemble method for local feature-based explanations using an RBM.
Revised: Section 3 presents the proposed ensemble method for local feature-based explanations using an RBM.

Original: In Section 4, we present the results of various experiments.
Revised: In Section 4, we present the results of various experiments.

Original: Sectiondiscusses limitations of our work and ways to address them in the future.
Revised: Sectiondiscusses limitations of our work and ways to address them in the future.

Original: Section 6 concludes our work with a short summary.
Revised: Section 6 concludes our work with a short summary.

Original:  Background and Related Work
Revised:  Background and Related Work

Original: This part of the manuscript provides the needed background and discusses related approaches.
Revised: This part of the manuscript provides the needed background and discusses related approaches.

Original: First, we present the basic notation used in this work and proceed by presenting two ensemble techniques for aggregating feature importance maps.
Revised: First, we present the basic notation used in this work and proceed by presenting two ensemble techniques for aggregating feature importance maps.

Original: 2.1 Feature Attribution Function
Revised: 2.1 Feature Attribution Function

Original: Formally, a feature attribution function can be seen as φ ( f, x , c x ) , where f is a black box model and x is an input data point from a corresponding class c x .
Revised: Formally, a feature attribution function can be seen as φ ( f, x , c x ) , where f is a black box model and x is an input data point from a corresponding class c x .

Original: The output of φ is an explanation vector or matrix
Revised: The output of φ is an explanation vector or matrix

Original: e f ( x ) , where each element of e f ( x ) is an importance score for the corresponding feature value in x .
Revised: e f ( x ) , where each element of e f ( x ) is an importance score for the corresponding feature value in x .

Original: A large positive or negative value in e f ( x ) indicates that the corresponding feature (pixel) has a large inﬂuence on the outcome of the black-box model f .
Revised: A large positive or negative value in e f ( x ) indicates that the corresponding feature (pixel) has a large inﬂuence on the outcome of the black-box model f .

Original: Assumption 2.

Revised: Assumption 2.

Original: In the following, we assume that a true feature attribution ¯ e f ( x ) for a given model f and input x exists and can be constructed by adequately aggregating available attributions
Revised: In the following, we assume that a true feature attribution ¯ e f ( x ) for a given model f and input x exists and can be constructed by adequately aggregating available attributions

Original: e f ( x ) ,i , i ∈ { 1 , ..., N } , where N is the number of baseline explanations (from N baseline methods).
Revised: e f ( x ) ,i , i ∈ { 1 , ..., N } , where N is the number of baseline explanations (from N baseline methods).

Original: For better readability and simplicity, from here we omit the index f ( x ) .
Revised: For better readability and simplicity, from here we omit the index f ( x ) .

Original: The goal of any explanation method φ is to obtain an attribution e that is as close as possible to ¯ e .
Revised: The goal of any explanation method φ is to obtain an attribution e that is as close as possible to ¯ e .

Original: Note that our method naturally generalizes to probabilistic local explanation methods [20].
Revised: Note that our method naturally generalizes to probabilistic local explanation methods [20].

Original: Given the before-mentioned assumption, we can say that there is a joint probability distribution of the pair ( e , ¯ e ) parametrized by θ .
Revised: Given the before-mentioned assumption, we can say that there is a joint probability distribution of the pair ( e , ¯ e ) parametrized by θ .

Original: [Equation]
Revised: [Equation]

Original: The joint distribution p θ ( e , ¯ e ) is not known, and neither are the marginals p θ ( e ) , p θ (¯ e ) .
Revised: The joint distribution p θ ( e , ¯ e ) is not known, and neither are the marginals p θ ( e ) , p θ (¯ e ) .

Original: For the following theoretical results we require that the explanation methods give independent explanations when conditioned on the true explanation.
Revised: For the following theoretical results we require that the explanation methods give independent explanations when conditioned on the true explanation.

Original: However, as with Naive Bayes methods, for practical purposes, this assumption can be violated without negatively impacting the aggregation quality [15].
Revised: However, as with Naive Bayes methods, for practical purposes, this assumption can be violated without negatively impacting the aggregation quality [15].

Original: Also note that we do assume some consistency between the explanations, following the assumption that feature attributions reﬂect the underlying (but unknown) importance distributions of the feature values [21].
Revised: Also note that we do assume some consistency between the explanations, following the assumption that feature attributions reﬂect the underlying (but unknown) importance distributions of the feature values [21].

Original: Assuming conditional independence between the provided baseline explanations given the (unknown) true explanation, we have
Revised: Assuming conditional independence between the provided baseline explanations given the (unknown) true explanation, we have

Original: [Equation]
Revised: [Equation]

Original: where e n is a baseline explanation in the ensemble involving N different baseline explanations.
Revised: where e n is a baseline explanation in the ensemble involving N different baseline explanations.

Original: 2.2 Ensemble Learning
Revised: 2.2 Ensemble Learning

Original: As we state in the introduction, ensemble learning is a well-studied approach for improving the performance of an ML system.
Revised: As we state in the introduction, ensemble learning is a well-studied approach for improving the performance of an ML system.

Original: One of the most basic ensemble methods employs the mean of results of base learners [13], where a base learner is a single algorithm from the ensemble.
Revised: One of the most basic ensemble methods employs the mean of results of base learners [13], where a base learner is a single algorithm from the ensemble.

Original: [Equation]
Revised: [Equation]

Original: A signiﬁcant drawback of the mean ensemble approach is that it still is sensitive to outliers or noisy estimations.
Revised: A signiﬁcant drawback of the mean ensemble approach is that it still is sensitive to outliers or noisy estimations.

Original: Furthermore, data scaling may strongly inﬂuence the aggregation.
Revised: Furthermore, data scaling may strongly inﬂuence the aggregation.

Original: In Section 4, these weaknesses of the mean ensemble approach are also seen in the experimental evaluation.
Revised: In Section 4, these weaknesses of the mean ensemble approach are also seen in the experimental evaluation.

Original: To mitigate these weaknesses, the authors of [22] propose to take the local uncertainty into account.
Revised: To mitigate these weaknesses, the authors of [22] propose to take the local uncertainty into account.

Original: To this end, they divide the mean by the local variance plus a constant (cid:15) for stability reasons, which results in the variance ensemble approach :
Revised: To this end, they divide the mean by the local variance plus a constant (cid:15) for stability reasons, which results in the variance ensemble approach :

Original: [Equation]
Revised: [Equation]

Original: where σ ∗ ( e i ∈{ 1 ,...,N } ) is the point-wise standard deviation over all the available explanations
Revised: where σ ∗ ( e i ∈{ 1 ,...,N } ) is the point-wise standard deviation over all the available explanations

Original: e i , i ∈ { 1 , ..., N } .
Revised: e i , i ∈ { 1 , ..., N } .

Original: This method assigns less relevance to explanations that have high disagreement with the remaining explanations.
Revised: This method assigns less relevance to explanations that have high disagreement with the remaining explanations.

Original: Also, the authors of [23] proposed a novel method to aggregate Shapley values through an explanation function that minimizes sensitivity.
Revised: Also, the authors of [23] proposed a novel method to aggregate Shapley values through an explanation function that minimizes sensitivity.

Original:  Ensemble Learning using Restricted Boltzmann Machines
Revised:  Ensemble Learning using Restricted Boltzmann Machines

Original: [Equation]
Revised: [Equation]

Original: three visible and two hidden units.
Revised: three visible and two hidden units.

Original: In our work, we use an RBM with a single hidden node.
Revised: In our work, we use an RBM with a single hidden node.

Original: In this section, we present an unsupervised aggregation of feature attribution maps using a Restricted Boltzmann
Revised: In this section, we present an unsupervised aggregation of feature attribution maps using a Restricted Boltzmann

Original: Machine (RBM).
Revised: Machine (RBM).

Original: Similar aggregation techniques have been proposed in other contexts, e.g., in [15, 18].
Revised: Similar aggregation techniques have been proposed in other contexts, e.g., in [15, 18].

Original: 3.1 The Restricted Boltzmann Machine
Revised: 3.1 The Restricted Boltzmann Machine

Original: An RBM is an undirected bipartite graph that can be parametrized by a neural network.
Revised: An RBM is an undirected bipartite graph that can be parametrized by a neural network.

Original: It is a variant of the
Revised: It is a variant of the

Original: Boltzmann Machine, with the additional property that there are no connections within both the group of visible nodes or the group of hidden nodes.
Revised: Boltzmann Machine, with the additional property that there are no connections within both the group of visible nodes or the group of hidden nodes.

Original: The advantage of this property is that nodes in one group are conditionally independent of each other, given that we know the state of the nodes in the other group.
Revised: The advantage of this property is that nodes in one group are conditionally independent of each other, given that we know the state of the nodes in the other group.

Original: One of the main characteristics of an RBM is that it can learn a probability distribution over its set of inputs.
Revised: One of the main characteristics of an RBM is that it can learn a probability distribution over its set of inputs.

Original: A graphical representation of an example RBM is shown in Figure 1.
Revised: A graphical representation of an example RBM is shown in Figure 1.

Original: The formal deﬁnition of an RBM is as follows.
Revised: The formal deﬁnition of an RBM is as follows.

Original: There is a set X of n visible binary random variables and a set H of m hidden binary random variables.
Revised: There is a set X of n visible binary random variables and a set H of m hidden binary random variables.

Original: The RBM has parameters
Revised: The RBM has parameters

Original: λ =
Revised: λ =

Original: ( W , a , b ) .
Revised: ( W , a , b ) .

Original: W is the weight matrix of the connections between the nodes, a is the bias of the visible layer and b is the bias of the hidden layer.
Revised: W is the weight matrix of the connections between the nodes, a is the bias of the visible layer and b is the bias of the hidden layer.

Original: Each possible state of the RBM, i.e. the particular values of ( X, H ) , is associated with the following energy function (in matrix notation):
Revised: Each possible state of the RBM, i.e. the particular values of ( X, H ) , is associated with the following energy function (in matrix notation):

Original: [Equation]
Revised: [Equation]

Original: which then can also be used to deﬁne the joint probability distribution for the visible and hidden vectors is deﬁned in terms of the energy function:
Revised: which then can also be used to deﬁne the joint probability distribution for the visible and hidden vectors is deﬁned in terms of the energy function:

Original: [Equation]
Revised: [Equation]

Original: where Z is the sum over e − E ( x , h ) for all possible conﬁgurations x , h , which can be seen as a normalization constant to ensure that all probabilities sum to 1 , also known as the partition function.
Revised: where Z is the sum over e − E ( x , h ) for all possible conﬁgurations x , h , which can be seen as a normalization constant to ensure that all probabilities sum to 1 , also known as the partition function.

Original: The optimization objective of the RBM is to maximize the expected log probability of a training sample x :
Revised: The optimization objective of the RBM is to maximize the expected log probability of a training sample x :

Original: [Equation]
Revised: [Equation]

Original: To train an RBM, a gradient-based optimization can be applied using the contrastive divergence algorithm [24, 25].
Revised: To train an RBM, a gradient-based optimization can be applied using the contrastive divergence algorithm [24, 25].

Original: 3.2 Aggregation of Local Explanations using an RBM
Revised: 3.2 Aggregation of Local Explanations using an RBM

Original: Given an RBM with N visible nodes and one hidden node, with input x , where N is the number of baseline explanations in our ensemble, it can be shown that the true posterior probability of y can be efﬁciently estimated (Lemma 4.1, Lemma 4.2 from [15]).
Revised: Given an RBM with N visible nodes and one hidden node, with input x , where N is the number of baseline explanations in our ensemble, it can be shown that the true posterior probability of y can be efﬁciently estimated (Lemma 4.1, Lemma 4.2 from [15]).

Original: Furthermore, given the previously discussed mild assumptions on the input data (which are in line with those in [15]), the maximum likelihood estimate ¯ λ MLE for the parameters of the RBM, the RBM posterior probability P ¯ λ MLE ( H = 1 | X = x ) converges to true posterior P θ ( Y = 1 | X = x ) .
Revised: Furthermore, given the previously discussed mild assumptions on the input data (which are in line with those in [15]), the maximum likelihood estimate ¯ λ MLE for the parameters of the RBM, the RBM posterior probability P ¯ λ MLE ( H = 1 | X = x ) converges to true posterior P θ ( Y = 1 | X = x ) .

Original: Hence, we are able to apply the RBM to the unsupervised aggregation of N available feature-based explanations.
Revised: Hence, we are able to apply the RBM to the unsupervised aggregation of N available feature-based explanations.

Original: We assume a joint distribution p θ ( e , ¯ e ) , and that the e i ’s are conditionally independent from each other given ¯ e .
Revised: We assume a joint distribution p θ ( e , ¯ e ) , and that the e i ’s are conditionally independent from each other given ¯ e .

Original: By ﬁtting the RBM we learn the parameters θ and thus obtain the relationship between our known explanations e i and the true explanation ¯ e .
Revised: By ﬁtting the RBM we learn the parameters θ and thus obtain the relationship between our known explanations e i and the true explanation ¯ e .

Original: The ensemble pipeline of the proposed method is depicted in Fig.
Revised: The ensemble pipeline of the proposed method is depicted in Fig.

Original: In order to preserve the spatial information for visual data using the RBM-based ensemble, we do a pixel-wise aggregation.
Revised: In order to preserve the spatial information for visual data using the RBM-based ensemble, we do a pixel-wise aggregation.

Original: Therefore, for each pixel we train a Bernoulli RBM with a single hidden unit.
Revised: Therefore, for each pixel we train a Bernoulli RBM with a single hidden unit.

Original: A known limitation of an RBM is the so-called ﬂipping issue [15, 18, 19], which arises from the
Revised: A known limitation of an RBM is the so-called ﬂipping issue [15, 18, 19], which arises from the

Original: RBM parametrization symmetry.
Revised: RBM parametrization symmetry.

Original: That is, the weights of the RBM can be ﬂipped symmetrically without changing the behavior of the RBM.
Revised: That is, the weights of the RBM can be ﬂipped symmetrically without changing the behavior of the RBM.

Original: In order to avoid this unwanted effect, we propose two approaches: ﬂip detection and metric optimization.
Revised: In order to avoid this unwanted effect, we propose two approaches: ﬂip detection and metric optimization.

Original: The ﬂip detection algorithm extends the idea from
Revised: The ﬂip detection algorithm extends the idea from

Original: Remark 4.3 in [15], by comparing the top 5 % of most important and 5 % of less important pixels to the mean baseline.
Revised: Remark 4.3 in [15], by comparing the top 5 % of most important and 5 % of less important pixels to the mean baseline.

Original: The algorithm inverts the current important scores if there is a strong disagreement between the proposed approach and the mean baseline.
Revised: The algorithm inverts the current important scores if there is a strong disagreement between the proposed approach and the mean baseline.

Original: The metric optimization method utilizes the chosen metric to overcome the ﬂipping issue.
Revised: The metric optimization method utilizes the chosen metric to overcome the ﬂipping issue.

Original: It compares two versions of the RBM ensemble results and selects the one with a better performance according to the selected metric.
Revised: It compares two versions of the RBM ensemble results and selects the one with a better performance according to the selected metric.

Original:  Experiments
Revised:  Experiments

Original: To demonstrate the effectiveness of the proposed ensemble algorithm we conduct various visual and quantitative experiments.
Revised: To demonstrate the effectiveness of the proposed ensemble algorithm we conduct various visual and quantitative experiments.

Original: First, we present the visual inspection results on the MNIST [27] and
Revised: First, we present the visual inspection results on the MNIST [27] and

Original: ImageNet [26] datasets in two settings, with and without noisy explanation maps in our ensemble.
Revised: ImageNet [26] datasets in two settings, with and without noisy explanation maps in our ensemble.

Original: Despite a growing body of research focusing on explainable ML, the fair quantitative comparison of local explanation (or saliency-based) algorithms is still an open question, since the existing methods mostly utilize the pixel perturbation strategy (e.g., removing the most or least important pixels and reporting the change in recognition quality) [28, 29].
Revised: Despite a growing body of research focusing on explainable ML, the fair quantitative comparison of local explanation (or saliency-based) algorithms is still an open question, since the existing methods mostly utilize the pixel perturbation strategy (e.g., removing the most or least important pixels and reporting the change in recognition quality) [28, 29].

Original: Also, such evaluations have a signiﬁcant drawback, replacing image pixels with black or "mean" or any other pixel values may lead to artifacts affecting the data distribution [29, 30].
Revised: Also, such evaluations have a signiﬁcant drawback, replacing image pixels with black or "mean" or any other pixel values may lead to artifacts affecting the data distribution [29, 30].

Original: Nevertheless, since pixel perturbation analyses are employed in many related works, for our quantitative analysis we select the following approaches: the pixel perturbation for insertion (IAUC) and deletion (DAUC).
Revised: Nevertheless, since pixel perturbation analyses are employed in many related works, for our quantitative analysis we select the following approaches: the pixel perturbation for insertion (IAUC) and deletion (DAUC).

Original: Furthermore, we utilize the iterative removal of features (IROF) analysis [31].
Revised: Furthermore, we utilize the iterative removal of features (IROF) analysis [31].

Original: We explain each evaluation method in detail in the corresponding subsections.
Revised: We explain each evaluation method in detail in the corresponding subsections.

Original: In our last experiment, we demonstrate that our ensemble approach can be also used within a singe feature attribution framework to achieve more robust and stable explanations.
Revised: In our last experiment, we demonstrate that our ensemble approach can be also used within a singe feature attribution framework to achieve more robust and stable explanations.

Original: Since, it has been shown that hyperparameters choice can signiﬁcantly affect the saliency maps [32].
Revised: Since, it has been shown that hyperparameters choice can signiﬁcantly affect the saliency maps [32].

Original: 4.1 Visual Inspection for Image Data
Revised: 4.1 Visual Inspection for Image Data

Original: In our ﬁrst experiment, a visual evaluation on images from ImageNet [26] and MNIST [27] is performed for several baseline and ensemble methods.
Revised: In our ﬁrst experiment, a visual evaluation on images from ImageNet [26] and MNIST [27] is performed for several baseline and ensemble methods.

Original: We provide benchmark outcomes for two settings, with and without ﬁfteen noisy baseline explanations in an ensemble.
Revised: We provide benchmark outcomes for two settings, with and without ﬁfteen noisy baseline explanations in an ensemble.

Original: The results are depicted in Table 1.
Revised: The results are depicted in Table 1.

Original: Without artiﬁcial noise in the ensemble.
Revised: Without artiﬁcial noise in the ensemble.

Original: We select four samples from ImageNet dataset [26] and ﬁve baseline explanation methods for the ensemble models: LIME [8], Guided Backpropagation (GB) [33], Integrated Gradients (IG) [34], Gradient SHAP (GS) [9], and SmoothGrad (SG) [35].
Revised: We select four samples from ImageNet dataset [26] and ﬁve baseline explanation methods for the ensemble models: LIME [8], Guided Backpropagation (GB) [33], Integrated Gradients (IG) [34], Gradient SHAP (GS) [9], and SmoothGrad (SG) [35].

Original: We compare the proposed RBM ensemble strategy to simple mean and variance ensembles [22].
Revised: We compare the proposed RBM ensemble strategy to simple mean and variance ensembles [22].

Original: The results in Table 1 show that our approach produces sharp and visually appealing saliency maps in comparison to other ensemble baselines.
Revised: The results in Table 1 show that our approach produces sharp and visually appealing saliency maps in comparison to other ensemble baselines.

Original: In comparison to the baseline explanation methods, the proposed ensemble technique seems to produce more reliable and robust results by highlighting commonalities among the baseline methods and by mitigating the noise coming from the single baseline methods.
Revised: In comparison to the baseline explanation methods, the proposed ensemble technique seems to produce more reliable and robust results by highlighting commonalities among the baseline methods and by mitigating the noise coming from the single baseline methods.

Original: With artiﬁcial noise in the ensemble.
Revised: With artiﬁcial noise in the ensemble.

Original: We challenge the discussed approaches by adding ﬁfteen baselines with random noise sampled from the standard normal distribution e rand ∼ N (0 , 1) to the ensemble.
Revised: We challenge the discussed approaches by adding ﬁfteen baselines with random noise sampled from the standard normal distribution e rand ∼ N (0 , 1) to the ensemble.

Original: The results in Table 1 reveal that the proposed RBM-based aggregation method mitigates noise and hence results in more robust saliency maps in comparison to the other ensemble baselines.
Revised: The results in Table 1 reveal that the proposed RBM-based aggregation method mitigates noise and hence results in more robust saliency maps in comparison to the other ensemble baselines.

Original: [Table]
Revised: [Table]

Original: noisy feature
Revised: noisy feature

Original: [Table]
Revised: [Table]

Original: noisy feature attribution maps
Revised: noisy feature attribution maps

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4.2 Pixel Perturbation Experiment
Revised: 4.2 Pixel Perturbation Experiment

Original: In the ﬁrst quantitative experiments
Revised: In the ﬁrst quantitative experiments

Original: [Table]
Revised: [Table]

Original: we compare multiple baseline models and ensemble methods on the CIFAR10 dataset [36] by removing the most important pixels (according to a scoring function) and reporting the area under a curve score (DAUC).
Revised: we compare multiple baseline models and ensemble methods on the CIFAR10 dataset [36] by removing the most important pixels (according to a scoring function) and reporting the area under a curve score (DAUC).

Original: In addition, we also follow the approach of inserting the most important pixels into an empty image and again report the area under a curve (IAUC).
Revised: In addition, we also follow the approach of inserting the most important pixels into an empty image and again report the area under a curve (IAUC).

Original: Thus, an ideal feature scoring function has a large IAUC and low DAUC.
Revised: Thus, an ideal feature scoring function has a large IAUC and low DAUC.

Original: These benchmark methods well accepted by the research community [37].
Revised: These benchmark methods well accepted by the research community [37].

Original: For this experiment, we select the following algorithms as baseline explanation methods: Gradient SHAP [9], DeepLIFT [3], LIME [8], Saliency maps [38],
Revised: For this experiment, we select the following algorithms as baseline explanation methods: Gradient SHAP [9], DeepLIFT [3], LIME [8], Saliency maps [38],

Original: SmoothGrad [35], Integrated Gradients [34], Guided Backpropagation [33].
Revised: SmoothGrad [35], Integrated Gradients [34], Guided Backpropagation [33].

Original: As suggested in [11], we add the original image as a baseline to the ensemble.
Revised: As suggested in [11], we add the original image as a baseline to the ensemble.

Original: However, according to our experiments, adding th original image to the ensemble does improve the overall ensemble performance.
Revised: However, according to our experiments, adding th original image to the ensemble does improve the overall ensemble performance.

Original: We report all scores for the baseline approaches and the ensemble methods in Table 2.
Revised: We report all scores for the baseline approaches and the ensemble methods in Table 2.

Original: 4.3 IROF Experiment
Revised: 4.3 IROF Experiment

Original: In [22] the authors propose the IROF measure as an extension to the work [39].
Revised: In [22] the authors propose the IROF measure as an extension to the work [39].

Original: The main idea of the
Revised: The main idea of the

Original: IROF benchmark is as follows: the image is divided into superpixels using the SLIC algorithm [40].
Revised: IROF benchmark is as follows: the image is divided into superpixels using the SLIC algorithm [40].

Original: Superpixels are regional blocks of pixels within an image where the contained pixels share a high similarity measure among each other.
Revised: Superpixels are regional blocks of pixels within an image where the contained pixels share a high similarity measure among each other.

Original: The relevancy for each superpixel is calculated by averaging over the attribution scores over all contained pixels (inside the superpixel).
Revised: The relevancy for each superpixel is calculated by averaging over the attribution scores over all contained pixels (inside the superpixel).

Original: After, the superpixels are sorted descending by their relevancy.
Revised: After, the superpixels are sorted descending by their relevancy.

Original: The entire superpixels are gradually replaced by a baseline and sent through the network again to measure the new recognition quality for the modiﬁed image wrt.
Revised: The entire superpixels are gradually replaced by a baseline and sent through the network again to measure the new recognition quality for the modiﬁed image wrt.

Original: to the target label.
Revised: to the target label.

Original: For more accurate attribution methods, the recognition quality decreases faster, and thus the area under the curve is lower.
Revised: For more accurate attribution methods, the recognition quality decreases faster, and thus the area under the curve is lower.

Original: The IROF score is deﬁned as the area over the curve (AOC):
Revised: The IROF score is deﬁned as the area over the curve (AOC):

Original: AOC = 1 − AUC .
Revised: AOC = 1 − AUC .

Original: Higher values, therefore, indicate a better attribution.
Revised: Higher values, therefore, indicate a better attribution.

Original: We use the same baseline methods as in the pixel perturbation experiment (Sec. 4.2).
Revised: We use the same baseline methods as in the pixel perturbation experiment (Sec. 4.2).

Original: The results are listed in Table 2.
Revised: The results are listed in Table 2.

Original: 4.4 An RBM Ensemble Within a Singe Explanation Framework
Revised: 4.4 An RBM Ensemble Within a Singe Explanation Framework

Original: In this experiment, we demonstrate that even for multiple baseline explanations of the same explanation method, the unsupervised ensemble with an RBM can lead to an improvement.
Revised: In this experiment, we demonstrate that even for multiple baseline explanations of the same explanation method, the unsupervised ensemble with an RBM can lead to an improvement.

Original: To this end, we select the LIME [8] method with a different hyperparameter - the number of superpixels in the image.
Revised: To this end, we select the LIME [8] method with a different hyperparameter - the number of superpixels in the image.

Original: For the baselines (LIME-0, LIME-1, and LIME-2), we used 10 , 100 , and 1000 superpixels per image, respectively.
Revised: For the baselines (LIME-0, LIME-1, and LIME-2), we used 10 , 100 , and 1000 superpixels per image, respectively.

Original: The results can be seen in Fig.
Revised: The results can be seen in Fig.

Original: The main idea is that each lime method has a different granularity level, thus highlighting distinct detail levels, and the proposed method’s aggregation may help improve the reliability and robustness of feature attributions.
Revised: The main idea is that each lime method has a different granularity level, thus highlighting distinct detail levels, and the proposed method’s aggregation may help improve the reliability and robustness of feature attributions.

Original: Original
Revised: Original

Original: LIMELIME-
Revised: LIMELIME-

Original: LIME- # of superpixels:[Figure]
Revised: LIMERBM-ensemble # of superpixels:

Original: 4.5 Reproducibility
Revised: 4.5 Reproducibility

Original: For reproducibility reasons, we describe the data preprocessing step used for all the experiments and provide information about packages used in this work.
Revised: For reproducibility reasons, we describe the data preprocessing step used for all the experiments and provide information about packages used in this work.

Original: Also, the code for every experiment is publicly available online (see the links provided in Section 1).
Revised: Also, the code for every experiment is publicly available online (see the links provided in Section 1).

Original: To achieve a fair comparison, the image data from all datasets was preprocessed in the same way for each baseline.
Revised: To achieve a fair comparison, the image data from all datasets was preprocessed in the same way for each baseline.

Original: We performed a per saliency map normalization before the aggregation.
Revised: We performed a per saliency map normalization before the aggregation.

Original: In every experiment, we used the ResNet18 neural network architecture [41], except for the experiment on the MNIST dataset where we utilized a simple ﬁve layers convolutional neural network.
Revised: In every experiment, we used the ResNet18 neural network architecture [41], except for the experiment on the MNIST dataset where we utilized a simple ﬁve layers convolutional neural network.

Original: We use a pre-trained model for ImageNet dataset [26] from torchvision library [42].
Revised: We use a pre-trained model for ImageNet dataset [26] from torchvision library [42].

Original: For the experiments we used the Bernoulli RBM implementation from the Scikit-Learn library [43] with following hyperparameters for each experiment: for the MNIST dataset we set the batch size to 5 , the learning rate to 0 .
Revised: For the experiments we used the Bernoulli RBM implementation from the Scikit-Learn library [43] with following hyperparameters for each experiment: for the MNIST dataset we set the batch size to 5 , the learning rate to 0 .

Original: 01 , and the number of iterations to 100 .
Revised: 01 , and the number of iterations to 100 .

Original: For CIFAR10 and ImageNet datasets we use the following hyperparameters: a batch size of 35 , a learning rate of 0 .
Revised: For CIFAR10 and ImageNet datasets we use the following hyperparameters: a batch size of 35 , a learning rate of 0 .

Original: 001 , and a number of iterations is 250 .
Revised: 001 , and a number of iterations is 250 .

Original: The rest of hyperparameters are default to the scikit-learn package.
Revised: The rest of hyperparameters are default to the scikit-learn package.

Original: For all baseline explanation techniques we use the publicly available open-source implementations from the captum library [44] with their default hyperparameters.
Revised: For all baseline explanation techniques we use the publicly available open-source implementations from the captum library [44] with their default hyperparameters.

Original:  Discussion and Future Work
Revised:  Discussion and Future Work

Original: The results of multiple experiments with the proposed RBM ensemble show its competitive performance compared to base explanation techniques and other ensemble approaches.
Revised: The results of multiple experiments with the proposed RBM ensemble show its competitive performance compared to base explanation techniques and other ensemble approaches.

Original: We hypothesize moderate performance of the RBM ensemble on the insertion (IAUC) benchmark is connected to our data preparation step since we ﬁlter the negative values for every saliency map in the ensemble.
Revised: We hypothesize moderate performance of the RBM ensemble on the insertion (IAUC) benchmark is connected to our data preparation step since we ﬁlter the negative values for every saliency map in the ensemble.

Original: The computational complexity of an ensemble method primarily depends on the base learner.
Revised: The computational complexity of an ensemble method primarily depends on the base learner.

Original: In our case, the base explanation techniques are relatively fast, especially on specialized hardware (GPU or
Revised: In our case, the base explanation techniques are relatively fast, especially on specialized hardware (GPU or

Original: TPU), where an RBM has low computational complexity.
Revised: TPU), where an RBM has low computational complexity.

Original: The gradient-based methods frequently produce noisy explanations.
Revised: The gradient-based methods frequently produce noisy explanations.

Original: We empirically demonstrated that our approach reduces the noise in the ﬁnal ensemble (Tab. 1).
Revised: We empirically demonstrated that our approach reduces the noise in the ﬁnal ensemble (Tab. 1).

Original: Therefore, we believe that the
Revised: Therefore, we believe that the

Original: RBM aggregation of multiple saliency maps from gradient-based feature attributions is a powerful tool for improving the overall reliability of local explanations.
Revised: RBM aggregation of multiple saliency maps from gradient-based feature attributions is a powerful tool for improving the overall reliability of local explanations.

Original: As part of our future work, we aim to evaluate our aggregation approach on larger datasets.
Revised: As part of our future work, we aim to evaluate our aggregation approach on larger datasets.

Original: Furthermore, methods for selecting a few quite reliable base explanations for aggregation might lead to efﬁcient explanations ensembles for larger datasets.
Revised: Furthermore, methods for selecting a few quite reliable base explanations for aggregation might lead to efﬁcient explanations ensembles for larger datasets.

Original: Finally we expect that the proposed approach can be easily adapted to handle local explanations over structured tabular data, where the explanation of deep neural networks is an essential task for many crucial applications such as healthcare and ﬁnance [45]. 
Revised: Finally we expect that the proposed approach can be easily adapted to handle local explanations over structured tabular data, where the explanation of deep neural networks is an essential task for many crucial applications such as healthcare and ﬁnance [45]. [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we presented a novel approach to unsupervised aggregation of feature-based explanations using Restricted Boltzmann Machines with the aim of reliably interpreting the inﬂuence of inputs on the output of deep neural networks.
Revised: In this work, we presented a novel approach to unsupervised aggregation of feature-based explanations using Restricted Boltzmann Machines with the aim of reliably interpreting the inﬂuence of inputs on the output of deep neural networks.

Original: In addition to explanatory reasons, the latter is also essential for debugging and diagnostic purposes and serves the long-term acceptance of deep learning in real-world applications.
Revised: In addition to explanatory reasons, the latter is also essential for debugging and diagnostic purposes and serves the long-term acceptance of deep learning in real-world applications.

Original: Using the proposed approach, we demonstrated through visual and quantitative experiments its ability to obtain more robust and reliable explanations than other existing ensemble methods.
Revised: Using the proposed approach we demonstrated through visual and quantitative experiments its ability to obtain more robust and reliable explanations than other existing ensemble methods.

Original: In a setting with noisy attribution maps in an ensemble, the proposed approach successfully selects only the valuable information, mitigating noise.
Revised: In a setting with noisy attribution maps in an ensemble, the proposed approach successfully selects only the valuable information, mitigating noise.

Original: Moreover, our work illuminates and mitigates the problem of possible contradictory results that may be obtained by different explanation and evaluation methods.
Revised: Moreover, our work illuminates and mitigates the problem of possible contradictory results that may be obtained by different explanation and evaluation methods.

Original: Finally, we note that our approach can also be used within a single interpretability framework to reduce the sensitivity of a feature-based explanatory approach to its hyperparameters.
Revised: Finally, we note that our approach can also be used within a single interpretability framework to reduce the sensitivity of a feature-based explanatory approach to its hyperparameters

Original: A Additional Experiments
Revised: A Additional Experiments

Original: Table 3 presents extended experimental results for the compression with or without noisy feature attribution maps in the ensemble.
Revised: Table 3 presents extended experimental results for the compression with or without noisy feature attribution maps in the ensemble.