Original: MAGNE X : A M ODEL A GNOSTIC G LOBAL N EURAL E XPLAINER
Revised: MAGNE X : A M ODEL A GNOSTIC G LOBAL N EURAL E XPLAINER

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Black-box decision models have been widely adopted both in industry and academia due to their excellent performance across many challenging tasks and domains.
Revised: Black-box decision models have been widely adopted both in industry and academia due to their excellent performance across many challenging tasks and domains.

Original: However, much criticism has been raised around modern AI systems, to a large extent due to their inability to produce explainable decisions that both their end-users and their developers can trust.
Revised: However, much criticism has been raised around modern AI systems, to a large extent due to their inability to produce explainable decisions that both their end-users and their developers can trust.

Original: The need for such decisions, i.e., decisions accompanied by a rationale for why they are made, has ignited much recent research.
Revised: The need for such decisions, i.e., decisions accompanied by a rationale for why they are made, has ignited much recent research.

Original: We propose MAGNE x, a global algorithm that leverages neural-network based explainers to produce rationales for any black-box decision model, neural or not.
Revised: We propose MAGNE x, a global algorithm that leverages neural-network based explainers to produce rationales for any black-box decision model, neural or not.

Original: MAGNE x is model-agnostic, and thus easily generalizable across domains and applications.
Revised: MAGNE x is model-agnostic, and thus easily generalizable across domains and applications.

Original: More importantly, MAGNE x is global, i.e., it learns to create rationales by optimizing for a number of instances at once, contrary to local methods that aim at explaining a single example.
Revised: More importantly, MAGNE x is global, i.e., it learns to create rationales by optimizing for a number of instances at once, contrary to local methods that aim at explaining a single example.

Original: The global nature of MAGNE x has two advantages over local methods: i) it generalizes across instances hence producing more faithful explanations, ii) it is computationally more efﬁcient during inference.
Revised: The global nature of MAGNE x has two advantages over local methods: i) it generalizes across instances hence producing more faithful explanations, ii) it is computationally more efﬁcient during inference.

Original: Our experiments conﬁrm that MAGNE x outperforms popular explainability algorithms both in explanation quality and in computational efﬁciency.
Revised: Our experiments conﬁrm that MAGNE x outperforms popular explainability algorithms both in explanation quality and in computational efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Black-box decision models have for some time now posed a dilemma between power and interpretability.
Revised: Black-box decision models have for some time now posed a dilemma between power and interpretability.

Original: For use cases where explanations are necessary, the inability of black-box models to supply them is often a deterrent to adoption.
Revised: For use cases where explanations are necessary, the inability of black-box models to supply them is often a deterrent to adoption.

Original: However, even in low-risk scenarios this lack of explainability often causes distrust to both the developers and the users of these models, who are often puzzled about how decisions emerge (Ribeiro et al., 2016).
Revised: However, even in low-risk scenarios this lack of explainability often causes distrust to both the developers and the users of these models, who are often puzzled about how decisions emerge (Ribeiro et al., 2016).

Original: Furthermore, explainability is an important mechanism to ensure black-box models act fairly and without bias (Sun et al., 2019).
Revised: Also, explainability is an important mechanism when investigating if black-box models act fairly and without bias (Sun et al., 2019).

Original: In this paper, we propose MAGNE x, a model-agnostic neural explainer that globally learns to explain an already trained model, neural or not.
Revised: In this paper, we propose MAGNE x, a model-agnostic neural explainer that globally learns to explain an already trained model, neural or not.

Original: In this post-hoc interpretability setting, most methods (Ribeiro et al., 2016; Sundararajan et al., 2017; Lundberg & Lee, 2017) create feature-based explanations, i.e., explanations that assign a score to each featurein the input of the model based on how important the feature is to the model’s decision.
Revised: In this post-hoc interpretability setting, most methods (Ribeiro et al., 2016; Sundararajan et al., 2017; Lundberg & Lee, 2017; Luo et al., 2020) create feature-based explanations, i.e., explanations that assign a score to each feature of the input based on how important the feature is to the model’s decision.

Original: This importance score may rely on some internal mechanism of the model we wish to explain, e.g., gradients (Sundararajan et al., 2017; Shrikumar et al., 2017) or attention (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).
Revised: This importance score may rely on some internal mechanism of the model we wish to explain, e.g., gradients (Sundararajan et al., 2017; Shrikumar et al., 2017) or attention (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019).

Original: Such methods, however, have limitations to the types of model they can explain, e.g., gradient-based methods work only with differentiable models.
Revised: Such methods have limitations to the types of model they can explain, e.g., gradient-based methods work only with differentiable models.

Original: Perturbation-based methods, e.g., LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), drop combinations of features for a speciﬁc input to the model and observe its output.
Revised: Perturbation-based methods, e.g., LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), drop combinations of features for a speciﬁc input to the model and observe its output.

Original: Omitted features that have a large impact on the output of the model across perturbations are deemed important, while other features are considered unimportant.
Revised: Omitted features that have a large impact on the output of the model across perturbations are deemed important, while other features are considered unimportant.

Original: This allows perturbation-based methods to be model-agnostic, but adds severe computational complexity, since a large number of perturbations is required per input instance to create quality explanations.
Revised: This allows perturbation-based methods to be model-agnostic, but adds severe computational complexity, since a large number of perturbations is required per input instance to create quality explanations.

Original: The search for an optimal solution by erasure (feature drop) is combinatorial and practically infeasible for even small feature spaces; the search space is the power set of the feature set resulting in a complexity of O (2 n ) for n features.
Revised: The search for an optimal solution by erasure (feature drop) is combinatorial and practically infeasible for even small feature spaces; the search space is the power set of the feature set resulting in a complexity of O (2 n ) for n features.

Original: Therefore, pertubation-based methods ﬁnd approximate (sub-optimal) solutions, but even this process is cumbersome, especially when the input is large (i.e., contains many features) and the explainability method is local (a different search must be performed for each input instance).
Revised: Therefore, pertubation-based methods ﬁnd approximate (sub-optimal) solutions, but even this process is cumbersome, especially when the input is large (i.e., contains many features) and the explainability method is local (a different search must be performed for each input instance).

Original: The scores are used to create a binary mask (Eq. 2) representing the parts of i to be retained.
Revised: The scores are used to create a binary mask (Eq. 2) representing the parts of i to be retained.

Original: The original and masked input are both passed through the pre-trained model and the outputs, y and ˆ y are used to calculate the sufﬁciency score.
Revised: The original and masked input are both passed through the pre-trained model and the outputs, y and ˆ y are used to calculate the sufﬁciency score.

Original: The explainer aims to maximize sufﬁciency while also masking as many features as possible, thus also maximizing sparsity .
Revised: The explainer aims to maximize sufﬁciency while also masking as many features as possible, thus also maximizing sparsity .

Original: MAGNE x also creates feature-based explanations, but contrary to most methods, its explainer is a neural network that globally learns to assign feature importance scores.
Revised: MAGNE x also creates feature-based explanations, but contrary to most methods, its explainer is a neural network that globally learns to assign feature importance scores.

Original: This is similar to the work of De Cao et al.
Revised: This is similar to the work of De Cao et al.

Original: (2020), which relies on gradients to produce feature importance scores
Revised: (2020), which relies on gradients to produce feature importance scores

Original: ; their explainer is a shallow network directly attached to the model being explained, requiring the gradients to ﬂow through both the explainer and the model.
Revised: ; their explainer is a shallow network directly attached to the model being explained, requiring the gradients to ﬂow through both the explainer and the model.

Original: This limits the approach of De Cao et al.
Revised: This limits the approach of De Cao et al.

Original: (2020) to explaining only differentiable models, while increasing the memory and computational complexity.
Revised: (2020) to explaining only differentiable models, while increasing the memory and computational complexity.

Original: Neural explanation modules have also been used in explainable by-design models (Lei et al., 2016;
Revised: Neural explanation modules have also been used in explainable by-design models (Lei et al., 2016;

Original: Bastings et al., 2019; Yu et al., 2019; Chang et al., 2019; Chalkidis et al., 2021).
Revised: Bastings et al., 2019; Yu et al., 2019; Chang et al., 2019; Chalkidis et al., 2021).

Original: These models mainly focus on human-centric explainability and jointly train a rationale extractor and a classiﬁer.
Revised: These models mainly focus on human-centric explainability and jointly train a rationale extractor and a classiﬁer.

Original: This setting poses an extra burden on the training procedure and disincentivizes exploring parts of the space of possible solutions the model can arrive to, possibly leading to loss of model performance at the expense of interpretability.
Revised: This setting poses an extra burden on the training procedure and disincentivizes exploring parts of the space of possible solutions the model can arrive to, possibly leading to loss of model performance at the expense of interpretability.

Original: MAGNE x disentangles the explainer from the model it wishes to explain, aiming to produce explanations that are faithful , i.e., accurately reﬂect the features considered important by the model (Lipton, 2018;
Revised: MAGNE x disentangles the explainer from the model it wishes to explain, aiming to produce explanations that are faithful , i.e., accurately reﬂect the features considered important by the model (Lipton, 2018;

Original: Jacovi & Goldberg, 2020).
Revised: Jacovi & Goldberg, 2020).

Original: The generated explanations do not have to agree with human annotated rationales (e.g., gold text snippets), which are often used as an optimal solution in explainable by-design methods.
Revised: The generated explanations do not have to agree with human annotated rationales (e.g., gold text snippets), which are often used as an optimal solution in explainable by-design methods.

Original: MAGNE x’s goal is to faithfully reveal information about a model’s inner workings to its developers and if its explanations agree with human intuition, then MAGNE x can also be used as an explanation algorithm for end-users.
Revised: MAGNE x’s goal is to faithfully reveal information about a model’s inner workings to its developers and if its explanations agree with human intuition, then MAGNE x can also be used as an explanation algorithm for end-users.

Original: If the explanations do not agree with human intuition, MAGNE x can reveal weaknesses of the explained model and/or of the data used to train it.
Revised: If the explanations do not agree with human intuition, MAGNE x can reveal weaknesses of the explained model and/or of the data used to train it.

Original: For instance, training data may not be diverse enough, allowing a model to overﬁt to insigniﬁcant features in its input.
Revised: For instance, training data may not be diverse enough, allowing a model to overﬁt to insigniﬁcant features in its input.

Original: The explainer of MAGNE x is a shallow neural network that attributes scores to input features (Figure 1).
Revised: The explainer of MAGNE x is a shallow neural network that attributes scores to input features (Figure 1).

Original: Its training is dictated by a two-fold objective that can be classiﬁed under constrained optimization.
Revised: Its training is dictated by a two-fold objective that can be classiﬁed under constrained optimization.

Original: The scoring of the explainer must be primarily sufﬁcient , i.e., the explanations when fed to the underlying model must result in the same output (DeYoung et al., 2020).
Revised: The scoring of the explainer must be primarily sufﬁcient , i.e., the explanations when fed to the underlying model must result in the same output (DeYoung et al., 2020).

Original: Also, among all possible sufﬁcient explanations the explainer is tasked with ﬁnding one that is maximally sparse , i.e., utilizes the minimum number of features (Lei et al., 2016).
Revised: Also, among all possible sufﬁcient explanations the explainer is tasked with ﬁnding one that is maximally sparse , i.e., utilizes the minimum number of features (Lei et al., 2016).

Original: This constrained objective, or a relaxed version of it, has been also used by perturbation-based systems (De Cao et al., 2020; Ribeiro et al., 2016).
Revised: This constrained objective, or a relaxed version of it, has been also used by perturbation-based systems (De Cao et al., 2020; Ribeiro et al., 2016).

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We propose MAGNE x a method that explains the inner workings of a pre-trained model in a post-hoc manner.
Revised: • We propose MAGNE x a method that explains the inner workings of a pre-trained model in a post-hoc manner.

Original: MAGNE x, is completely model agnostic and thus can explain any pre-trained model across modalities (e.g., vision, text) and applications.
Revised: MAGNE x, is completely model agnostic and thus can explain any pre-trained model across modalities (e.g., vision, text) and applications.

Original: • The proposed approach is global, allowing MAGNE x to generalize across instances; learning from many samples has a regularization effect alleviating hindsight bias (Fischhoff & Beyth, 1975) which is a common phenomenon in machine learning problems (Mahdavi & Rahimian, 2017) and more speciﬁcaly in perturbation-based explainability (De Cao et al.,
Revised: • The proposed approach is global, allowing MAGNE x to generalize across instances; learning from many samples has a regularization effect alleviating hindsight bias (Fischhoff & Beyth, 1975) which is a common phenomenon in machine learning problems (Mahdavi & Rahimian, 2017) and more speciﬁcaly in perturbation-based explainability (De Cao et al.,

Original: Also, the global nature of MAGNE x transfers the computationally expensive feature search to the training stage allowing for more efﬁcient inference than its competitors.
Revised: Also, the global nature of MAGNE x transfers the computationally expensive feature search to the training stage allowing for more efﬁcient inference than its competitors.

Original: • The experiments showed that MAGNE x produces explanations of better quality than popular explainability methods ( LIME (Ribeiro et al., 2016), IG (Sundararajan et al., 2017)), while also being more stable across instances and much more efﬁcient during inference.
Revised: • The experiments showed that MAGNE x produces explanations of better quality than popular explainability methods ( LIME (Ribeiro et al., 2016), IG (Sundararajan et al., 2017)), while also being more stable across instances and much more efﬁcient during inference.

Original:  M ETHODOLOGY 2. F ORMULATION
Revised:  M ETHODOLOGY 2. F ORMULATION

Original: Let M : I → O be the model we wish to explain.
Revised: Let M : I → O be the model we wish to explain.

Original: Each input i ∈
Revised: Each input i ∈

Original: I is a set of input features such that
Revised: I is a set of input features such that

Original: i = { f 1 , f 2 , . .
Revised: i = { f 1 , f 2 , . .

Original: . , f n } , where n is the number of features.
Revised: . , f n } , where n is the number of features.

Original: An explainer, E , operates on i and associates each feature in i with an importance score s j ∈ [0 , 1] for j ∈ { 1 , 2 , . .
Revised: An explainer, E , operates on i and associates each feature in i with an importance score s j ∈ [0 , 1] for j ∈ { 1 , 2 , . .

Original: . , n } :
Revised: . , n } :

Original: [Equation]
Revised: [Equation]

Original: A masking function µ , operates on an input i = { f 1 , f 2 , . .
Revised: A masking function µ , operates on an input i = { f 1 , f 2 , . .

Original: . , f n } and its corresponding scoring s
Revised: . , f n } and its corresponding scoring s

Original: = { s 1 , s 2 , . .
Revised: = { s 1 , s 2 , . .

Original: , s n } to produce a masked version of i :
Revised: , s n } to produce a masked version of i :

Original: [Equation]
Revised: [Equation]

Original: where m is a feature which carries no information for M and is chosen according to the task, e.g., a black pixel in computer vision or a pad token in natural language processing.
Revised: where m is a feature which carries no information for M and is chosen according to the task, e.g., a black pixel in computer vision or a pad token in natural language processing.

Original: In practise, µ is realized in different ways in training and inference.
Revised: In practise, µ is realized in different ways in training and inference.

Original: During training whether a feature f j will be replaced by m is determined by performing a biased coin ﬂip based on s j (the score of f j ), while during inference µ is realized as:
Revised: During training whether a feature f j will be replaced by m is determined by performing a biased coin ﬂip based on s j (the score of f j ), while during inference µ is realized as:

Original: [Equation]
Revised: [Equation]

Original: Given an input i and a scoring s = E ( i ) , we compute the quality of the explainer’s scoring in terms of sufﬁciency ( su ) and sparsity ( sp ), deﬁned as follows:
Revised: Given an input i and a scoring s = E ( i ) , we compute the quality of the explainer’s scoring in terms of sufﬁciency ( su ) and sparsity ( sp ), deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where, g su : O × O → [0 , 1] is a function measuring to what extent the output of M when presented with µ ( i , s ) resembles the output of M when presented with i , and g sp computes the percentage of features which have been replaced with m .
Revised: where, g su : O × O → [0 , 1] is a function measuring to what extent the output of M when presented with µ ( i , s ) resembles the output of M when presented with i , and g sp computes the percentage of features which have been replaced with m .

Original: Since we opt for maximally sparse inputs with high sufﬁciency, the total quality score for the explanation is computed as:
Revised: Since we opt for maximally sparse inputs with high sufﬁciency, the total quality score for the explanation is computed as:

Original: [Equation]
Revised: [Equation]

Original: Note that we force explanations that have a sufﬁciency of at least υ to ensure a minimum quality.
Revised: Note that we force explanations that have a sufﬁciency of at least υ to ensure a minimum quality.

Original: Unless otherwise speciﬁed, we use a g su tailored to classiﬁcation where we assume O = [0 , 1] c and c is the number of classes.
Revised: Unless otherwise speciﬁed, we use a g su tailored to classiﬁcation where we assume O = [0 , 1] c and c is the number of classes.

Original: Classiﬁcation sufﬁciency is then calculated as:
Revised: Classiﬁcation sufﬁciency is then calculated as:

Original: [Equation]
Revised: [Equation]

Original: where p = argmax( y ) is the predicted class in y ∈
Revised: where p = argmax( y ) is the predicted class in y ∈

Original: O and g su is bounded in [0 , 1] with higher scores signaling more faithful explanations.
Revised: O and g su is bounded in [0 , 1] with higher scores signaling more faithful explanations.

Original: The term max( y ) − ˆ y p measures the divergence between the top probability estimate across classes in y and the probability estimate for the same class in ˆ y .
Revised: The term max( y ) − ˆ y p measures the divergence between the top probability estimate across classes in y and the probability estimate for the same class in ˆ y .

Original: 2. L EARNING
Revised: 2. L EARNING

Original: Our explainer is realized as a neural network with parameters θ throughout.
Revised: Our explainer is realized as a neural network with parameters θ throughout.

Original: We aim to ﬁnd the optimal values θ ∗ which maximize q (Eq. 6) across a number of training examples { i 1 , i 2 , . .
Revised: We aim to ﬁnd the optimal values θ ∗ which maximize q (Eq. 6) across a number of training examples { i 1 , i 2 , . .

Original: , i m } ⊆
Revised: , i m } ⊆

Original: I .
Revised: I .

Original: However, the standard backpropagation optimization approach falls short in this case since it is impossible to produce gradients to update the explainer.
Revised: However, the standard backpropagation optimization approach falls short in this case since it is impossible to produce gradients to update the explainer.

Original: While the explainer itself is differentiable, in order for a sufﬁciency score to be computed for some input i a hard choice must be made on
Revised: While the explainer itself is differentiable, in order for a sufﬁciency score to be computed for some input i a hard choice must be made on

Original: [Equation]
Revised: [Equation]

Original: which features in i to retain and which to substitute with m
Revised: which features in i to retain and which to substitute with m

Original: The masking function µ is therefore non-differentiable.
Revised: The masking function µ is therefore non-differentiable.

Original: More importantly, our approach is completely model-agnostic and we make no assumptions about whether the model we wish to explain ( M ) is differentiable or not.
Revised: More importantly, our approach is completely model-agnostic and we make no assumptions about whether the model we wish to explain ( M ) is differentiable or not.

Original: In the simplest scenario, where only the masking function is non-differentiable, a number of approaches have attempted to produce gradient estimations with methods other than the score estimator, based on REINFORCE (Williams, 1992).
Revised: In the simplest scenario, where only the masking function is non-differentiable, a number of approaches have attempted to produce gradient estimations with methods other than the score estimator, based on REINFORCE (Williams, 1992).

Original: The most common of these approaches are the straightthrough estimator (Chang et al., 2019; Chalkidis et al., 2021) and relaxation to binary variables (Louizos et al., 2018; Bastings et al., 2019; De Cao et al., 2020) which leverages the reparametrization trick (Kingma & Welling, 2013).
Revised: The most common of these approaches are the straightthrough estimator (Chang et al., 2019; Chalkidis et al., 2021) and relaxation to binary variables (Louizos et al., 2018; Bastings et al., 2019; De Cao et al., 2020) which leverages the reparametrization trick (Kingma & Welling, 2013).

Original: Both of these approaches can be used only when the masking function is the only non-differentiable component, i.e., they require M to be differentiable, thus breaking the model-agnostic nature of the explainer, which is a requirement in MAGNE x.
Revised: Both of these approaches can be used only when the masking function is the only non-differentiable component, i.e., they require M to be differentiable, thus breaking the model-agnostic nature of the explainer, which is a requirement in MAGNE x.

Original: To retain the model-agnostic nature of our formulation and alleviate large computational strain to our method we opt to train our model by estimating gradients for updates to our explainer network with the score estimator ( REINFORCE ).
Revised: To retain the model-agnostic nature of our formulation and alleviate large computational strain to our method we opt to train our model by estimating gradients for updates to our explainer network with the score estimator ( REINFORCE ).

Original: For training on a single input i we create a multi-variable policy using the output of our explainer.
Revised: For training on a single input i we create a multi-variable policy using the output of our explainer.

Original: [Equation]
Revised: [Equation]

Original: 8) where B is the Bernoulli distribution.
Revised: 8) where B is the Bernoulli distribution.

Original: Sampling from this policy is equivalent to sampling from each of the Bernoulli distributions independently.
Revised: Sampling from this policy is equivalent to sampling from each of the Bernoulli distributions independently.

Original: Therefore a sample τ ∼ π θ ( i ) is a sequence of binary variables indicating the presence or the absence of the feature at position j .
Revised: Therefore a sample τ ∼ π θ ( i ) is a sequence of binary variables indicating the presence or the absence of the feature at position j .

Original: We train our method using the score estimator which in this case can be written as:
Revised: We train our method using the score estimator which in this case can be written as:

Original: [Equation]
Revised: [Equation]

Original: Following Williams (1992) we can rewrite the above gradient in the form:
Revised: Following Williams (1992) we can rewrite the above gradient in the form:

Original: [Equation]
Revised: [Equation]

Original: where P( τ | i; θ ) = (cid:81) | i | j =1 P ( τ j | π θ ( i ) j ) .
Revised: where P( τ | i; θ ) = (cid:81) | i | j =1 P ( τ j | π θ ( i ) j ) .

Original: We can easily approximate Eq.
Revised: We can easily approximate Eq.

Original: 10 by Monte-Carlo sampling.
Revised: 10 by Monte-Carlo sampling.

Original: 2 We further add a baseline in order to reduce the variance of the gradient estimator:
Revised: 2 We further add a baseline in order to reduce the variance of the gradient estimator:

Original: [Equation]
Revised: [Equation]

Original: In Mnih et al.
Revised: In Mnih et al.

Original: (2014), which bears some similarities to our setting a learned baseline was used.
Revised: (2014), which bears some similarities to our setting a learned baseline was used.

Original: Here, we use a moving average baseline which seems to be sufﬁcient for our use cases.
Revised: Here, we use a moving average baseline which seems to be sufﬁcient for our use cases.

Original: While similar methods relying on differentiable relaxation to binary variables have been shown to outperform REINFORCE in some cases (Bastings et al., 2019; De Cao et al., 2020), we choose this gradient estimator for a number of reasons.
Revised: While similar methods relying on differentiable relaxation to binary variables have been shown to outperform REINFORCE in some cases (Bastings et al., 2019; De Cao et al., 2020), we choose this gradient estimator for a number of reasons.

Original: Firstly, we want our method to remain purely modelagnostic, a requirement which cannot be satisﬁed by the estimators in Bastings et al.
Revised: Firstly, we want our method to remain purely modelagnostic, a requirement which cannot be satisﬁed by the estimators in Bastings et al.

Original: and
Revised: and

Original: De Cao et al.
Revised: De Cao et al.

Original: (2020) which support only differentiable models.
Revised: (2020) which support only differentiable models.

Original: Secondly, the fact that the estimator works by simple exposure to a scalar metric (Eq. 6), which does not need to have a gradient, greatly reduces the space complexity of the method, allowing a higher degree of parallelism on the same hardware, and in practice allowing very complex models to be explained in reasonable time.
Revised: Secondly, the fact that the estimator works by simple exposure to a scalar metric (Eq. 6), which does not need to have a gradient, greatly reduces the space complexity of the method, allowing a higher degree of parallelism on the same hardware, and in practice allowing very complex models to be explained in reasonable time.

Original: Lastly, this problem involves constrained optimization.
Revised: Lastly, this problem involves constrained optimization.

Original: Looking back at the deﬁnition of our total metric (Eq. 6) we can see that we are optimizing sparsity subject to sufﬁciency being higher than some threshold.
Revised: Looking back at the deﬁnition of our total metric (Eq. 6) we can see that we are optimizing sparsity subject to sufﬁciency being higher than some threshold.

Original: This objective is therefore non-differentiable and De Cao et al.
Revised: This objective is therefore non-differentiable and De Cao et al.

Original: (2020) and
Revised: (2020) and

Original: Bastings et al.
Revised: Bastings et al.

Original: (2019) employ Lagrangian relaxation to approximate the constrained objective in a differentiable manner.
Revised: (2019) employ Lagrangian relaxation to approximate the constrained objective in a differentiable manner.

Original: This adds a new hyperparameter, the Lagrangian multiplier, which needs to be tuned, adding further overhead to the explainer’s development procedure and is in all cases an approximation of the true constrained objective.
Revised: This adds a new hyperparameter, the Lagrangian multiplier, which needs to be tuned, adding further overhead to the explainer’s development procedure and is in all cases an approximation of the true constrained objective.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We tested MAGNE x in three challenging settings across modalities, i.e., image classiﬁcation, sentiment classiﬁcation, and question answering.
Revised: We tested MAGNE x in three challenging settings across modalities, i.e., image classiﬁcation, sentiment classiﬁcation, and question answering.

Original: Although, some of these tasks are simple to handle with modern models, they pose a challenging setting for perturbation-based algorithms, since feature spaces are large compared to previous work (Ribeiro et al., 2016; Lei et al., 2016; Bastings et al., 2019; De Cao et al., 2020).
Revised: Although, some of these tasks are simple to handle with modern models, they pose a challenging setting for perturbation-based algorithms, since feature spaces are large compared to previous work (Ribeiro et al., 2016; Lei et al., 2016; Bastings et al., 2019; De Cao et al., 2020).

Original: This makes it difﬁcult for such methods to identify subsets of the input which are sparse and yet expressive enough to lead to the same output.
Revised: This makes it difﬁcult for such methods to identify subsets of the input which are sparse and yet expressive enough to lead to the same output.

Original: 3. B ASELINES AND EVALUATION
Revised: 3. B ASELINES AND EVALUATION

Original: We compare MAGNE x against Lime (Ribeiro et al., 2016) and Integrated Gradients ( IG ) (Sundararajan et al., 2017), two popular post-hoc explainability methods.
Revised: We compare MAGNE x against Lime (Ribeiro et al., 2016) and Integrated Gradients ( IG ) (Sundararajan et al., 2017), two popular post-hoc explainability methods.

Original: Recall that the explainer of MAGNE x ( E ) outputs a score s j ∈ [0 , 1] for each feature f j of an input i .
Revised: Recall that the explainer of MAGNE x ( E ) outputs a score s j ∈ [0 , 1] for each feature f j of an input i .

Original: This score can be interpreted as the probability of f j being important for a model to produce the same output.
Revised: This score can be interpreted as the probability of f j being important for a model to produce the same output.

Original: During training these probabilistic scores are converted to a binary value with a biased coin ﬂip.
Revised: During training these probabilistic scores are converted to a binary value with a biased coin ﬂip.

Original: During inference we cast a feature as important if its respective explainer score is above a treshold (cid:15) ( 0 .
Revised: During inference we cast a feature as important if its respective explainer score is above a treshold (cid:15) ( 0 .

Original: 5 in our experiments).
Revised: 5 in our experiments).

Original: On the other hand,the scores of both LIME and IG are unbounded and with no obvious meaning. This makes it non-trivial to decide which features are important and which ones are not.
Revised: On the other hand, both LIME and IG compute relative feature importance (i.e. whether a feature is more important than another) but have no explicit threshold to decide which features to keep.

Revised: This is left up to the user as a post-processing step.

Original: We therefore evaluate both of these methods by selecting the top k most highly scored features, where k is the number of features selected by E ( MAGNE x) for the same input.
Revised: We therefore evaluate both of these methods by selecting the top k most highly scored features, where k is the number of features selected by E ( MAGNE x) for the same input.

Original: In other words, we evaluate the sufﬁciency of all explanations at the sparsity level achieved by E .
Revised: In other words, we evaluate the sufﬁciency of all explanations at the sparsity level achieved by E .

Original: For each method we also report the time required to produce explanations averaged across inputs.
Revised: For each method we also report the time required to produce explanations averaged across inputs.

Original: Further, we intentionally make no attempt to compare explanations against human annotated rationales.
Revised: Further, we intentionally make no attempt to compare explanations against human annotated rationales.

Original: Our main goal is to produce explanations which are faithful to the underlying model, i.e., accurately reﬂect the features in an input which are important for the model (Lipton, 2018; Jacovi & Goldberg, 2020), which is generally ensured when the sufﬁciency scores are high.
Revised: Our main goal is to produce explanations which are faithful to the underlying model, i.e., accurately reﬂect the features in an input which are important for the model (Lipton, 2018; Jacovi & Goldberg, 2020), which is generally ensured when the sufﬁciency scores are high.

Original: Whether these faithful explanations align with what a human would consider a correct explanation is an open question and beyond the scope of this work.
Revised: Whether these faithful explanations align with what a human would consider a correct explanation is an open question and beyond the scope of this work.

Original: 3. T ECHNICAL DETAILS
Revised: 3. T ECHNICAL DETAILS

Original: For IG , following Sundararajan et al.
Revised: For IG , following Sundararajan et al.

Original: , we use 50 steps in the approximation of the integral throughout.
Revised: , we use 50 steps in the approximation of the integral throughout.

Original: 3 Concerning LIME , shallow models allow a larger number of perturbations to be drawn.
Revised: 4 Concerning LIME , shallow models allow a larger number of perturbations to be drawn.

Original: Therefore, in image classiﬁcation where we use MAGNE x to explain shallow models, we allow 1 ,perturbations to be drawn.
Revised: Therefore, in image classiﬁcation where we use MAGNE x to explain shallow models, we allow 1 ,perturbations to be drawn.

Original: On the other hand, for sentiment classiﬁcation the model complexity increases and we allow 1 , 000 perturbations, to ensure that the explanations will be created in reasonable amount of time (less than 10 seconds per input).
Revised: On the other hand, for sentiment classiﬁcation the model complexity increases and we allow 1 , 000 perturbations, to ensure that the explanations will be created in reasonable amount of time (less than 10 seconds per input).

Original: For similar reasons, in question answering where we try to explain the most complex model in this work, we allow only 700 perturbations per example which again keeps explanation generation time around the 10 second mark per input.
Revised: For similar reasons, in question answering where we try to explain the most complex model in this work, we allow only 700 perturbations per example which again keeps explanation generation time around the 10 second mark per input.

Original: 3. I MAGE CLASSIFICATION
Revised: 3. I MAGE CLASSIFICATION

Original: For image classiﬁcation we use the popular MNIST dataset (Deng, 2012).
Revised: For image classiﬁcation we use the popular MNIST dataset (Deng, 2012).

Original: The input features are pixels, and we choose E to be a two-layer Convolutional Neural Network ( CNN ) (LeCun et al., 1990) which creates a vector representation for each pixel, followed by a shared linear projection to output a single score per pixel.
Revised: The input features are pixels, and we choose E to be a two-layer Convolutional Neural Network ( CNN ) (LeCun et al., 1990) which creates a vector representation for each pixel, followed by a shared linear projection to output a single score per pixel.

Original: The feature space size is the number of pixels in an image ( 28 × 28 = 784 ).
Revised: The feature space size is the number of pixels in an image ( 28 × 28 = 784 ).

Original: 3.3. E XPLAINING R ANDOM F OREST
Revised: 3.3. E XPLAINING R ANDOM F OREST

Original: We train a random decision forest (Ho, 1995) to perform digit classiﬁcation from raw pixels.
Revised: We train a random decision forest (Ho, 1995) to perform digit classiﬁcation from raw pixels.

Original: The model achieves a perfect accuracy on the test set.
Revised: The model achieves a perfect accuracy on the test set.

Original: In this case, we only compare against LIME because random forests are not differentiable and IG is unable to produce explanations.
Revised: In this case, we only compare against LIME because random forests are not differentiable and IG is unable to produce explanations.

Original: Tableshows the results.
Revised: Tableshows the results.

Original: MAGNE x is able to achieve much higher sufﬁciency than LIME while being faster during inference.
Revised: MAGNE x is able to achieve much higher sufﬁciency than LIME while being faster during inference.

Original: Explanations are also more stable (smaller standard deviation across inputs).[Table]
Revised: Explanations are also more stable (smaller standard deviation across inputs).

Original: . E XPLAINING A CNN
Revised: 3.3. E XPLAINING A CNN

Original: We also train a two-layer CNN on the same task and compare MAGNE x against LIME and IG since the CNN is able to produce gradients.
Revised: We also train a two-layer CNN on the same task and compare MAGNE x against LIME and IG since the CNN is able to produce gradients.

Original: The CNN reaches near-perfect accuracy on the test set ( 99% ).
Revised: The CNN reaches near-perfect accuracy on the test set ( 99% ).

Original: Our results can be seen in Table 2.
Revised: Our results can be seen in Table 2.

Original: All methods have very high sufﬁciency scores, producing faithful explanations.
Revised: All methods have very high sufﬁciency scores, producing faithful We only perform a mild

Revised: We also attempted to use a simple gradient approach (Jacobian) but the results were signiﬁcantly inferior.

Original:  This could be attributed to the fact that the pre-trained CNN is much more robust to slight changes in the input images.
Revised: [Table] explanations. This could be attributed to the fact that the pre-trained CNN is much more robust to slight changes in the input images.

Original: It appears to have generalized better than the random forests and can be more easily explained, as LIME performs well here while it was had difﬁculties when explaining the random forests.
Revised: It appears to have generalized better than the random forests and can be more easily explained [Table] as LIME performs well here while it was had difﬁculties when explaining the random forests.

Original: This is further supported by the fact that the sparsity here is higher, meaning that we can mask a larger percentage of each input while retaining very high sufﬁciency.
Revised: This is further supported by the fact that the sparsity here is higher, meaning that we can mask a larger percentage of each input while retaining very high sufﬁciency.

Original: When it comes to efﬁciency our method remains much faster during inference than its competitors.
Revised: When it comes to efﬁciency our method remains much faster during inference than its competitors.

Original: [Table]
Revised: [Table]

Original: 3. S ENTIMENT C LASSIFICATION
Revised: 3. S ENTIMENT C LASSIFICATION

Original: We ﬁne-tune BERT (Devlin et al., 2019) for binary sentiment classiﬁcation on the IMDB review dataset (Maas et al., 2011).
Revised: We ﬁne-tune BERT (Devlin et al., 2019) for binary sentiment classiﬁcation on the IMDB review dataset (Maas et al., 2011).

Original: In this task, the input is a sequence of tokens.
Revised: In this task, the input is a sequence of tokens.

Original: Again we choose a dataset with a large number of features (tokens) per instance, to ensure our method is tested for its scalability to realistic scenarios.
Revised: Again we choose a dataset with a large number of features (tokens) per instance, to ensure our method is tested for its scalability to realistic scenarios.

Original: On the validation set, the mean number of tokens per review is 303 .
Revised: On the validation set, the mean number of tokens per review is 303 .

Original: The maximum allowed number of tokens is 512 ( 13% of validation instances).
Revised: The maximum allowed number of tokens is 512 ( 13% of validation instances).

Original: BERT achieves 94% accuracy on the test set.
Revised: BERT achieves 94% accuracy on the test set.

Original: Here E is a single-layer GRU model (Cho et al., 2014) operating on the contexualized embeddings drawn from the last layer of the ﬁne-tuned BERT model.
Revised: Here E is a single-layer GRU model (Cho et al., 2014) operating on the contexualized embeddings drawn from the last layer of the ﬁne-tuned BERT model.

Original: In effect the
Revised: In effect the

Original: GRU further contexualizes these embeddings before passing them to a shared linear projection to create the score for each token.
Revised: GRU further contexualizes these embeddings before passing them to a shared linear projection to create the score for each token.

Original: Updates are only performed on the GRU and the linear projection, i.e., the ﬁne-tuned BERT remains frozen.
Revised: Updates are only performed on the GRU and the linear projection, i.e., the ﬁne-tuned BERT remains frozen.

Original: The whole architecture of MAGNE x for this setting can be seen in Figure 2.
Revised: The whole architecture of MAGNE x for this setting can be seen in Figure 2.

Original: As mentioned in Section 2.2, a reason for choosing REINFORCE is the alleviation of a lot of computational stress from the training procedure.
Revised: As mentioned in Section 2.2, a reason for choosing REINFORCE is the alleviation of a lot of computational stress from the training procedure.

Original: If we were to use differentiable masking (relaxation to binary variables) to create semi-hard scores for each token, we would have to produce gradients for the feedback given back from the pre-trained model.
Revised: If we were to use differentiable masking (relaxation to binary variables) to create semi-hard scores for each token, we would have to produce gradients for the feedback given back from the pre-trained model.

Original: The computation graph in this case would be further tasked with tracking gradients for the BERT model and all its intermediate computation throughout the explainer’s training procedure.
Revised: The computation graph in this case would be further tasked with tracking gradients for the BERT model and all its intermediate computation throughout the explainer’s training procedure.

Original: Our results can be seen in Table 3.
Revised: Our results can be seen in Table 3.

Original: MAGNE x produces comparable or better explanations than its competitors, while being faster during inference.
Revised: MAGNE x produces comparable or better explanations than its competitors, while being faster during inference.

Original: One additional advantage of MAGNE x is that it alleviates hindsight bias , which can often be observed in perturbation-based explainability.
Revised: One additional advantage of MAGNE x is that it alleviates hindsight bias , which can often be observed in perturbation-based explainability.

Original: Whenperforming erasure (i.e., dropping features from the input to a model) there is no guarantee that low scored features are indeed not useful for the underlying model. The token great in Figure 3b
Revised: When

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Both explanations have high sufﬁciency, but we believe that (a) is a more faithful interpretation of the model’s computation.
Revised: Both explanations have high sufﬁciency, but we believe that (a) is a more faithful interpretation of the model’s computation.

Original: LIME appears to have fallen victim of hindsight bias identifying a single feature ( great ) as important.
Revised: LIME appears to have fallen victim of hindsight bias identifying a single feature ( great ) as important.

Original: This does not mean that the underlying model ignores everything else.
Revised: This does not mean that the underlying model ignores everything else.

Original: MAGNE x considers more features to be important as its global training acts against hindsight bias.
Revised: MAGNE x considers more features to be important as its global training acts against hindsight bias.

Revised: performing erasure (i.e., dropping features from the input to a model) there is no guarantee that low scored features are indeed not useful for the underlying model.

Original:  is scored very highly in comparison to all other tokens.
Revised: The token great in Figure 3b is scored very highly in comparison to all other tokens.

Original: Retaining this token alone would lead to the same output.
Revised: Retaining this token alone would lead to the same output.

Original: This does not necessarily mean that the pre-trained model does not consider any other tokens when casting a decision.
Revised: This does not necessarily mean that the pre-trained model does not consider any other tokens when casting a decision.

Original: Although we employ no explicit mechanism to tackle this issue, we believe that we make steps towards the right direction due to two characteristics of our method.
Revised: Although we employ no explicit mechanism to tackle this issue, we believe that we make steps towards the right direction due to two characteristics of our method.

Original: Firstly, MAGNE x is global, i.e., it learns a model across a large number of instances, which we believe to act as a regularizer against hindsight bias.
Revised: Firstly, MAGNE x is global, i.e., it learns a model across a large number of instances, which we believe to act as a regularizer against hindsight bias.

Original: The exact optimum of erasure for a speciﬁc instance can easily fall victim to hindsight bias and therefore approximations of this exact optimum (e.g., LIME ) can often exhibit this same behaviour.
Revised: The exact optimum of erasure for a speciﬁc instance can easily fall victim to hindsight bias and therefore approximations of this exact optimum (e.g., LIME ) can often exhibit this same behaviour.

Original: On the other hand, global context allows
Revised: On the other hand, global context allows

Original: MAGNE x to reveal patterns that generalize across instances instead of local ones.
Revised: MAGNE x to reveal patterns that generalize across instances instead of local ones.

Original: For instance, great being associated with a positive prediction is a pattern local to an instance that cannot be established globally.
Revised: For instance, great being associated with a positive prediction is a pattern local to an instance that cannot be established globally.

Original: It is therefore, disincentivised as a solution in MAGNE x’s global training regime.
Revised: It is therefore, disincentivised as a solution in MAGNE x’s global training regime.

Original: We do not argue that such patterns are non-existent, rather that they are important only when they can be established across a variety of instances and can otherwise create extremely misleading explanations (which score very well according to our metrics, i.e., they are sufﬁcient while also being very sparse) but are at the same time not faithful explanations.
Revised: We do not argue that such patterns are non-existent, rather that they are important only when they can be established across a variety of instances and can otherwise create extremely misleading explanations (which score very well according to our metrics, i.e., they are sufﬁcient while also being very sparse) but are at the same time not faithful explanations.

Original: Secondly, E operates in the space of hidden representations, i.e., BERT token embeddings, which arguably capture contextualized morpho-syntactic and semantic information, allowing MAGNE x to learn more general explanation patterns.
Revised: Secondly, E operates in the space of hidden representations, i.e., BERT token embeddings, which arguably capture contextualized morpho-syntactic and semantic information, allowing MAGNE x to learn more general explanation patterns.

Original: 3. Q UESTION A NSWERING
Revised: 3. Q UESTION A NSWERING

Original: For this task we use a large variant of BERT ( 24 layers, 16 attention heads, 1024 hidden size), pretrained and ﬁne-tuned on SQUAD (Rajpurkar et al.
Revised: For this task we use a large variant of BERT ( 24 layers, 16 attention heads, 1024 hidden size), pretrained and ﬁne-tuned on SQUAD (Rajpurkar et al.

Original: The model attempts to predict answer spans within some context given a question.
Revised: The model attempts to predict answer spans within some context given a question.

Original: It achieves an F1 score of 93 .
Revised: It achieves an F1 score of 93 .

Original: 2% and an exact match score of 86 .
Revised: 2% and an exact match score of 86 .

Original: We show the results of various explainers on the task in Table 4.
Revised: We show the results of various explainers on the task in Table 4.

Original: Our explainer continues to produce explanations that have higher sufﬁciency scores than the other methods while also being considerably faster.
Revised: Our explainer continues to produce explanations that have higher sufﬁciency scores than the other methods while also being considerably faster.

Original: A side note here is that for this task sufﬁciency is realized as the
Revised: A side note here is that for this task sufﬁciency is realized as the

Original: Jaccard index.
Revised: Jaccard index.

Original: So between two predicted spans y =
Revised: So between two predicted spans y =

Original: { f 1 , f 2 , . .
Revised: { f 1 , f 2 , . .

Original: f m } and ˆ y
Revised: f m } and ˆ y

Original: = { ˆ f 1 , ˆ f 2 , . . .
Revised: = { ˆ f 1 , ˆ f 2 , . . .

Original: , ˆ f n } , sufﬁciency is measured as su( y, ˆ y ) =
Revised: , ˆ f n } , sufﬁciency is measured as su( y, ˆ y ) =

Original: ( y (cid:83) ˆ y ) /
Revised: ( y (cid:83) ˆ y ) /

Original: ( y (cid:84) ˆ y ) .
Revised: ( y (cid:84) ˆ y ) .

Original: The Jaccard index severely punishes small mistakes.
Revised: The Jaccard index severely punishes small mistakes.

Original: For example, the Jaccard index for the spans ‘Stanford University’ and ‘the Standford
Revised: For example, the Jaccard index for the spans ‘Stanford University’ and ‘the Standford

Original: University’ is 2 / 3 despite the answers being almost identical.
Revised: University’ is 2 / 3 despite the answers being almost identical.

Original: To some extent this explains the large standard deviations (Table 4) reported for all methods.
Revised: To some extent this explains the large standard deviations (Table 4) reported for all methods.

Original: Finally, similarly to the sentiment analysis task, LIME suffers from the hindsight bias problem.
Revised: Finally, similarly to the sentiment analysis task, LIME suffers from the hindsight bias problem.

Original: Figure 4 shows an indicative example where the explanations produced by MAGNE x are more well formed than those of LIME .
Revised: Figure 4 shows an indicative example where the explanations produced by MAGNE x are more well formed than those of LIME .

Original: [Table]
Revised: [Table]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The outburst in the ﬁeld of explainability in machine learning started with Ribeiro et al.
Revised: The outburst in the ﬁeld of explainability in machine learning started with Ribeiro et al.

Original: (2016) who proposed LIME , a model-agnostic perturbation-based algorithm that generates explanations for machine learning models.
Revised: (2016) who proposed LIME , a model-agnostic perturbation-based algorithm that generates explanations for machine learning models.

Original: Since then, apart from perturbation-based methods, gradient based methods were explored (Shrikumar et al., 2017; Sundararajan et al., 2017)).
Revised: Since then, apart from perturbation-based methods, gradient based methods were explored (Shrikumar et al., 2017; Sundararajan et al., 2017)).

Original: More recently, Jain & Wallace (2019) and Wiegreffe & Pinter (2019), driven by the increasing popularity of Transformer-based models (Vaswani et al., 2017) in a number of tasks across modalities (Devlin et al., 2019; Schneider et al., 2019; Dosovitskiy et al., 2021), have initiated the discussion on whether the attention mechanism in these models can provide quality explanations.
Revised: More recently, Jain & Wallace (2019) and Wiegreffe & Pinter (2019), driven by the increasing popularity of Transformer-based models (Vaswani et al., 2017) in a number of tasks across modalities (Devlin et al., 2019; Schneider et al., 2019; Dosovitskiy et al., 2021), have initiated the discussion on whether the attention mechanism in these models can provide quality explanations.

Original: In parallel, a lot of work focused on creating explainable neural networks by-design (Lei et al., 2016; Bastings et al., 2019; Yu et al., 2019; Chang et al., 2019; Chalkidis et al., 2021).
Revised: In parallel, a lot of work focused on creating explainable neural networks by-design (Lei et al., 2016; Bastings et al., 2019; Yu et al., 2019; Chang et al., 2019; Chalkidis et al., 2021).

Original: These systems typically contain two components.
Revised: These systems typically contain two components.

Original: The ﬁrst component selects rationales (subsets of the input) which are then fed to the second component for classiﬁcation.
Revised: The ﬁrst component selects rationales (subsets of the input) which are then fed to the second component for classiﬁcation.

Original: They are typically trained with similar objectives to perturbation-based methods, i.e., sufﬁciency and sparsity, other objectives such ascontinuity Lei et al. and comprehensiveness Yu et al. ; Chalkidis et al.
Revised: They are typically trained with similar objectives to perturbation-based methods, i.e., sufﬁciency and sparsity, other objectives such as

Original: (2021) have also been tested.

Original: However, the initial system (Lei et al., 2016) performed gradient estimation for

Original: [Figure]
Revised: [Figure]

Original: set. Filtering these to produce binary scores as we do during evaluation creates the reduced inputs i) who wrote about the great pest ##ile ##nce in 1893 ?
Revised: ) who wrote about the great pest ##ile ## in 1893 ?

Original: the historianfrancis aidan gas##quet wrote about the ‘ great pest ##ile and ii) who wrote about ##ile##nce 1893 ?
Revised: the historian aidan gas wrote about the ‘ great pest ##ile and ii) who wrote about ##ile

Original: the francis gas##quet wrote great 1893 suggested “ some adopt black interpretation other roman ##1 .
Revised: ## gas wrote great 1893 suggested “ some adopt black interpretation other roman ##1 .

Original: Global context allows MAGNE x to produce comparatively well formed inputs to present to the pre-trained model.
Revised: Global context allows MAGNE x to produce comparatively well formed inputs to present to the pre-trained model.

Revised: continuity Lei et al.

Original:  the rationale extractor with REINFORCE without baseline reduction making the system unstable due to the nature of the learning algorithm.
Revised: and comprehensiveness Yu et al. ; Chalkidis et al. (2021) have also been tested. However, the initial system (Lei et al., 2016) performed gradient estimation for the rationale extractor with REINFORCE without baseline reduction making the system unstable due to the nature of the learning algorithm.

Original: Bastings et al.
Revised: Bastings et al.

Original: (2019) relaxed this binary rationale extraction process by following Louizos et al.
Revised: (2019) relaxed this binary rationale extraction process by following Louizos et al.

Original: and therefore making the objective differentiable.
Revised: and therefore making the objective differentiable.

Original: Nonetheless, all further work done in this area (Yu et al., 2019; Chang et al., 2019) optimizes at least two neural networks concurrently and uses some sort of gradient estimation either through REINFORCE , the reparametrization trick, or straight-trough estimators.
Revised: Nonetheless, all further work done in this area (Yu et al., 2019; Chang et al., 2019) optimizes at least two neural networks concurrently and uses some sort of gradient estimation either through REINFORCE , the reparametrization trick, or straight-trough estimators.

Original: Naturally, these systems are often very hard to train due to their complexity and their innate inability to produce exact gradients due to their stochastic nodes.
Revised: Naturally, these systems are often very hard to train due to their complexity and their innate inability to produce exact gradients due to their stochastic nodes.

Original: Finally, recent work (Jain et al., 2020; Situ et al., 2021) has attempted to use auxiliary explanations generated by some attribution method (such as LIME or integrated gradients) as supervision.
Revised: Finally, recent work (Jain et al., 2020; Situ et al., 2021) has attempted to use auxiliary explanations generated by some attribution method (such as LIME or integrated gradients) as supervision.

Original: For instance, one of the methods explored by Situ et al.
Revised: For instance, one of the methods explored by Situ et al.

Original: (2021) was to train a rationale extractor on explanations produced by LIME .
Revised: (2021) was to train a rationale extractor on explanations produced by LIME .

Original: They also experimented with integrated gradients, sumilarly to Jain et al.
Revised: They also experimented with integrated gradients, sumilarly to Jain et al.

Original: (2020) to train a neural network to create explanations in the post-hoc setting.
Revised: (2020) to train a neural network to create explanations in the post-hoc setting.

Original:  C ONCLUSION & F UTURE W ORK
Revised:  C ONCLUSION & F UTURE W ORK

Original: We introduced MAGNE x, a post-hoc explainability algorithm, that is global and completely modelagnostic.
Revised: We introduced MAGNE x, a post-hoc explainability algorithm, that is global and completely modelagnostic.

Original: We experimentally showed that our approach outperforms popular post-hoc algorithms in terms of the faithfulness of its explanations and in computational complexity across tasks and modalities.
Revised: We experimentally showed that our approach outperforms popular post-hoc algorithms in terms of the faithfulness of its explanations and in computational complexity across tasks and modalities.

Original: In addition, the global nature of MAGNE x seems to alleviate the hindsight bias problem that seemed to trouble local perturbation-based explainability methods.
Revised: In addition, the global nature of MAGNE x seems to alleviate the hindsight bias problem that seemed to trouble local perturbation-based explainability methods.

Original: In the future we aim to pursue a number of interesting directions.
Revised: In the future we aim to pursue a number of interesting directions.

Original: We would like to combine our method with online learning to produce higher quality explanations.
Revised: We would like to combine our method with online learning to produce higher quality explanations.

Original: The explanations learned ofﬂine can be used as a starting point and further search can be performed on a per instance basis.
Revised: The explanations learned ofﬂine can be used as a starting point and further search can be performed on a per instance basis.

Original: Also, a number of improvements have been proposed in recent years for policy gradient methods in reinforcement learning with one prime example being that of Schulman et al. Some of these ideas could be incorporated inan improved version of the score estimator allowing for better optimization. Lastly, this approach can be evaluated in a number of new settings (e.g., visual question answering, summarization, text generation) possibly revealing interesting results.
Revised: We are also planning on performing human evaluation between MAGNE x and its baselines to determine how informative its explanations in the eyes of a human. Lastly, we will explore counterfactual explainability (Goyal et al., 2019; Elazar et al., 2021) and how it can be used to improve both the training regime of MAGNE x as well as our evaluation framework.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS
Revised: A A PPENDIX A. I MPLEMENTATION D ETAILS

Original: We implemented all models in AllenNLP (Gardner et al., 2018) and PyTorch (Paszke et al., 2019).
Revised: We implemented all models in AllenNLP (Gardner et al., 2018) and PyTorch (Paszke et al., 2019).

Original: For experiments with text (Sections 3.4 and 3.5) we used the Transformers library of Huggingface (Wolf et al., 2020).
Revised: For experiments with text (Sections 3.4 and 3.5) we used the Transformers library of Huggingface (Wolf et al., 2020).

Original: For MAGNE x we found automatic tuning to be slow in developement since
Revised: For MAGNE x we found automatic tuning to be slow in developement since

Original: REINFORCE tends to diverge very quickly when the learning rate is non-optimal.
Revised: REINFORCE tends to diverge very quickly when the learning rate is non-optimal.

Original: We therefore manually tuned the hyper-parameters in each experiment by allowing a small number of batches to ﬂow through the model and observing its metrics.
Revised: We therefore manually tuned the hyper-parameters in each experiment by allowing a small number of batches to ﬂow through the model and observing its metrics.

Original: Non-promising runs were terminated very quickly.
Revised: Non-promising runs were terminated very quickly

Original: We performed this procedure for sentiment classiﬁcation and question answering for learning rates ∈ { 10 − 4 , 10 − 5 } .
Revised: We performed this procedure for sentiment classiﬁcation and question answering for learning rates ∈ { 10 − 4 , 10 − 5 } .

Original: The optimal learning rates were 10 − 4 for sentiment classiﬁcation and 10 − 5 for question answering.
Revised: The optimal learning rates were 10 − 4 for sentiment classiﬁcation and 10 − 5 for question answering.

Original: For image classiﬁcation (Section 3.3) we used a learning rate of 10 − 3 .
Revised: For image classiﬁcation (Section 3.3) we used a learning rate of 10 − 3 .

Original: Optimization was carried out with Adam (Kingma & Ba, 2015) throughout the experiments except when ﬁne-tuning BERT -base in sentiment classiﬁcation where AdamW (Loshchilov & Hutter, 2019) was used.
Revised: Optimization was carried out with Adam (Kingma & Ba, 2015) throughout the experiments except when ﬁne-tuning BERT -base in sentiment classiﬁcation where AdamW (Loshchilov & Hutter, 2019) was used.

Original: Finally, for our baselines ( LIME , IG ) we use Captum (Kokhlikyan et al., 2020).
Revised: Finally, for our baselines ( LIME , IG ) we use Captum (Kokhlikyan et al., 2020).

Original: A. S CORE ESTIMATOR P ROOF
Revised: A. S CORE ESTIMATOR P ROOF

Original: We begin by creating a policy parametrized by θ .
Revised: We begin by creating a policy parametrized by θ .

Original: For some input i the policy can be written as:
Revised: For some input i the policy can be written as:

Original: [Equation]
Revised: [Equation]

Original: Our loss is the expected return across samples τ drawn from π θ ( τ ∼ π θ ( i )) with gradient:
Revised: Our loss is the expected return across samples τ drawn from π θ ( τ ∼ π θ ( i )) with gradient:

Original: [Equation]
Revised: [Equation]

Original: The probability of trajectory τ which is a sequence of binary variables being sampled from π θ ( i ) is:
Revised: The probability of trajectory τ which is a sequence of binary variables being sampled from π θ ( i ) is:

Original: [Equation]
Revised: [Equation]

Original: We can rewrite Eq. 13 by taking the deﬁnition of expectation as:
Revised: We can rewrite Eq. 13 by taking the deﬁnition of expectation as:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We can add the gradient in Eq.
Revised: We can add the gradient in Eq.

Original: 15 under the integral:
Revised: 15 under the integral:

Original: [Equation]
Revised: [Equation]

Original: and rewrite Eq.
Revised: and rewrite Eq.

Original: 16 by utilizing log( f ( x )) (cid:48) =
Revised: 16 by utilizing log( f ( x )) (cid:48) =

Original: f (cid:48) ( x ) /f
Revised: f (cid:48) ( x ) /f

Original: ( x ) as:
Revised: ( x ) as:

Original: [Equation]
Revised: [Equation]

Original: By reverting back to expectation form we arrive at:
Revised: By reverting back to expectation form we arrive at:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: This allows us to sample τ ∼ π θ ( i ) and compute the gradient with Monte Carlo Sampling.
Revised: This allows us to sample τ ∼ π θ ( i ) and compute the gradient with Monte Carlo Sampling.

Revised: A. I MAGE C LASSIFICATION R ATIONALES

Revised: We show rationales for MNIST classiﬁcation when the model being explained is a Random Forest (Figure 5) or a CNN (Figure 6).

Revised: MAGNE x produces explanations of higher quality than LIME .

Revised: [Figure]

Revised: a Random Forest.

Revised: [Figure]