Original: S CALING S YMBOLIC M ETHODS USING G RADIENTS FOR N EURAL M ODEL E XPLANATION
Revised: S CALING S YMBOLIC M ETHODS USING G RADIENTS FOR N EURAL M ODEL E XPLANATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Symbolic techniques based on Satisﬁability Modulo Theory (SMT) solvers have been proposed for analyzing and verifying neural network properties, but their usage has been fairly limited owing to their poor scalability with larger networks.
Revised: Symbolic techniques based on Satisﬁability Modulo Theory (SMT) solvers have been proposed for analyzing and verifying neural network properties, but their usage has been fairly limited owing to their poor scalability with larger networks.

Original: In this work we propose a technique for combining gradient-based methods with symbolic techniques to scale such analyses and demonstrate its application for model explanation.
Revised: In this work we propose a technique for combining gradient-based methods with symbolic techniques to scale such analyses and demonstrate its application for model explanation.

Original: In particular, we apply this technique to identify minimal regions in an input that are most relevant for a neural network’s prediction
Revised: In particular, we apply this technique to identify minimal regions in an input that are most relevant for a neural network’s prediction

Original: Our approach uses gradient information (based on Integrated Gradients) to focus on a subset of neurons in the ﬁrst layer, which allows our technique to scale to large networks
Revised: Our approach uses gradient information (based on Integrated Gradients) to focus on a subset of neurons in the ﬁrst layer, which allows our technique to scale to large networks

Original: The corresponding SMT constraints encode the minimal input mask discovery problem such that after masking the input the activations of the selected neurons are still above a threshold
Revised: The corresponding SMT constraints encode the minimal input mask discovery problem such that after masking the input the activations of the selected neurons are still above a threshold

Original: After solving for the minimal masks our approach scores the mask regions to generate a relative ordering of the features within the mask
Revised: After solving for the minimal masks our approach scores the mask regions to generate a relative ordering of the features within the mask

Original: This produces a saliency map which explains “where a model is looking” when making a prediction
Revised: This produces a saliency map which explains “where a model is looking” when making a prediction

Original: We evaluate our technique on three datasets
Revised: We evaluate our technique on three datasets

Original: MNIST
Revised: MNIST

Original: ImageNet, and Beer
Revised: ImageNet, and Beer

Original: Reviews, and demonstrate both quantitatively and qualitatively that the regions generated by our approach are sparser and achieve higher saliency scores compared to the gradient
Revised: Reviews, and demonstrate both quantitatively and qualitatively that the regions generated by our approach are sparser and achieve higher saliency scores compared to the gradient

Original: -based methods alone
Revised: -based methods alone

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Satisﬁability Modulo Theory (SMT) solvers (Barrett & Tinelli, 2018) are routinely used for symbolic modeling and verifying correctness of software programs (Srivastava et al., 2009), and more recently they have also been used for verifying properties of deep neural networks (Katz et al., 2017).
Revised: Satisﬁability Modulo Theory (SMT) solvers (Barrett & Tinelli, 2018) are routinely used for symbolic modeling and verifying correctness of software programs (Srivastava et al., 2009), and more recently they have also been used for verifying properties of deep neural networks (Katz et al., 2017).

Original: SMT
Revised: SMT

Original: Solvers in their current form are difﬁcult to scale to large networks.
Revised: Solvers in their current form are difﬁcult to scale to large networks.

Original: Model explanation is one such domain where SMT solvers have been used but they are limited to very small sized networks (Gopinath et al., 2019; Ignatiev et al., 2019).
Revised: Model explanation is one such domain where SMT solvers have been used but they are limited to very small sized networks (Gopinath et al., 2019; Ignatiev et al., 2019).

Original: The goal of our work is to address the issue of scalability of SMT solvers by using gradient information, thus enabling their use for different applications.
Revised: The goal of our work is to address the issue of scalability of SMT solvers by using gradient information, thus enabling their use for different applications.

Original: In this work, we present a new application of SMT solvers for explaining neural network decisions.
Revised: In this work, we present a new application of SMT solvers for explaining neural network decisions.

Original: Model explanation can be viewed as identifying a minimal set of features in a given input that is critical to a model’s prediction (Carter et al., 2018;
Revised: Model explanation can be viewed as identifying a minimal set of features in a given input that is critical to a model’s prediction (Carter et al., 2018;

Original: Macdonald et al., 2019).
Revised: Macdonald et al., 2019).

Original: Such a problem formulation for identifying a minimal set lends itself to the use of SMT solvers for this task.
Revised: Such a problem formulation for identifying a minimal set lends itself to the use of SMT solvers for this task.

Original: We can encode a neural network using real arithmetic (Katz et al., 2017) and use an SMT solver to optimize over the constraints to identify a minimal set of inputs that can explain the prediction.
Revised: We can encode a neural network using real arithmetic (Katz et al., 2017) and use an SMT solver to optimize over the constraints to identify a minimal set of inputs that can explain the prediction.

Original: However, there are two key challenges in this approach.
Revised: However, there are two key challenges in this approach.

Original: First, we cannot generate reliable explanations based on ﬁnal model prediction as the minimal input is typically out of distribution.
Revised: First, we cannot generate reliable explanations based on ﬁnal model prediction as the minimal input is typically out of distribution.

Original: Second, solving such a formulation is challenging for SMT solvers as the decision procedures for solving these constraints have exponential complexity, and is further exacerbated by the large number of parameters in typical neural network models.
Revised: Second, solving such a formulation is challenging for SMT solvers as the decision procedures for solving these constraints have exponential complexity, and is further exacerbated by the large number of parameters in typical neural network models.

Original: Thus, previous approaches for SMT-based analysis of neural networks have been quite limited, and have only been able to scale to networks with few thousands of parameters.
Revised: Thus, previous approaches for SMT-based analysis of neural networks have been quite limited, and have only been able to scale to networks with few thousands of parameters.

Original: To solve these challenges, instead of doing minimization by encoding the entire network, our approach takes advantage of the gradient information, speciﬁcally Integrated Gradients (IG) (Sundararajan et al., 2017), in lieu of encoding the deeper layers, and encodes a much simpler set of linear constraints pertaining to the layer closest to the input.
Revised: To solve these challenges, instead of doing minimization by encoding the entire network, our approach takes advantage of the gradient information, speciﬁcally Integrated Gradients (IG) (Sundararajan et al., 2017), in lieu of encoding the deeper layers, and encodes a much simpler set of linear constraints pertaining to the layer closest to the input.

Original: We encode the mathematical equations of a neural network as SMT constraints using the theory of Linear Real Arithmetic (LRA), and use z3 solver (Bjørner et al., 2015) as it additionally supports optimization constraints such as minimization.
Revised: We encode the mathematical equations of a neural network as SMT constraints using the theory of Linear Real Arithmetic (LRA), and use z3 solver (Bjørner et al., 2015) as it additionally supports optimization constraints such as minimization.

Original: The SMT solver then ﬁnds a minimal subset (of input features) by performing minimization on these equations.
Revised: The SMT solver then ﬁnds a minimal subset (of input features) by performing minimization on these equations.

Original: Thus, our approach, which we refer to as SMUG, is able to scale S ymbolic M ethods U sing G radient information while still providing a faithful explanation of the neural network’s decision.
Revised: Thus, our approach, which we refer to as SMUG, is able to scale S ymbolic M ethods U sing G radient information while still providing a faithful explanation of the neural network’s decision.

Original: SMUG is built on two properties.
Revised: SMUG is built on two properties.

Original: First, based on the target prediction, SMUG uses gradient information propagated from the deeper layers to identify neurons that are important in the ﬁrst layer, and only encodes those.
Revised: First, based on the target prediction, SMUG uses gradient information propagated from the deeper layers to identify neurons that are important in the ﬁrst layer, and only encodes those.

Original: For this, we use IG (Sundararajan et al., 2017) instead of relying on gradients alone.
Revised: For this, we use IG (Sundararajan et al., 2017) instead of relying on gradients alone.

Original: Second, for the optimization, a set of input pixels are determined to be relevant for prediction if they are able to activate the neurons deemed important, and maintain a fraction of their activation as the original (full) input image.
Revised: Second, for the optimization, a set of input pixels are determined to be relevant for prediction if they are able to activate the neurons deemed important, and maintain a fraction of their activation as the original (full) input image.

Original: Empirically, we observe good performance on visual and text classiﬁcation tasks.
Revised: Empirically, we observe good performance on visual and text classiﬁcation tasks.

Original: We evaluate SMUG on three datasets: MNIST (LeCun et al., 2010), ImageNet (Deng et al., 2009), and Beer Reviews (McAuley et al., 2012).
Revised: We evaluate SMUG on three datasets: MNIST (LeCun et al., 2010), ImageNet (Deng et al., 2009), and Beer Reviews (McAuley et al., 2012).

Original: We show that we can fully encode the minimal feature identiﬁcation problem for a small feedforward network (without gradient-based neuron selection) for MNIST, but this full SMT encoding scales poorly for even intermediate sized networks.
Revised: We show that we can fully encode the minimal feature identiﬁcation problem for a small feedforward network (without gradient-based neuron selection) for MNIST, but this full SMT encoding scales poorly for even intermediate sized networks.

Original: On
Revised: On

Original: ImageNet, we observe that our method performs better than Integrated Gradients (Sundararajan et al., 2017) and several strong baselines.
Revised: ImageNet, we observe that our method performs better than Integrated Gradients (Sundararajan et al., 2017) and several strong baselines.

Original: Additionally, we observe that our approach ﬁnds signiﬁcantly sparser masks (on average 17% of the original image size).
Revised: Additionally, we observe that our approach ﬁnds signiﬁcantly sparser masks (on average 17% of the original image size).

Original: Finally, we also show that our technique is applicable to text models where it performs competitively with other methods including SIS (Carter et al., 2018) and Integrated Gradients (Sundararajan et al., 2017).
Revised: Finally, we also show that our technique is applicable to text models where it performs competitively with other methods including SIS (Carter et al., 2018) and Integrated Gradients (Sundararajan et al., 2017).

Original: This paper makes the following key contributions:
Revised: This paper makes the following key contributions:

Original: • We present a technique (SMUG) to encode the minimal input feature discovery problem for neural model explanation using SMT solvers.
Revised: • We present a technique (SMUG) to encode the minimal input feature discovery problem for neural model explanation using SMT solvers.

Original: • Our approach uses gradient information to scale SMT-based analysis of neural networks to larger models and input features.
Revised: • Our approach uses gradient information to scale SMT-based analysis of neural networks to larger models and input features.

Original: Further, it also overcomes the issue of choosing a “baseline” parameter for Integrated Gradients (Kapishnikov et al., 2019; Sturmfels et al., 2020).
Revised: Further, it also overcomes the issue of choosing a “baseline” parameter for Integrated Gradients (Kapishnikov et al., 2019; Sturmfels et al., 2020).

Original: • We empirically evaluate SMUG on image and text datasets, and show that the minimal features identiﬁed by it are both quantitatively and qualitatively better than several baselines.
Revised: • We empirically evaluate SMUG on image and text datasets, and show that the minimal features identiﬁed by it are both quantitatively and qualitatively better than several baselines.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: SMT based symbolic techniques have been used for verifying neural network properties (Huang et al., 2017; Katz et al., 2017).
Revised: SMT based symbolic techniques have been used for verifying neural network properties (Huang et al., 2017; Katz et al., 2017).

Original: Reluplex (Katz et al., 2017) extends the simplex method to handle ReLU functions by leveraging its piecewise linear property and presents an iterative procedure for gradual satisfaction of the constraints.
Revised: Reluplex (Katz et al., 2017) extends the simplex method to handle ReLU functions by leveraging its piecewise linear property and presents an iterative procedure for gradual satisfaction of the constraints.

Original: (Huang et al., 2017) proposes a layer-wise analysis using a reﬁnementbased approach with SMT solvers for verifying the absence of adversarial input perturbations.
Revised: (Huang et al., 2017) proposes a layer-wise analysis using a reﬁnementbased approach with SMT solvers for verifying the absence of adversarial input perturbations.

Original: (Zhang et al., 2018) present a linear programming (LP) formulation again using the piecewise linear property of ReLU to ﬁnd minimal changes to an input to change a network’s classiﬁcation decisions.
Revised: (Zhang et al., 2018) present a linear programming (LP) formulation again using the piecewise linear property of ReLU to ﬁnd minimal changes to an input to change a network’s classiﬁcation decisions.

Original: (Gopinath et al., 2019) uses Reluplex to learn input properties in the form of convex predicates over neuron activations, which in turn capture different behaviors of a neural network.
Revised: (Gopinath et al., 2019) uses Reluplex to learn input properties in the form of convex predicates over neuron activations, which in turn capture different behaviors of a neural network.

Original: While these techniques have shown promising results, scaling these approaches for larger neural networks and performing richer analysis based on global input features still remains a challenge.
Revised: While these techniques have shown promising results, scaling these approaches for larger neural networks and performing richer analysis based on global input features still remains a challenge.

Original: While the above SMT based techniques focus on verifying properties of deep networks, our work focuses on applying symbolic techniques to the related task of model explanation, i.e. to say where a model is “looking”, by solving for the input features responsible for a model’s prediction.
Revised: While the above SMT based techniques focus on verifying properties of deep networks, our work focuses on applying symbolic techniques to the related task of model explanation, i.e. to say where a model is “looking”, by solving for the input features responsible for a model’s prediction.

Original: Some explanation techniques are model agnostic (i.e., black-box) while others are back-propagation based.
Revised: Some explanation techniques are model agnostic (i.e., black-box) while others are back-propagation based.

Original: Model agnostic (black-box) explanation techniques such as SIS, L IME (Alvarez-Melis & Jaakkola, 2018;
Revised: Model agnostic (black-box) explanation techniques such as SIS, L IME (Alvarez-Melis & Jaakkola, 2018;

Original: Carter et al., 2018) have a similar formulation of the problem as ours in the sense that they perturb the input pixels by masking them and optimize to identify minimal regions affecting the performance of the model.
Revised: Carter et al., 2018) have a similar formulation of the problem as ours in the sense that they perturb the input pixels by masking them and optimize to identify minimal regions affecting the performance of the model.

Original: This formulation can lead to evaluating the model on out of distribution samples (Hooker et al., 2019) with potential for adversarial attacks (Slack et al., 2020).
Revised: This formulation can lead to evaluating the model on out of distribution samples (Hooker et al., 2019) with potential for adversarial attacks (Slack et al., 2020).

Original: In contrast, back-propagation based methods (Bach et al., 2015; Sundararajan et al., 2017; Selvaraju et al., 2016) examine the gradients of the model with respect to an input instance to determine pixel attribution.
Revised: In contrast, back-propagation based methods (Bach et al., 2015; Sundararajan et al., 2017; Selvaraju et al., 2016) examine the gradients of the model with respect to an input instance to determine pixel attribution.

Original: Our work builds on the IG method (Sundararajan et al., 2017).
Revised: Our work builds on the IG method (Sundararajan et al., 2017).

Original: IG integrates gradients along the “intensity” path where the input (image or text embedding) is scaled from an information-less baseline
Revised: IG integrates gradients along the “intensity” path where the input (image or text embedding) is scaled from an information-less baseline

Original: (all zeros input, e.g., all black or random noise image) to a speciﬁc instance.
Revised: (all zeros input, e.g., all black or random noise image) to a speciﬁc instance.

Original: This helps the model determine attribution at the pixel level.
Revised: This helps the model determine attribution at the pixel level.

Original: In our work, we use IG to determine important nodes in the ﬁrst layer (closest to the input).
Revised: In our work, we use IG to determine important nodes in the ﬁrst layer (closest to the input).

Original: The key improvement of our technique over IG is that, by using IG only on the ﬁrst layer and then using SMT solver based solution to determine saliency on the input, we not only preserve faithfulness, but also overcome the issue of choosing an appropriate baseline image for IG (Kapishnikov et al., 2019; Sturmfels et al., 2020; Xu et al., 2020).
Revised: The key improvement of our technique over IG is that, by using IG only on the ﬁrst layer and then using SMT solver based solution to determine saliency on the input, we not only preserve faithfulness, but also overcome the issue of choosing an appropriate baseline image for IG (Kapishnikov et al., 2019; Sturmfels et al., 2020; Xu et al., 2020).

Original:  M ETHOD : S CALING S YMBOLIC M ETHODS U SING G RADIENTS (SMUG)
Revised:  M ETHOD : S CALING S YMBOLIC M ETHODS U SING G RADIENTS (SMUG)

Original: We describe our approach SMUG, which combines attribution based on gradient information with an SMT-based encoding of the minimal input feature identiﬁcation problem.
Revised: We describe our approach SMUG, which combines attribution based on gradient information with an SMT-based encoding of the minimal input feature identiﬁcation problem.

Original: We then show how to generate saliency maps from the predicted boolean mask for image and text applications.
Revised: We then show how to generate saliency maps from the predicted boolean mask for image and text applications.

Original: 3. C HALLENGES WITH ENCODING NEURAL NETWORKS FOR EXPLANATION
Revised: 3. C HALLENGES WITH ENCODING NEURAL NETWORKS FOR EXPLANATION

Original: Previous works (Katz et al., 2017) have shown it’s possible to use SMT solvers to encode the semantics of neural networks with speciﬁc activation functions, while others (Carter et al., 2018;
Revised: Previous works (Katz et al., 2017) have shown it’s possible to use SMT solvers to encode the semantics of neural networks with speciﬁc activation functions, while others (Carter et al., 2018;

Original: Macdonald et al., 2019) have used the idea of minimality to identify importance of inputs in the context of model explanation.
Revised: Macdonald et al., 2019) have used the idea of minimality to identify importance of inputs in the context of model explanation.

Original: In particular, given a neural network N θ ( θ denotes the parameters), let
Revised: In particular, given a neural network N θ ( θ denotes the parameters), let

Original: X ∈ R m × n denote an input image with m × n pixels, M ∈ { 0 , 1 } m × n an unknown binary mask, and L i ( · ) the output (i.e., activations) of the i th layer.
Revised: X ∈ R m × n denote an input image with m × n pixels, M ∈ { 0 , 1 } m × n an unknown binary mask, and L i ( · ) the output (i.e., activations) of the i th layer.

Original: Also, let L | L |− 1 ( N θ ( X )) denote the logits of the ﬁnal layer.
Revised: Also, let L | L |− 1 ( N θ ( X )) denote the logits of the ﬁnal layer.

Original: Previous works (Carter et al., 2018;
Revised: Previous works (Carter et al., 2018;

Original: Macdonald et al., 2019) have used the following formulation for explanation.
Revised: Macdonald et al., 2019) have used the following formulation for explanation.

Original: [Equation]
Revised: [Equation]

Original: argmax L | L |− 1 ( N θ ( M (cid:12) X ))
Revised: argmax L | L |− 1 ( N θ ( M (cid:12) X ))

Original: However, there are two key problems with this formulation.
Revised: However, there are two key problems with this formulation.

Original: First, the masked inputs found by solving this formulation are typically out of training distribution, so explanation techniques based on model conﬁdence over a forward pass of the masked inputs (such as the above formulation) no longer produce reliable explanations.
Revised: First, the masked inputs found by solving this formulation are typically out of training distribution, so explanation techniques based on model conﬁdence over a forward pass of the masked inputs (such as the above formulation) no longer produce reliable explanations.

Original: Second, using such a formulation directly with SMT solvers is not tractable as the number of constraints in the encoding grows linearly with increasing network size and SMT decision procedure for Non-linear Real Arithmetic is doubly exponential.
Revised: Second, using such a formulation directly with SMT solvers is not tractable as the number of constraints in the encoding grows linearly with increasing network size and SMT decision procedure for Non-linear Real Arithmetic is doubly exponential.

Original: Thus, when we apply such an encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4, Appendix.
Revised: Thus, when we apply such an encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4, Appendix.

Original: A).
Revised: A).

Original: 3. O UR S OLUTION : C OMBINING G RADIENTS WITH S YMBOLIC E NCODING
Revised: 3. O UR S OLUTION : C OMBINING G RADIENTS WITH S YMBOLIC E NCODING

Original: Our proposal is to use gradient based attributions to overcome both these issues: 1) as a proxy for the deeper layers (for scalability), and 2) to identify a subset of neurons in the ﬁrst layer capturing information crucial for the model’s prediction, and encode them using SMT which avoids performing a forward pass on out-of-distribution inputs.
Revised: Our proposal is to use gradient based attributions to overcome both these issues: 1) as a proxy for the deeper layers (for scalability), and 2) to identify a subset of neurons in the ﬁrst layer capturing information crucial for the model’s prediction, and encode them using SMT which avoids performing a forward pass on out-of-distribution inputs.

Original: Speciﬁcally, we use Integrated Gradients (IG) (Sundararajan et al., 2017) to score the neurons in order of relevance by treating the ﬁrst layer activations ( L 1 ) as an input to the subsequent network.
Revised: Speciﬁcally, we use Integrated Gradients (IG) (Sundararajan et al., 2017) to score the neurons in order of relevance by treating the ﬁrst layer activations ( L 1 ) as an input to the subsequent network.

Original: This assigns an attribution score to each hidden node where a node with a positive score means that the information captured by that neuron is relevant to the prediction and a node with a negative / zero score is considered irrelevant.
Revised: This assigns an attribution score to each hidden node where a node with a positive score means that the information captured by that neuron is relevant to the prediction and a node with a negative / zero score is considered irrelevant.

Original: More speciﬁcally, suppose F : R a × b × c → [0 , 1] d represents a deep network and x represents the input.
Revised: More speciﬁcally, suppose F : R a × b × c → [0 , 1] d represents a deep network and x represents the input.

Revised: F often refers to the full network (from the input to the ﬁnal softmax) but in our case, it refers to the nodes in the network second through the ﬁnal layer.

Revised: a, b, and c refer to the dimensions of the tensor input to the network (analogous to width, height, and channels) and d is the number of output nodes in the ﬁnal softmax layer (number of classes).

Original: Integrated gradients are obtained by accumulating the gradients at all points along the straight line path from an “information-less” baseline x (cid:48) to the input x .
Revised: Integrated gradients are obtained by accumulating the gradients at all points along the straight line path from an “information-less” baseline x (cid:48) to the input x .

Original: The information-less baseline in this case is an all zeros tensor.
Revised: The information-less baseline in this case is an all zeros tensor.

Original: The path can be parameterized as g ( 
Revised: The path can be parameterized as g ( α ) =

Original: [Equation]
Revised: x (cid:48) + α · ( x − x (cid:48) ) .

Original: IG is then given by Eq. 1.
Revised: IG is then given by Eq. 1.

Original: [Equation]
Revised: [Equation]

Original: Top-k Neurons.
Revised: Top-k Neurons.

Original: The second key idea of our approach is that we only consider activations with the highest positive attributions.
Revised: The second key idea of our approach is that we only consider activations with the highest positive attributions.

Original: Empirically, we still observe scaling issues with SMT when considering all ﬁrst layer neurons ( L 1 ) with +ve attributions, so a method for picking a subset of neurons (top-k attributions) is important for practical application.
Revised: Empirically, we still observe scaling issues with SMT when considering all ﬁrst layer neurons ( L 1 ) with +ve attributions, so a method for picking a subset of neurons (top-k attributions) is important for practical application.

Original: With these two ideas, we can now formulate our problem as follows.
Revised: With these two ideas, we can now formulate our problem as follows.

Original: Let, D k represent the set of k important nodes with the highest attributions in IG ( N θ ( X )) , and γ be a parameter which regulates how “active” the neurons remain after masking..
Revised: Let, D k represent the set of k important nodes with the highest attributions in IG ( N θ ( X )) , and γ be a parameter which regulates how “active” the neurons remain after masking 1 .The

Original: The goal now is to learn a mask M such that:
Revised: goal now is to learn a mask M such that:

Original: [Equation]
Revised: [Equation]

Original: IG ensures that the information captured by the neurons in D k are relevant for the model’s prediction.
Revised: IG ensures that the information captured by the neurons in D k are relevant for the model’s prediction.

Original: Thus, the set of features corresponding to these neurons contributes towards the relevant information.
Revised: Thus, the set of features corresponding to these neurons contributes towards the relevant information.

Original: Let’s denote this set as S .
Revised: Let’s denote this set as S .

Original: To ﬁlter less relevant features from S we introduce a notion of minimization.
Revised: To ﬁlter less relevant features from S we introduce a notion of minimization.

Original: Thus, we deﬁne "relevant features" as a minimal subset of features in S that causes the nodes in D k to remain active.
Revised: Thus, we deﬁne "relevant features" as a minimal subset of features in S that causes the nodes in D k to remain active.

Original: The notion being - if for a masked input the "important nodes" (for the original input) are active then it means that the information relevant for the prediction of the original image is contained in the mask.
Revised: The notion being - if for a masked input the "important nodes" (for the original input) are active then it means that the information relevant for the prediction of the original image is contained in the mask.

Original: Thus, the mask highlights features relevant for the model’s prediction.
Revised: Thus, the mask highlights features relevant for the model’s prediction.

Original: 3. SMT FORMULATION OF M INIMAL I NPUT M ASK D ISCOVERY P ROBLEM
Revised: 3. SMT FORMULATION OF M INIMAL I NPUT M ASK D ISCOVERY P ROBLEM

Original: Given D k , a set of k neurons with highest positive attributions in IG ( N θ ( x )) , our goal is to ﬁnd a minimal mask such that the activations of these neurons are above some threshold ( γ ) times their original activation values.
Revised: Given D k , a set of k neurons with highest positive attributions in IG ( N θ ( X )) , our goal is to ﬁnd a minimal mask such that the activations of these neurons are above some threshold ( γ ) times their original activation values.

Original: Eq.
Revised: Eq.

Original: 3 shows the constraints for a minimal mask.
Revised: 3 shows the constraints for a minimal mask.

Original: The ﬁrst set of constraint speciﬁes that the unknown mask variable M can only have 0 and 1 as possible entries in the matrix.
Revised: The ﬁrst set of constraint speciﬁes that the unknown mask variable M can only have 0 and 1 as possible entries in the matrix.

Original: The second and third set of constraints encode the activation values of the ﬁrst layer of network with corresponding masked and original inputs respectively.
Revised: The second and third set of constraints encode the activation values of the ﬁrst layer of network with corresponding masked and original inputs respectively.

Original: The fourth set of constraint states that the activations of these k neurons should be at least γ times the original activation values, and the ﬁnal constraint adds the optimization constraint to minimize the sum of all the mask bits.
Revised: The fourth set of constraint states that the activations of these k neurons should be at least γ times the original activation values, and the ﬁnal constraint adds the optimization constraint to minimize the sum of all the mask bits.

Original: Note that here we show the formulation for a feedforward network and a input X with 2 channels, but it can be extended to convolutional networks and an input with 3 rd channel as well in a straightforward manner where the mask variables across the same channel share the same mask variable.
Revised: Note that here we show the formulation for a feedforward network and a input X with 2 channels, but it can be extended to convolutional networks and an input with 3 rd channel as well in a straightforward manner where the mask variables across the same channel share the same mask variable.

Original: [Equation]
Revised: [Equation]

Original: 3. C ONSTRUCTING S ALIENCY M AP FROM B INARY M ASK
Revised: 3. C ONSTRUCTING S ALIENCY M AP FROM B INARY M ASK

Original: The SMT solver generates a minimal binary input mask by solving the constraints shown in Eq. 3.
Revised: The SMT solver generates a minimal binary input mask by solving the constraints shown in Eq. 3.

Original: We further use the attribution scores obtained from IG to assign importance scores to each mask pixel.
Revised: We further use the attribution scores obtained from IG to assign importance scores to each mask pixel.

Original: A mask variable M ij that is assigned a value of 1 by SMT is assigned a score s ij computed as:
Revised: A mask variable M ij that is assigned a value of 1 by SMT is assigned a score s ij computed as:

Original: [Equation]
Revised: [Equation]

Original: where α ( o p ) denotes the attribution score assigned by IG for neuron o p and the indicator function denotes that pixel x ij is present in the receptive ﬁeld of o p , i.e. it is present in the linear SMT equation used to compute o n .
Revised: where α ( o p ) denotes the attribution score assigned by IG for neuron o p and the indicator function denotes that pixel x ij is present in the receptive ﬁeld of o p , i.e. it is present in the linear SMT equation used to compute o n .

Original: These scores are then used to compute a continuous saliency map for an input (Appendix. 9).
Revised: These scores are then used to compute a continuous saliency map for an input (Appendix. 9).

Original: Finally, to amplify the pixel differences for visualization purposes in gray scale, we scale the non-zero score values between 0 . 5 and 1 .
Revised: Finally, to amplify the pixel differences for visualization purposes in gray scale, we non between 0 . 5 and 1 .

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Datasets.
Revised: Datasets.

Original: We empirically evaluate SMUG on two image datasets, MNIST (LeCun et al., 2010), and ImageNet (Deng et al., 2009), as well as a text dataset of Beer Reviews from (McAuley et al., 2012).
Revised: We empirically evaluate SMUG on two image datasets, MNIST (LeCun et al., 2010), and ImageNet (Deng et al., 2009), as well as a text dataset of Beer Reviews from (McAuley et al., 2012).

Original: MNIST:
Revised: MNIST:

Original: We use the MNIST dataset to show the scalability of the full network encoding in SMT (presented in Section 3.1).
Revised: We use the MNIST dataset to show the scalability of the full network encoding in SMT (presented in Section 3.1).

Original: We use a feedforward model consisting of one hidden layer with 32 nodes (ReLU activation) and 10 output nodes with sigmoid, one each for 10 digits (0 - 9).
Revised: We use a feedforward model consisting of one hidden layer with 32 nodes (ReLU activation) and 10 output nodes with sigmoid, one each for 10 digits (0 - 9).

Original: For 100 images chosen randomly from the validation set, the SMT solver could solve the constraint shown in Eq. (returns S AT ) for only 41 of the images.
Revised: For 100 images chosen randomly from the validation set, the SMT solver could solve the constraint shown in Eq.(returns S AT ) for only 41 of the images.

Original: For the remaining 59 images, the solver returns U NKNOWN , which means the given set of constraints was too difﬁcult for the solver to solve.
Revised: For the remaining 59 images, the solver returns U NKNOWN , which means the given set of constraints was too difﬁcult for the solver to solve.

Original: ImageNet: We use 3304 images ( 224 × 224 ) with ground truth bounding boxes from the validation set of ImageNet.
Revised: ImageNet: We use 3304 images ( 224 × 224 ) with ground truth bounding boxes from the validation set of ImageNet.

Original: The images for which the model classiﬁcation was correct and a ground truth bounding box annotation for the object class was available were chosen.
Revised: The images for which the model classiﬁcation was correct and a ground truth bounding box annotation for the object class was available were chosen.

Original: We use the Inception-v model from (Szegedy et al., 2015) which classiﬁes images into one of the 1000 ImageNet classes.
Revised: We use the Inception-v model from (Szegedy et al., 2015) which classiﬁes images into one of the 1000 ImageNet classes.

Original: Beer Reviews: To evaluate SMUG on a textual task we consider the review rating prediction task on the Beer Reviews dataset consisting of 70 k training examples, 3 k validation and 7 k test examples.
Revised: Beer Reviews: To evaluate SMUG on a textual task we consider the review rating prediction task on the Beer Reviews dataset consisting of 70 k training examples, 3 k validation and 7 k test examples.

Original: Additionally, the dataset comes with ground truth annotations where humans provide the rationale (select words) that correspond to the rating and review.
Revised: Additionally, the dataset comes with ground truth annotations where humans provide the rationale (select words) that correspond to the rating and review.

Original: We train a 1D CNN model to predict the rating for the aroma of the beer on a scale from 0 to 1.
Revised: We train a 1D CNN model to predict the rating for the aroma of the beer on a scale from 0 to 1.

Original: This model is identical to the model used in (Carter et al., 2018) and consists of a convolution layer with 128 kernels followed by a ReLU, a fully connected layer, and a sigmoid.
Revised: This model is identical to the model used in (Carter et al., 2018) and consists of a convolution layer with 128 kernels followed by a ReLU, a fully connected layer, and a sigmoid.

Original: It achieves a validation mean square error of 0 . 032 .
Revised: It achieves a validation mean square error of 0 . 032 .

Original: Metrics.
Revised: Metrics.

Original: Assessing the quality of the saliency maps, especially binary masks, is challenging.
Revised: Assessing the quality of the saliency maps, especially binary masks, is challenging.

Original: The change in conﬁdence of the classiﬁer (between the original and masked) image alone may not be a reliable measure since the masked input could fall out of the training distribution (Hooker et al., 2019).
Revised: The change in conﬁdence of the classiﬁer (between the original and masked) image alone may not be a reliable measure since the masked input could fall out of the training distribution (Hooker et al., 2019).

Original: Instead, we use the metric proposed in (Dabkowski & Gal, 2017) shown in Eq.
Revised: Instead, we use the metric proposed in (Dabkowski & Gal, 2017) shown in Eq.

Original: This metric, which we term Log Sparsity Conﬁdence difference ( LSC ) score, ﬁrst ﬁnds the tightest bounding box that captures the entire mask, then computes conﬁdence on the cropped box resized to the original image size (we use bilinear interpolation).
Revised: This metric, which we term Log Sparsity Conﬁdence difference ( LSC ) score, ﬁrst ﬁnds the tightest bounding box that captures the entire mask, then computes conﬁdence on the cropped box resized to the original image size (we use bilinear interpolation).

Original: This not only helps keep images closer to the training distribution, but also helps evaluate explanations without the need for groundtruth annotations.
Revised: This not only helps keep images closer to the training distribution, but also helps evaluate explanations without the need for groundtruth annotations.

Original: The
Revised: The

Original: LSC score is computed as:
Revised: LSC score is computed as:

Original: [Equation]
Revised: [Equation]

Original: where a is the fractional area of the rectangular cropped image and c is the conﬁdence of the classiﬁer for the true label on the cropped image.
Revised: where a is the fractional area of the rectangular cropped image and c is the conﬁdence of the classiﬁer for the true label on the cropped image.

Original: A saliency map that is compact and allows the model to still recognize the object would result in a lower LSC score.
Revised: A saliency map that is compact and allows the model to still recognize the object would result in a lower LSC score.

Original: LSC captures model conﬁdence as well as compactness of the identiﬁed salient regions, both of which are desirable when evaluating an explanation.
Revised: LSC captures model conﬁdence as well as compactness of the identiﬁed salient regions, both of which are desirable when evaluating an explanation.

Original: The compactness in particular also makes it suitable for evaluating SMT based methods for the effect of minimization.
Revised: The compactness in particular also makes it suitable for evaluating SMT based methods for the effect of minimization.

Original: We adapt LSC to also assess continuous valued saliency maps by, 1) setting a threshold on the continuous valued saliency map to convert them to a binary mask and 2) iterating over multiple thresholds (in steps) to identify the one that results in the best LSC score.
Revised: We adapt LSC to also assess continuous valued saliency maps by, 1) setting a threshold on the continuous valued saliency map to convert them to a binary mask and 2) iterating over multiple thresholds (in steps) to identify the one that results in the best LSC score.

Original: We also report the fraction of images for which the mask generated by a given method is better (i.e. produces an equal or lower LSC score) than other methods, which we refer to as Win% .
Revised: We also report the fraction of images for which the mask generated by a given method is better (i.e. produces an equal or lower LSC score) than other methods, which we refer to as Win% .

Original: Comparison methods.
Revised: Comparison methods.

Original: The ﬁnal saliency mask for SMUG comes from Eq.
Revised: The ﬁnal saliency mask for SMUG comes from Eq.

Original: 4 (Sec. 3.4).
Revised: 4 (Sec. 3.4).

Original: We compare this to the saliency maps, and bounding boxes from several baselines described below.
Revised: We compare this to the saliency maps, and bounding boxes from several baselines described below.

Original: SMUG base is a variant of SMUG that does not peform SMT-based minimization.
Revised: SMUG base is a variant of SMUG that does not peform SMT-based minimization.

Original: Here, in Eq. 2, we simply set M ij = 1 for each pixel x ij that is in the receptive ﬁelds of the top- k neurons in the ﬁrst layer ( L 1 ) selected by IG.
Revised: Here, in Eq. 2, we simply set M ij = 1 for each pixel x ij that is in the receptive ﬁelds of the top- k neurons in the ﬁrst layer ( L 1 ) selected by IG.

Original: We note here that in case of both SMUG and SMUG base in the formulations in Eq. 2 and 3, we set k = 3000 , γ = 0 for ImageNet, and k = 100 , γ = 0 for text experiments (this choice is explored more in the supplementary material).
Revised: We note here that in case of both SMUG and SMUG base in the formulations in Eq. 2 and 3, we set k = 3000 , γ = 0 for ImageNet, and k = 100 , γ = 0 for text experiments (this choice is explored more in the supplementary material).

Original: IG corresponds to Integrated Gradients (Sundararajan et al., 2017) with the black image as a baseline.
Revised: IG corresponds to Integrated Gradients (Sundararajan et al., 2017) with the black image as a baseline.

Original: G RAD C AM (Selvaraju et al., 2017) uses a weighted average of the CNN ﬁlters for saliency, with weights informed by the gradients.
Revised: G RAD C AM (Selvaraju et al., 2017) uses a weighted average of the CNN ﬁlters for saliency, with weights informed by the gradients.

Original: SIS refers to Sufﬁcient Input Subset (Carter et al., 2018), which ﬁnds multiple disjoint subsets of input features (in decreasing order of relevance) which individually allow a conﬁdent classiﬁcation.
Revised: SIS refers to Sufﬁcient Input Subset (Carter et al., 2018), which ﬁnds multiple disjoint subsets of input features (in decreasing order of relevance) which individually allow a conﬁdent classiﬁcation.

Original: SIS did not scale for ImageNet and we only compare against it on the text dataset.
Revised: SIS did not scale for ImageNet and we only compare against it on the text dataset.

Original: G ROUND T RUTH corresponds to the baseline that uses human annotated bounding box, which capture the object corresponding to the image label.
Revised: G ROUND T RUTH corresponds to the baseline that uses human annotated bounding box, which capture the object corresponding to the image label.

Original: M AX B OX denotes maximal mask spanning the entire image.
Revised: M AX B OX denotes maximal mask spanning the entire image.

Original: C ENTER B OX uses a bounding box placed at the center of the image covering half of the image area.
Revised: C ENTER B OX uses a bounding box placed at the center of the image covering half of the image area.

Original: O PT B OX refers to a bounding box that approximtely optimizes for LSC.
Revised: O PT B OX refers to a bounding box that approximtely optimizes for LSC.

Original: The saliency metric in Eq.relies on ﬁnding a single bounding box for an image.
Revised: The saliency metric in Eq.relies on ﬁnding a single bounding box for an image.

Original: To ﬁnd a box that directly maximizes the metric, we ﬁrst discretize the image into subgrids of 10 × 10 pixels; and then perform a brute force search by selecting 2 points on the grid (to represent opposite corners of a rectangle) and identify a subgrid with the best score.
Revised: To ﬁnd a box that directly maximizes the metric, we ﬁrst discretize the image into subgrids of 10 × 10 pixels; and then perform a brute force search by selecting 2 points on the grid (to represent opposite corners of a rectangle) and identify a subgrid with the best score.

Original:  R ESULTS 5. I MAGE N ET
Revised:  R ESULTS 5. I MAGE N ET

Original: As mentioned previously

Original: [Table] when computing masks using SMUG [Table] for ImageNet we set k = 3000 in
Revised: As mentioned previously, when computing masks using SMUG, for ImageNet we set k = 3000 in

Original: Eq. 2 and 3.
Revised: Eq. 2 and 3.

Original: Further, each masking variable M ij is used to represent a 4 × 4 grid of pixels instead of a single pixel (to reduce running time).
Revised: Further, each masking variable M ij is used to represent a 4 × 4 grid of pixels instead of a single pixel (to reduce running time).

Original: Table 1 presents quantitative results reporting the median
Revised: Table 1 presents quantitative results reporting the median

Original: LSC score and Win% values.
Revised: LSC score and Win% values.

Original: Fig.
Revised: Fig.

Original: 1 present qualitative examples
Revised: 1 present qualitative examples

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Figure]

Original: IG vs SMUG and SMUG base .
Revised: IG vs SMUG and SMUG base .

Original: From Table 1, we observe that SMUG and SMUG base achieve a signiﬁcantly better score ( − 1 . 26 and − 1 .
Revised: From Table 1, we observe that SMUG and SMUG base achieve a signiﬁcantly better score ( − 1 . 26 and − 1 .

Original: 23 resp.)
Revised: 23 resp.)

Original: compared to IG ( − 0 . 34 ).
Revised: compared to IG ( − 0 . 34 ).

Original: As observed in some qualitative examples (Fig. 1), SMUG tends to assign high scores to a much more localized set of pixels whereas IG distributes high scores more widely (spatially).
Revised: As observed in some qualitative examples (Fig. 1), SMUG tends to assign high scores to a much more localized set of pixels whereas IG distributes high scores more widely (spatially).

Original: As LSC metric favors compactness, which is desirable for human interpretability, it results in better scores for SMUG and SMUG base .
Revised: As LSC metric favors compactness, which is desirable for human interpretability, it results in better scores for SMUG and SMUG base .

Original: Choice of baseline for IG.
Revised: Choice of baseline for IG.

Original: Another reason why SMUG base and SMUG outperform IG is that, they apply IG to the ﬁrst layer of the network (as opposed to the input/image layer).
Revised: Another reason why SMUG base and SMUG outperform IG is that, they apply IG to the ﬁrst layer of the network (as opposed to the input/image layer).

Original: IG attribution in the pixel space is known to be noisy (Smilkov et al., 2017), further attributions produced by IG depend on the choice of the baseline (Kapishnikov et al., 2019; Xu et al., 2020; Sturmfels et al., 2020).
Revised: IG attribution in the pixel space is known to be noisy (Smilkov et al., 2017), further attributions produced by IG depend on the choice of the baseline (Kapishnikov et al., 2019; Xu et al., 2020; Sturmfels et al., 2020).

Original: The reason for this can be observed from Eq.
Revised: The reason for this can be observed from Eq.

Original: In Eq.
Revised: In Eq.

Original: 1, x (cid:48) represents the baseline “information-less” image.
Revised: 1, x (cid:48) represents the baseline “information-less” image.

Original: Based on this, the input dimensions close to the baseline receive very low attributions even though they might be important.
Revised: Based on this, the input dimensions close to the baseline receive very low attributions even though they might be important.

Original: i.e., if i, j denote pixel locations, when x i,j − x (cid:48)
Revised: i.e., if i, j denote pixel locations, when x i,j − x (cid:48)

Original: i,j ≈ 0 , the attribution
Revised: i,j ≈ 0 , the attribution

Original: IG i,j ( x ) ≈ 0 irrespective of how important the pixels are.
Revised: IG i,j ( x ) ≈ 0 irrespective of how important the pixels are.

Original: For instance, black pixels (RGB value of (0, 0, 0)) will receive an exact 0 attribution for a black baseline.
Revised: For instance, black pixels (RGB value of (0, 0, 0)) will receive an exact 0 attribution for a black baseline.

Original: In fact, for any baseline, IG will be insensitive to the dimensions close to the baseline value.
Revised: In fact, for any baseline, IG will be insensitive to the dimensions close to the baseline value.

Original: When IG is applied to the ﬁrst layer activations however, the nodes with near 0 activations will by default receive less attribution.
Revised: When IG is applied to the ﬁrst layer activations however, the nodes with near 0 activations will by default receive less attribution.

Original: Thus, we believe that 0 activations in the ﬁrst layer is a more natural baseline for IG for ReLU networks, which is quantitatively observable in better LSC scores.
Revised: Thus, we believe that 0 activations in the ﬁrst layer is a more natural baseline for IG for ReLU networks, which is quantitatively observable in better LSC scores.

Original: [Table]
Revised: SMUG vs SMUG base .

Original: Based on the LSC scores in Table 1, SMUG narrowly outperforms SMUG base .
Revised: Based on the LSC scores in Table 1, SMUG narrowly outperforms SMUG base .

Original: Recall however, that SMUG is a sparser version of SMUG base obtained from the minimization constraints of the SMT solver (Eqns. 2 and 3).
Revised: Recall however, that SMUG is a sparser version of SMUG base obtained from the minimization constraints of the SMT solver (Eqns. 2 and 3).

Original: To compare the2 methods, we also measure sparsity of the masks.
Revised: LSC scores use the sparsity of the bounding boxes as a proxy for the sparsity of the saliency map. As we can see in Fig. 1(a) and 1(b) that the bounding boxes for SMUG and [Figure] SMUG base have similar sizes even though the former produces much sparser saliency maps.

Revised: This causes both SMUG and SMUG base to receive similar LSC scores.

Revised: Thus, the LSC scores alone don’t reveal the full picture and we need to consider the sparsity as well.

Original: This is deﬁned as the fraction of the pixels with non-zero attributions to the total number of pixels in the image.
Revised: This is deﬁned as the fraction of the pixels with non-zero attributions to the total number of pixels in the image.

Original: We ﬁnd that the average SMUG base mask has a sparsity of 43% while the average SMUG mask has a sparsity of just 17% .
Revised: We ﬁnd that the average SMUG base mask has a sparsity of 43% while the average SMUG mask has a sparsity of just 17% .

Original: This is also evident from the examples in Fig.
Revised: This is also evident from the examples in Fig.

Revised: SMUG and SMUG base receive similar

Original: The symbolic encoding effectively masks pixelsless relevant to the prediction while retaining the model’s conﬁdence, and hence maintains a high LSC score.
Revised: LSC scores which suggests that the symbolic encoding successfully retains pixels relevant to the prediction while removing about 66% pixels from SMUG base .

Original: G ROUND T RUTH , C ENTER B OX .
Revised: G ROUND T RUTH , C ENTER B OX .

Original: Based on qualitative examples Figs.
Revised: Based on qualitative examples Figs.

Original: 1, we can observe that in almost all cases the object is at the center of the image.
Revised: 1, we can observe that in almost all cases the object is at the center of the image.

Original: Hence, C ENTER B OX is likely to capture some part of the image.
Revised: Hence, C ENTER B OX is likely to capture some part of the image.

Original: Further, a fair number of objects are large covering much of the image e.g.,
Revised: Further, a fair number of objects are large covering much of the image e.g.,

Original: Fig.
Revised: Fig.

Original: 1 Macaque, Collie, Robin, Dalmation .
Revised: 1 Macaque, Collie, Robin, Dalmation .

Original: In these cases, the groundtruth bounding boxes are also large to fully cover all pixels corresponding to the object.
Revised: In these cases, the groundtruth bounding boxes are also large to fully cover all pixels corresponding to the object.

Original: In contrast, SMUG saliency maps are more compact for both large and small objects, and hence achieve a better LSC score.
Revised: In contrast, SMUG saliency maps are more compact for both large and small objects, and hence achieve a better LSC score.

Original: 5. D ISCUSSION : A NALYZING THE LSC METRIC AND O PT B OX
Revised: 5. D ISCUSSION : A NALYZING THE LSC METRIC AND O PT B OX

Original: [Figure]

Original: The LSC metric makes a trade-off when optimizing for both compactness and conﬁdence.
Revised: The LSC metric makes a trade-off when optimizing for both compactness and conﬁdence.

Original: To analyze this we look at several qualitative examples (Fig. 2) of the bounding boxes identiﬁed by the O PT B OX  brute-force approach to optimize the LSC metric.
Revised: To analyze this we look at several qualitative examples (Fig. 2) of the bounding boxes identiﬁed by the O PT B OX [Figure] [Figure] brute-force approach to optimize the LSC metric.

Original: We observe that O PT B OX often ﬁnds bounding boxes that are very small, typically a sufﬁciently discriminative region or pattern in the image (e.g., typewriter keyboard , paddlewheel , vine snake in Fig. 2), or the full object if the object is itself small (e.g., basketball , plunger ).
Revised: We observe that O PT B OX often ﬁnds bounding boxes that are very small, typically a sufﬁciently discriminative region or pattern in the image (e g., typewriter keyboard , paddlewheel , vine snake in Fig. 2) or the full object if the object is itself small (e.g., basketball , plunger ).

Original: In all these cases we ﬁnd that SMUG highlights several other aspects of the object as well (typewriter’s tape; the dog’s eyes, nose and ears, etc.)[Figure]
Revised: In all these cases we ﬁnd that SMUG highlights several other aspects of the object as well (typewriter’s tape; the dog’s eyes, nose and ears, etc.)

Original: In particular, O PT B OX seems to exploit the fact that the LSC metric is somewhat invariant to the size/scale of the object (because of the resizing).
Revised: In particular, O PT B OX seems to exploit the fact that the LSC metric is somewhat invariant to the size/scale of the object (because of the resizing).

Original: This can be seen from the examples in Fig.
Revised: This can be seen from the examples in Fig.

Original: LSC favors compact regions where the model continues to have high conﬁdence, and the O PT B OX score (Fig. 3(f)) is very good for the box around the small cat alone.
Revised: LSC favors compact regions where the model continues to have high conﬁdence, and the O PT B OX score (Fig. 3(f)) is very good for the box around the small cat alone.

Original: However, as evidenced from the conﬁdences of the model for Fig. 3 (a), (b) and (c) a good saliency map should assign greater attribution to cat big .
Revised: However, as evidenced from the conﬁdences of the model for Fig. 3 (a), (b) and (c) a good saliency map should assign greater attribution to cat big .

Original: In this respect, both SMUG and IG capture the model’s behavior correctly.
Revised: In this respect, both SMUG and IG capture the model’s behavior correctly.

Original: Moreover, SMUG attributions highlight the facial and body features of cat big .
Revised: Moreover, SMUG attributions highlight the facial and body features of cat big .

Original: 5. T EXT D ATASET : B EER R EVIEWS
Revised: 5. T EXT D ATASET : B EER R EVIEWS

Original: We present randomly selected qualitative examples from the test set comparing with other methods including SIS, IG, and G ROUND T RUTH in Fig.
Revised: We present randomly selected qualitative examples from the test set comparing with other methods including SIS, IG, and G ROUND T RUTH in Fig.

Original: 4 and Appendix.
Revised: 4 and Appendix.

Original: C. The solution of SIS consists of multiple disjoint set of words of varying relevance.
Revised: C. The solution of SIS consists of multiple disjoint set of words of varying relevance.

Original: A saliency map is constructed by scoring the words in the sets between (0,1] on the basis of relevance of the set.
Revised: A saliency map is constructed by scoring the words in the sets between (0,1] on the basis of relevance of the set.

Original: However, evaluation is harder.
Revised: However, evaluation is harder.

Original: Unlike images where the masked image can be cropped and resized as input to compute the LSC metric, the same strategy cannot be followed on the text model.
Revised: Unlike images where the masked image can be cropped and resized as input to compute the LSC metric, the same strategy cannot be followed on the text model.

Original: Speciﬁcally, ImageNet models are trained with extensive data-augmentation including random crops and resizing, and the modiﬁed image is less likely to be out-of-distribution.
Revised: Speciﬁcally, ImageNet models are trained with extensive data-augmentation including random crops and resizing, and the modiﬁed image is less likely to be out-of-distribution.

Original: Whereas in the case of text, this model doesn’t employ any form of meaningful augmentation, and the masked text is much more likely to come from a distribution that has not been seen during training.
Revised: Whereas in the case of text, this model doesn’t employ any form of meaningful augmentation, and the masked text is much more likely to come from a distribution that has not been seen during training.

Original: Hence LSC is not applicable here.
Revised: Hence LSC is not applicable here.

Original: [Table]
Revised: [Table]

Revised: [Figure]

Original: IG  
Revised: IG  -

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present an approach that uses SMT solvers for computing minimal input features that are relevant for a neural network prediction.
Revised: We present an approach that uses SMT solvers for computing minimal input features that are relevant for a neural network prediction.

Original: In particular, it uses attribution scores from Integrated Gradients to ﬁnd a subset of important neurons in the ﬁrst layer of the network, which allows the SMT encoding of constraints to scale to larger networks for ﬁnding minimal input masks.
Revised: In particular, it uses attribution scores from Integrated Gradients to ﬁnd a subset of important neurons in the ﬁrst layer of the network, which allows the SMT encoding of constraints to scale to larger networks for ﬁnding minimal input masks.

Original: We evaluate our technique to analyze models trained on image and text datasets and show that the saliency maps generated by our approach are competitive or better than existing approaches and produce sparser masks.
Revised: We evaluate our technique to analyze models trained on image and text datasets and show that the saliency maps generated by our approach are competitive or better than existing approaches and produce sparser masks.

Original: A PPENDIX
Revised: A PPENDIX

Original: Here, we present additional quantitative and qualitative examples supporting the results in Sec. 5.
Revised: Here, we present additional quantitative and qualitative examples supporting the results in Sec. 5.

Original: In particular Sec.
Revised: In particular Sec.

Original: A shows qualitative examples on MNIST, Sec.
Revised: A shows qualitative examples on MNIST, Sec.

Original: B presents quantitative and qualitative comparisons with regard to the choice of key parameters of our proposed SMUG explanation technique.
Revised: B presents quantitative and qualitative comparisons with regard to the choice of key parameters of our proposed SMUG explanation technique.

Original: Sec.
Revised: Sec.

Original: C presents additional qualitative examples comparing the output of SMUG with other methods on text samples from the Beer Reviews dataset.
Revised: C presents additional qualitative examples comparing the output of SMUG with other methods on text samples from the Beer Reviews dataset.

Original: Sec.
Revised: Sec.

Original: D presents several more qualitative examples from ImageNet comparing the saliency masks produced by SMUG with those produced by other methods.
Revised: D presents several more qualitative examples from ImageNet comparing the saliency masks produced by SMUG with those produced by other methods.

Original: In Sec.
Revised: In Sec.

Original: D.1 we also discuss an example where an explanation technique can be used to identify potential biases of the trained model.
Revised: D.1 we also discuss an example where an explanation technique can be used to identify potential biases of the trained model.

Original: Finally, Sec.
Revised: Finally, Sec.

Original: E presents some examples of concrete constraints that the solver optimizes.
Revised: E presents some examples of concrete constraints that the solver optimizes.

Original: A MNIST
Revised: A MNIST

Original: In this section, we present some more details about our experiments with MNIST using the full SMT encoding from Eq.
Revised: In this section, we present some more details about our experiments with MNIST using the full SMT encoding from Eq.

Original: 1 in Sec 3.1.
Revised: 1 in Sec 3.1.

Original: SMT solvers can be used to encode the semantics of a neural network (Katz et al., 2017).
Revised: SMT solvers can be used to encode the semantics of a neural network (Katz et al., 2017).

Original: In particular, given a fully connected neural network with n hidden layers, weights W
Revised: In particular, given a fully connected neural network with n hidden layers, weights W

Original: =
Revised: =

Original: { W 1 , W 2 . . .
Revised: { W 1 , W 2 . . .

Original: , W n } , biases
Revised: , W n } , biases

Original: B = { b 1 , b 2 . . .
Revised: B = { b 1 , b 2 . . .

Original: , b n } , activation function φ , and ﬁnal layer softmax σ , we can use the SMT theory of nonlinear real arithmetic to obtain a symbolic encoding of the network.
Revised: , b n } , activation function φ , and ﬁnal layer softmax σ , we can use the SMT theory of nonlinear real arithmetic to obtain a symbolic encoding of the network.

Original: Let X ∈ R m × n denote an input image with m × n pixels, M ∈ { 0 , 1 } m × n an unknown binary mask, L i the output (i.e., activations) of the i th layer ( L 

Revised: Let X ∈ R m × n denote an input image with m × n pixels, M ∈ { 0 , 1 } m × n an unknown binary mask, L i the output (i.e., activations) of the i th layer ( L

Original: =
Revised: =

Original: X is the input) and α ( p j ) the output of j th logit in the ﬁnal layer:
Revised: X is the input) and α ( p j ) the output of j th logit in the ﬁnal layer:

Original: L i ≡ φ ( W i L i − 1 + b i ) α
Revised: L i ≡ φ ( W i L i − 1 + b i ) α

Original: ( p j , W, B, X ) ≡ σ j ( W n L n − 1 + b n )
Revised: ( p j , W, B, X ) ≡ σ j ( W n L n − 1 + b n )

Original: Given this symbolic encoding, we can encode the minimal input mask generation problem as:
Revised: Given this symbolic encoding, we can encode the minimal input mask generation problem as:

Original: [Equation]
Revised: [Equation]

Original: , W, B, M (cid:12) X ) ∀ l (cid:54) = label (6) where p label and p l refer to the logits corresponding to the true label and the other labels respectively.
Revised: , W, B, M (cid:12) X ) ∀ l (cid:54) = label (6) where p label and p l refer to the logits corresponding to the true label and the other labels respectively.

Original: The number of constraints grow with increasing network size and SMT decision procedure for Nonlinear Real Arithmetic is doubly exponential.
Revised: The number of constraints grow with increasing network size and SMT decision procedure for Nonlinear Real Arithmetic is doubly exponential.

Original: Even for piecewise ReLU networks, SMT decision procedures for Linear Real Arithmetic combine simplex-based methods (exponential complexity) with other decision procedures such as Boolean logic (NP-complete complexity), which causes the solving times to grow dramatically with network size.
Revised: Even for piecewise ReLU networks, SMT decision procedures for Linear Real Arithmetic combine simplex-based methods (exponential complexity) with other decision procedures such as Boolean logic (NP-complete complexity), which causes the solving times to grow dramatically with network size.

Original: When we apply this encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4).
Revised: When we apply this encoding even for a small feed-forward network on MNIST dataset, the SMT solver does not scale well (Section 4).

Original: This motivates our proposed approach for using gradient information to simplify the SMT constraints.
Revised: This motivates our proposed approach for using gradient information to simplify the SMT constraints.

Original: Table 2 shows the SMT solver runtimes and whether the constraints were solved (S AT ).
Revised: Table 2 shows the SMT solver runtimes and whether the constraints were solved (S AT ).

Original: We observe that with a timeout of 60 minutes, the SMT solver could solve the full constraints for only 34 of the 100 images.
Revised: We observe that with a timeout of 60 minutes, the SMT solver could solve the full constraints for only 34 of the 100 images.

Original: Another interesting point to observe is that the solver could not solve any of the instances for digits 0 and 3 .
Revised: Another interesting point to observe is that the solver could not solve any of the instances for digits 0 and 3 .

Original: We also show some of the minimal masks generated by the SMT solver for few MNIST images in Figure 5.
Revised: We also show some of the minimal masks generated by the SMT solver for few MNIST images in Figure 5.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B H YPERPARAMETER C HOICES
Revised: B H YPERPARAMETER C HOICES

Original: In our proposed approach, the choice of top- k , and γ (Eqns. 3, 4, 5) have an effect on the ﬁnal quality of the explanations, and the time it takes for the solver to identify the mask.
Revised: In our proposed approach, the choice of top- k , and γ (Eqns. 3, 4, 5) have an effect on the ﬁnal quality of the explanations, and the time it takes for the solver to identify the mask.

Original: This section presents quantitative and qualitative comparisons for different choices of top- k and γ .
Revised: This section presents quantitative and qualitative comparisons for different choices of top- k and γ .

Original: B. Q UANTITATIVE COMPARISON FOR DIFFERENT CHOICES OF TOP - k AND γ
Revised: B. Q UANTITATIVE COMPARISON FOR DIFFERENT CHOICES OF TOP - k AND γ

Original: Fig.
Revised: Fig.

Original: 6 presents quantitative comparisons for different choices of top- k and γ . top-
Revised: 6 presents quantitative comparisons for different choices of top- k and γ . top-

Original: k We analyze images with top- k ∈ {500, 1000, 3000, 5000}.
Revised: k We analyze images with top- k ∈ {500, 1000, 3000, 5000}.

Original: Increasing the k value increases the receptive ﬁeld and the discovered input masks also grow in size with increasing values of k .
Revised: Increasing the k value increases the receptive ﬁeld and the discovered input masks also grow in size with increasing values of k .

Original: Figureshows how the solver runtime and the mask size vary with k .
Revised: Figureshows how the solver runtime and the mask size vary with k .

Original: As expected, larger k values results in larger number of constraints and therefore larger solving times as well as larger mask sizes.
Revised: As expected, larger k values results in larger number of constraints and therefore larger solving times as well as larger mask sizes.

Original: Gamma We analyze the effect of γ ∈ {0.0, 0.5, 0.9}, also shown in in Fig.
Revised: Gamma We analyze the effect of γ ∈ {0.0, 0.5, 0.9}, also shown in in Fig.

Original: We observe that by decreasing gamma values, the masks become sparser.
Revised: We observe that by decreasing gamma values, the masks become sparser.

Original: The key reason behind this is that with smaller gamma values, the SMTsolver is less constrained to maintain the original neural activations for the selected neurons, and hence can ignore additional input pixels that do not have a large effect.
Revised: The key reason behind this is that with smaller gamma values, the SMTsolver is less constrained to maintain the original neural activations for the selected neurons, and hence can ignore additional input pixels that do not have a large effect.

Original: It is also noteworthy to notice that the solver run-time increases with decreasing value of γ .
Revised: It is also noteworthy to notice that the solver run-time increases with decreasing value of γ .

Original: [Table]
Revised: [Figure]

Original: ON

Original: [Table]

Original: [Table]

Original: Fig.
Revised: Fig.

Original: 7 presents qualitative examples of the saliency maps on Imagenet examples for different choices of top- k and γ .
Revised: 7 presents qualitative examples of the saliency maps on Imagenet examples for different choices of top- k and γ .

Original: C Q UALITATIVE TEXT EXAMPLES
Revised: C Q UALITATIVE TEXT EXAMPLES

Original: Fig.
Revised: Fig.

Original: 8 presents additional examples comparing the output of SMUG with SIS, SMUG base , and IG on the Beer Reviews dataset. 
Revised: 8 presents additional examples comparing the output of SMUG with SIS, SMUG base , and IG on the Beer Reviews dataset. [Figure] [Table] [Figure]

Revised: [Table]

Original: D A DDITIONAL QUALITATIVE IMAGE EXAMPLES : SMUG
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: D A DDITIONAL QUALITATIVE IMAGE EXAMPLES : SMUG

Original: Fig.
Revised: Fig.

Original: 9 presents boolean masks and the saliency maps produced by SMUG on several ImageNet examples.
Revised: 9 presents boolean masks and the saliency maps produced by SMUG on several ImageNet examples.

Original: Fig.
Revised: Fig.

Original: 10 and 11 present additional examples comparing the saliency masks and bounding box (for LSC) produced by SMUG, SMUG base , and IG.
Revised: 10 and 11 present additional examples comparing the saliency masks and bounding box (for LSC) produced by SMUG, SMUG base , and IG.

Original: [Table]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D. B IASES
Revised: D. B IASES

Original: Model explanation techniques can also be particularly useful in studying model biases.
Revised: Model explanation techniques can also be particularly useful in studying model biases.

Original: Fig.shows some examples where the model correctly predicts the class as “parallel bars” but it appears to actually focus more on the person leaping over the bar to make the prediction as opposed to looking at the bar itself.
Revised: Fig.shows some examples where the model correctly predicts the class as “parallel bars” but it appears to actually focus more on the person leaping over the bar to make the prediction as opposed to looking at the bar itself.

Original: These can help us understand and identify cases where the model has developed a bias (in this case, based on training data).
Revised: These can help us understand and identify cases where the model has developed a bias (in this case, based on training data).

Original: E SMT CONSTRAINTS
Revised: E SMT CONSTRAINTS

Original: In this section, we present an example set of SMT constraints obtained by our technique for an example image from ImageNet.
Revised: In this section, we present an example set of SMT constraints obtained by our technique for an example image from ImageNet.

Original: For brevity, we show the set of constraints for k = 5 and γ = 0 .
Revised: For brevity, we show the set of constraints for k = 5 and γ = 0 .

Original: As mentioned in Sec 5.1, each masking variable M ij corresponds to a 4 × 4 grid of pixels, where the grid is denoted by X i : i +3 ,j : j +3 .
Revised: As mentioned in Sec 5.1, each masking variable M ij corresponds to a 4 × 4 grid of pixels, where the grid is denoted by X i : i +3 ,j : j +3 .

Original: For example, the mask variable M 132 , 135 corresponds to the pixel[Figure] grid X 132:135 , 132:135 .
Revised: For example, the mask variable M 132 , 135 corresponds to the pixel grid X 132:135 , 132:135 .

Original: Following Eq.
Revised: Following Eq.

Original: 4 in Sec 3.3, the SMT constraints corresponding for top k =
Revised: 4 in Sec 3.3, the SMT constraints corresponding for top k =

Original: IG positive attributions in the ﬁrst layer are given by: 
Revised: IG positive attributions in the ﬁrst layer are given by: 99 .

Original: [Figure]

Original: 53 X 132 , 132 − 58 .
Revised: 53 X 132 , 132 − 58 .

Original: 37 X 132 , 136 + 4 .
Revised: 37 X 132 , 136 + 4 .

Original: 88 X 132 , 140 − 141 .
Revised: 88 X 132 , 140 − 141 .

Original: 25 X 136 , 132 + 639 .
Revised: 25 X 136 , 132 + 639 .

Original: 97 X 136 , 136 + 10 .
Revised: 97 X 136 , 136 + 10 .

Original: 29 X 136 , 140 − 9 .
Revised: 29 X 136 , 140 − 9 .

Original: 66 X 140 , 132 + 20 .
Revised: 66 X 140 , 132 + 20 .

Original: 30 X 140 , 136 − 25 .
Revised: 30 X 140 , 136 − 25 .

Original: 19 X 140 , 140 − 0 .
Revised: 19 X 140 , 140 − 0 .

Original: 58 >− 270 .
Revised: 58 >− 270 .

Original: 67 M 120 , 150 + 101 .
Revised: 67 M 120 , 150 + 101 .

Original: 23 M 142 , 144 + 10 .
Revised: 23 M 142 , 144 + 10 .

Original: 38 M 113 , 124 + 207 .
Revised: 38 M 113 , 124 + 207 .

Original: 98 M 122 , 121 + 640 .
Revised: 98 M 122 , 121 + 640 .

Original: 64 M 121 , 121 − 100 . M 121 , 126 + 25 . 06 M 121 , 165 − 75 .
Revised: [Table]

Original: M 121 , 156 + 75 . 47 M 112 , 154 − 0 .
Revised: [Figure]

Original: 36 >2925 .
Revised: [Figure]

Original: 38 X 144 , 132 − 395 .
Revised: 38 X 144 , 132 − 395 .

Original: 09 X 144 , 136 + 81 .
Revised: 09 X 144 , 136 + 81 .

Original: 61 X 148 , 132 − 999 .
Revised: 61 X 148 , 132 − 999 .

Original: 88 X 148 , 136 − 82 .
Revised: 88 X 148 , 136 − 82 .

Original: 70 X 152 , 132 + 17 .
Revised: 70 X 152 , 132 + 17 .

Original: 08 X 152 , 136 + 0 . 21 >− 20 .
Revised: 08 X 152 , 136 + 0 . 21 >− 20 .

Original: 87 X 76 , 80 +8 .
Revised: 87 X 76 , 80 +8 .

Original: 40 X 76 , 84 − 122 .
Revised: 40 X 76 , 84 − 122 .

Original: 72 X 80 , 80 +929 .
Revised: 72 X 80 , 80 +929 .

Original: 71 X 80 , 84 +85 .
Revised: 71 X 80 , 84 +85 .

Original: 52 X 84 , 80 +138 .
Revised: 52 X 84 , 80 +138 .

Original: 99 X 84 , 84 − 0 .
Revised: 99 X 84 , 84 − 0 .

Original: 01 > 231 .
Revised: 01 > 231 .

Original: 34 X 168 , 148 +722 .
Revised: 34 X 168 , 148 +722 .

Original: 71 X 168 , 152 +80 .
Revised: 71 X 168 , 152 +80 .

Original: 18 X 172 , 148 +663 .
Revised: 18 X 172 , 148 +663 .

Original: 96 X 172 , 152 +5 .
Revised: 96 X 172 , 152 +5 .

Original: 37 X 176 , 148 +4 .
Revised: 37 X 176 , 148 +4 .

Original: 63 X 176 , 152 + 0 .
Revised: 63 X 176 , 152 + 0 .

Original: 12 >
Revised: 12 >

Original: [Figure]

Original: [Figure]