Original: C ONSTRAINT -D RIVEN
Revised: C ONSTRAINT -D RIVEN

Original: E
Revised: E

Original: XPLANATIONS OF B LACK -
Revised: XPLANATIONS OF B LACK -

Original: B OX ML M ODELS
Revised: B OX ML M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modern machine learning techniques have enjoyed widespread success, but are plagued by lack of transparency in their decision making, which has led to the emergence of the ﬁeld of explainable AI.
Revised: Modern machine learning techniques have enjoyed widespread success, but are plagued by lack of transparency in their decision making, which has led to the emergence of the ﬁeld of explainable AI.

Original: One popular approach called LIME, seeks to explain an opaque model’s behavior, by training a surrogate interpretable model to be locally faithful on perturbed instances.
Revised: One popular approach called LIME, seeks to explain an opaque model’s behavior, by training a surrogate interpretable model to be locally faithful on perturbed instances.

Original: Despite being model-agnostic and easyto-use, it is known that LIME’s explanations can be unstable and are susceptible to adversarial attacks as a result of Out-Of-Distribution (OOD) sampling.
Revised: Despite being model-agnostic and easyto-use, it is known that LIME’s explanations can be unstable and are susceptible to adversarial attacks as a result of Out-Of-Distribution (OOD) sampling.

Original: The quality of explanations is also calculated heuristically, and lacks a strong theoretical foundation.
Revised: The quality of explanations is also calculated heuristically, and lacks a strong theoretical foundation.

Original: In spite of numerous attempts to remedy some of these issues, making the LIME framework more trustworthy and reliable remains an open problem.
Revised: In spite of numerous attempts to remedy some of these issues, making the LIME framework more trustworthy and reliable remains an open problem.

Original: In this work, we demonstrate that the OOD sampling problem stems from rigidity of the perturbation procedure.
Revised: In this work, we demonstrate that the OOD sampling problem stems from rigidity of the perturbation procedure.

Original: To resolve this issue, we propose a theoretically sound framework based on uniform sampling of user-deﬁned subspaces.
Revised: To resolve this issue, we propose a theoretically sound framework based on uniform sampling of user-deﬁned subspaces.

Original: Through logical constraints, we afford the end-user the ﬂexibility to delineate the precise subspace of the input domain to be explained.
Revised: Through logical constraints, we afford the end-user the ﬂexibility to delineate the precise subspace of the input domain to be explained.

Original: This not only helps mitigate the problem of
Revised: This not only helps mitigate the problem of

Original: OOD sampling, but also allow experts to drill down and uncover bugs and biases hidden deep inside the model.
Revised: OOD sampling, but also allow experts to drill down and uncover bugs and biases hidden deep inside the model.

Original: For testing the quality of generated explanations, we develop a novel estimation algorithm that is able to certiﬁably measure the true value of metrics such as ﬁdelity up to any desired degree of accuracy, which can help in building trust in the generated explanations.
Revised: For testing the quality of generated explanations, we develop an efﬁcient estimation algorithm that is able to certiﬁably measure the true value of metrics such as ﬁdelity up to any desired degree of accuracy, which can help in building trust in the generated explanations.

Original: Our framework called
Revised: Our framework called

Original: CLIME can be applied to any ML model, and extensive experiments demonstrate its versatility on real-world problems.
Revised: CLIME can be applied to any ML model, and extensive experiments demonstrate its versatility on real-world problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Advances in Machine Learning (ML) in the last decade have resulted in new applications to safetycritical and human-sensitive domains such as driver-less cars, health, ﬁnance, education and the like (c.f.
Revised: Advances in Machine Learning (ML) in the last decade have resulted in new applications to safetycritical and human-sensitive domains such as driver-less cars, health, ﬁnance, education and the like (c.f.

Original: (Lipton, 2018)).
Revised: (Lipton, 2018)).

Original: In order to build trust in automated ML decision processes, it is not sufﬁcient to only verify properties of the model; the concerned human authority also needs to understand the reasoning behind predictions (DARPA, 2016; Goodman & Flaxman, 2017; Lipton, 2018).
Revised: In order to build trust in automated ML decision processes, it is not sufﬁcient to only verify properties of the model; the concerned human authority also needs to understand the reasoning behind predictions (DARPA, 2016; Goodman & Flaxman, 2017; Lipton, 2018).

Original: Highly successful models such as Neural Networks and ensembles, however, are often complex, and even experts ﬁnd it hard to decipher their inner workings.
Revised: Highly successful models such as Neural Networks and ensembles, however, are often complex, and even experts ﬁnd it hard to decipher their inner workings.

Original: Such opaque decision processes are unacceptable for safety-critical domains where a wrong decision can have serious consequences.
Revised: Such opaque decision processes are unacceptable for safety-critical domains where a wrong decision can have serious consequences.

Original: This has led to the emergence of the ﬁeld of eXplainable AI (XAI), which targets development of both naturally interpretable models (e.g. decision trees, lists, or sets) (Hu et al., 2019;
Revised: This has led to the emergence of the ﬁeld of eXplainable AI (XAI), which targets development of both naturally interpretable models (e.g. decision trees, lists, or sets) (Hu et al., 2019;

Original: Angelino et al., 2018; Rudin, 2019; Avellaneda, 2020) as well as post-hoc explanations for opaque models (Ribeiro et al., 2016;
Revised: Angelino et al., 2018; Rudin, 2019; Avellaneda, 2020) as well as post-hoc explanations for opaque models (Ribeiro et al., 2016;

Original: Lundberg & Lee, 2017).
Revised: Lundberg & Lee, 2017).

Original: Although interpretable models have been gaining traction, state-of-the-art approaches in most domains are uninterpretable and therefore necessitate post-hoc explanations.
Revised: Although interpretable models have been gaining traction, state-of-the-art approaches in most domains are uninterpretable and therefore necessitate post-hoc explanations.

Original: One popular post-hoc approach is the Locally Interpretable Model-agnostic Explanation (LIME) framework (Ribeiro et al., 2016), which seeks to explain individual predictions by capturing the local behaviour of the opaque model in an approximate but interpretable way.
Revised: One popular post-hoc approach is the Locally Interpretable Model-agnostic Explanation (LIME) framework (Ribeiro et al., 2016), which seeks to explain individual predictions by capturing the local behaviour of the opaque model in an approximate but interpretable way.

Original: To do so, it trains a surrogate interpretable classiﬁer to be faithful to the opaque model in a small neighborhood around a user-provided data instance.
Revised: To do so, it trains a surrogate interpretable classiﬁer to be faithful to the opaque model in a small neighborhood around a user-provided data instance.

Original: Speciﬁcally, the given data instance is perturbed to generate a synthetic data set on which an interpretable classiﬁer such as a linear model is trained with the objective of having high ﬁdelity to the original model.
Revised: Speciﬁcally, the given data instance is perturbed to generate a synthetic data set on which an interpretable classiﬁer such as a linear model is trained with the objective of having high ﬁdelity to the original model.

Original: The human decision-maker can inspect the coefﬁcients of the linear model to understand which features contributed positively or negatively to the prediction.
Revised: The human decision-maker can inspect the coefﬁcients of the linear model to understand which features contributed positively or negatively to the prediction.

Original: LIME is model-agnostic as it treats the opaque model as a black-box.
Revised: LIME is model-agnostic as it treats the opaque model as a black-box.

Original: It is applicable to a wide variety of data such as text, tabular and image, and is easy to use.
Revised: It is applicable to a wide variety of data such as text, tabular and image, and is easy to use.

Original: Nevertheless, it suffers from sensitivity to out-of-distribution sampling, which undermines trust in the explanations and makes them susceptible to adversarial attacks (Slack et al., 2020).
Revised: Nevertheless, it suffers from sensitivity to out-of-distribution sampling, which undermines trust in the explanations and makes them susceptible to adversarial attacks (Slack et al., 2020).

Original: Post-hoc explanation techniques are increasingly being used by ML experts to debug their models.
Revised: Post-hoc explanation techniques are increasingly being used by ML experts to debug their models.

Original: In this setting, LIME does not provide the user with the ﬂexibility to reﬁne explanations by drilling down to bug-prone corners of the input domain.
Revised: In this setting, LIME does not provide the user with the ﬂexibility to reﬁne explanations by drilling down to bug-prone corners of the input domain.

Original: Further,
Revised: Further,

Original: LIME has limited capabilities for measuring explanation quality accurately.
Revised: LIME has limited capabilities for measuring explanation quality accurately.

Original: For instance, checking whether the generated explanation is faithful to the original model can only be done on a heuristically deﬁned number of samples, which can be unreliable.
Revised: For instance, checking whether the generated explanation is faithful to the original model can only be done on a heuristically deﬁned number of samples, which can be unreliable.

Original: While later works such as Anchor (Ribeiro et al., 2018) and SHAP (Lundberg & Lee, 2017) mitigate some of these issues, LIME remains a popular framework and making it more robust and reliable remains an open problem.
Revised: While later works such as Anchor (Ribeiro et al., 2018) and SHAP (Lundberg & Lee, 2017) mitigate some of these issues, LIME remains a popular framework and making it more robust and reliable remains an open problem.

Original: Towards this goal, we ﬁrst demonstrate that LIME’s rigid perturbation procedure is at the root of many of these problems, due to its inability to capture the original data distribution.
Revised: Towards this goal, we ﬁrst demonstrate that LIME’s rigid perturbation procedure is at the root of many of these problems, due to its inability to capture the original data distribution.

Original: We craft concrete examples which clearly show LIME generating misleading explanations due to OOD sampling.
Revised: We craft concrete examples which clearly show LIME generating misleading explanations due to OOD sampling.

Original: Further, LIME affords the end-user limited control over the sub-space to be explained; since the notions of ‘locality’ and ‘neighborhood’ are deﬁned implicitly by LIME’s perturbation procedure.
Revised: Further, LIME affords the end-user limited control over the sub-space to be explained; since the notions of ‘locality’ and ‘neighborhood’ are deﬁned implicitly by LIME’s perturbation procedure.

Original: XAI, however, is inherently human-centric and one-size-ﬁts-all approaches are not powerful enough to handle all user-needs and application domains.
Revised: XAI, however, is inherently human-centric and one-size-ﬁts-all approaches are not powerful enough to handle all user-needs and application domains (Sokol et al., 2019).

Original: Instead, we propose to generalize the LIME framework by letting the user deﬁne the speciﬁc subspace of the input domain to be explained, through the use of logical constraints.
Revised: Instead, we propose to generalize the LIME framework by letting the user deﬁne the speciﬁc subspace of the input domain to be explained, through the use of logical constraints.

Original: Boolean constraints can capture a wide variety of distributions and recent advances in SAT solving and hashing technology have enabled fast solution sampling from large complex formulas (Soos et al., 2020).
Revised: Boolean constraints can capture a wide variety of distributions and recent advances in SAT solving and hashing technology have enabled fast solution sampling from large complex formulas (Soos et al., 2020).

Original: Making LIME’s neighborhood generation ﬂexible in this way resolves both limitations of LIME.
Revised: Making LIME’s neighborhood generation ﬂexible in this way resolves both limitations of LIME.

Original: First, our approach helps in mitigating the problem of OOD sampling.
Revised: First, our approach helps in mitigating the problem of OOD sampling.

Original: Second, it allows the user to drill down and reﬁne explanations.
Revised: Second, it allows the user to drill down and reﬁne explanations.

Original: For instance, it might be not sufﬁcient for doctors to simply know which test results contributed to a model’s prediction of cancer; it is also important to understand the prediction in the context of the patient’s speciﬁc age group, ethnicity etc.
Revised: For instance, it might be not sufﬁcient for doctors to simply know which test results contributed to a model’s prediction of cancer; it is also important to understand the prediction in the context of the patient’s speciﬁc age group, ethnicity etc.

Original: Such requirements can be naturally represented as constraints.
Revised: Such requirements can be naturally represented as constraints.

Original: In the same vein, constraints can also be used to zoom in to bug-prone corners of the input space to uncover potential problems in the model.
Revised: In the same vein, constraints can also be used to zoom in to bug-prone corners of the input space to uncover potential problems in the model.

Original: This is especially useful for model debugging which is a recent direction for ML research (Kang et al., 2020).
Revised: This is especially useful for model debugging which is a recent direction for ML research (Kang et al., 2020).

Original: Letting users deﬁne sub-spaces to be explained also necessitates a theoretically grounded method of measuring explanation quality, as a poor quality score can indicate to the user, the need for reﬁning constraints.
Revised: Letting users deﬁne sub-spaces to be explained also necessitates a theoretically grounded method of measuring explanation quality, as a poor quality score can indicate to the user, the need for reﬁning constraints.

Original: Existing works compute these metrics heuristically without formal guarantees of accuracy.
Revised: Existing works compute these metrics heuristically without formal guarantees of accuracy.

Original: In this light, we propose a theoretical framework and a novel estimation algorithm that enables measurement of the true value of metrics like ﬁdelity, up to any desired accuracy, in a model-agnostic way.
Revised: In this light, we propose a theoretical framework and an efﬁcient estimation algorithm that enables measurement of the true value of metrics like ﬁdelity, up to any desired accuracy, in a model-agnostic way.

Original: Through extensive experiments, we demonstrate  applications  to real world problems such as uncovering model biases and detecting adversarial attacks.
Revised: Through extensive experiments, we demonstrate the scalability of our estimation framework, as well as applications of CLIME to real world problems such as uncovering model biases and detecting adversarial attacks.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: Framework for precisely crafting explanations for speciﬁc subspaces of the input domain through logical constraints 2.
Revised: Framework for precisely crafting explanations for speciﬁc subspaces of the input domain through logical constraints 2.

Original: A theoretical framework and novel algorithm for estimating the ‘true’ value of metrics like ﬁdelity up to any desired accuracy 3.
Revised: A theoretical framework and an efﬁcient algorithm for estimating the ‘true’ value of metrics like ﬁdelity up to any desired accuracy 3.

Original: Empirical study which demonstrates the efﬁcacy of constraints in • Mitigating problem of OOD sampling •
Revised: Empirical study which demonstrates the efﬁcacy of constraints in • Mitigating problem of OOD sampling •

Original: Detecting adversarial attacks • Zooming in and reﬁning explanations for uncovering hidden biases
Revised: Detecting adversarial attacks • Zooming in and reﬁning explanations for uncovering hidden biases

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem formulation.
Revised: Problem formulation.

Original: We follow notations from Ribeiro et al.
Revised: We follow notations from Ribeiro et al.

Original: Let D =
Revised: Let D =

Original: ( X, y )
Revised: ( X, y )

Original: = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .
Revised: = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . .

Original: , ( x n , y n ) } denote the input dataset from some distribution D where , x i ∈R
Revised: , ( x n , y n ) } denote the input dataset from some distribution D where , x i ∈

Original: d is a vector that captures the feature values of the i th sample, and y i ∈ {C 0 , C 1 } is the corresponding class label

Original: Let f : R d → [0 , 1] denote the opaque classiﬁer that takes a data point x i as input and returns the probability of x i belonging to C 1 .
Revised: Let f : R d → [0 , 1] denote the opaque classiﬁer that takes a data point x i as input and returns the probability of x i belonging to C 1 .

Original: We assume that an instance x is assigned label l f ( x ) =
Revised: We assume that an instance x is assigned label l f ( x ) =

Original: C 

Revised: C

Original: if f ( x ) ≥ 0 .
Revised: if f ( x ) ≥ 0 .

Original: 5 and l f ( x )
Revised: 5 and l f ( x )

Original: =
Revised: =

Original: C 0 otherwise.
Revised: C 0 otherwise.

Original: Given a classiﬁer f , the task is to learn an explainer model g such that g mimics the behavior of f in the neighborhood of some given point x .
Revised: Given a classiﬁer f , the task is to learn an explainer model g such that g mimics the behavior of f in the neighborhood of some given point x .

Original: The type of classiﬁer g is chosen to be visually or textually interpretable, so that it can serve as an explanation for f ’s behavior in the neighborhood of x .
Revised: The type of classiﬁer g is chosen to be visually or textually interpretable, so that it can serve as an explanation for f ’s behavior in the neighborhood of x .

Original: Overview of LIME.
Revised: Overview of LIME.

Original: LIME builds the explainer function g on an ‘interpretable domain’ of inputs rather than the original domain.
Revised: LIME builds the explainer function g on an ‘interpretable domain’ of inputs rather than the original domain.

Original: To do so, it maps original features (that can be continuous or categorical) to Boolean features.
Revised: To do so, it maps original features (that can be continuous or categorical) to Boolean features.

Original: While x ∈ R d represents an instance in the original domain, we use prime-notation, i.e. x (cid:48) ∈ { 0 , 1 } d (cid:48) to represent an instance in the interpretable domain.
Revised: While x ∈ R d represents an instance in the original domain, we use prime-notation, i.e. x (cid:48) ∈ { 0 , 1 } d (cid:48) to represent an instance in the interpretable domain.

Original: Using Boolean features is a natural choice for ‘interpretable domain’, as we can understand explanations in terms of a presence/absence of a feature’s value.
Revised: Using Boolean features is a natural choice for ‘interpretable domain’, as we can understand explanations in terms of a presence/absence of a feature’s value.

Original: Thus g operates in the interpretable domain { 0 , 1 } d (cid:48) .
Revised: Thus g operates in the interpretable domain { 0 , 1 } d (cid:48) .

Original: LIME randomly perturbs the given instance x to generate neighbors z 1 , z 2 , . . .
Revised: LIME randomly perturbs the given instance x to generate neighbors z 1 , z 2 , . . .

Original: with corresponding interpretable versions z (cid:48) 1 , z (cid:48) 2 , . . . .
Revised: with corresponding interpretable versions z (cid:48) 1 , z (cid:48) 2 , . . . .

Original: The set of sampled neighbors i.e., the neighborhood of x , is denoted by Z ( Z (cid:48) in the interpretable space) .
Revised: The set of sampled neighbors i.e., the neighborhood of x , is denoted by Z ( Z (cid:48) in the interpretable space) .

Original: Additionally, we denote the universe of all possible neighbors of x (not just the ones sampled) by U Z and U Z (cid:48) .
Revised: Additionally, we denote the universe of all possible neighbors of x (not just the ones sampled) by U Z and U Z (cid:48) .

Original: Let the complexity of an explanation g be denoted as Ω( g ) (complexity of a linear model can be the number of non-zero weights), and let π x ( z ) denote the proximity measure between inputs x and z ∈ Z ( π x ( z ) can be deﬁned using cosine or L 2 distance).
Revised: Let the complexity of an explanation g be denoted as Ω( g ) (complexity of a linear model can be the number of non-zero weights), and let π x ( z ) denote the proximity measure between inputs x and z ∈ Z ( π x ( z ) can be deﬁned using cosine or L 2 distance).

Original: The objective function for training g is crafted to ensure that g : (1) approximates the behavior of f accurately in the vicinity of x where the proximity measure is high, and (2) achieves low complexity and is thereby interpretable.
Revised: The objective function for training g is crafted to ensure that g : (1) approximates the behavior of f accurately in the vicinity of x where the proximity measure is high, and (2) achieves low complexity and is thereby interpretable.

Original: The explanation is obtained as g ∗ =
Revised: The explanation is obtained as g ∗ =

Original: argmin g ∈ G L ( π x , g, f ) + Ω( g ) where G is the set of all linear classiﬁers and the loss function L is deﬁned as: L ( f, h, π x ) =
Revised: argmin g ∈ G L ( π x , g, f ) + Ω( g ) where G is the set of all linear classiﬁers and the loss function L is deﬁned as: L ( f, h, π x ) =

Original: (cid:80) z ∈Z ,z (cid:48) ∈Z (cid:48)
Revised: (cid:80) z ∈Z ,z (cid:48) ∈Z (cid:48)

Original: [ f ( z ) − g ( z (cid:48) )]
Revised: [ f ( z ) − g ( z (cid:48) )]

Original: 2 π x ( z ) .
Revised: 2 π x ( z ) .

Original: These are heuristically deﬁned functions.
Revised: These are heuristically deﬁned functions.

Original: Intuitively, the loss function captures how unfaithful g is to f in the neighborhood of x .
Revised: Intuitively, the loss function captures how unfaithful g is to f in the neighborhood of x .

Original: Boolean (logical) constraints and uniform sampling.
Revised: Boolean (logical) constraints and uniform sampling.

Original: We use notation standard in the area of
Revised: We use notation standard in the area of

Original: Boolean Satisﬁability (SAT).
Revised: Boolean Satisﬁability (SAT).

Original: A Boolean formula over n variables φ : { 0 , 1 } n → { 0 , 1 } assigns a truth value 0 / 1 or false/true to each of the 2 n assignments to it’s variables and is constructed using logical operators like AND ( ∧ ), OR( ∨ ), NOT ( ¬ ), XOR ( ⊕ ) etc.
Revised: A Boolean formula over n variables φ : { 0 , 1 } n → { 0 , 1 } assigns a truth value 0 / 1 or false/true to each of the 2 n assignments to it’s variables and is constructed using logical operators like AND ( ∧ ), OR( ∨ ), NOT ( ¬ ), XOR ( ⊕ ) etc.

Original: An assignment of truth values to variables denoted s ∈ { 0 , 1 } n is said to satisfy φ (denoted s | = φ
Revised: An assignment of truth values to variables denoted s ∈ { 0 , 1 } n is said to satisfy φ (denoted s | = φ

Original: ) iff φ ( s )
Revised: ) iff φ ( s )

Original: = 1 .
Revised: = 1 .

Original: The total number of assignments that satisfy φ is denoted as # φ =
Revised: The total number of assignments that satisfy φ is denoted as # φ =

Original: (cid:80) s ∈{ 0 , 1 } n φ ( s ) .
Revised: (cid:80) s ∈{ 0 , 1 } n φ ( s ) .

Original: An algorithm is said to be a (perfectly) uniform sampler if it takes as input an arbitrary formula φ and returns an assignment s ∗ such that ∀ s | = φ , we have Pr [ s ∗ =
Revised: An algorithm is said to be a (perfectly) uniform sampler if it takes as input an arbitrary formula φ and returns an assignment s ∗ such that ∀ s | = φ , we have Pr [ s ∗ =

Original: s ]
Revised: s ]

Original: = 1# φ .
Revised: = 1# φ .

Original: An almost-uniform sampler is an algorithm that takes, along with φ , a parameter ε > 0 as input and returns s ∗ such that ∀ s | = φ , we have(1+ ε )# φ ≤ Pr [ s ∗ = s ] ≤ 1+ ε # φ .
Revised: An almost-uniform sampler is an algorithm that takes, along with φ , a parameter ε > 0 as input and returns s ∗ such that ∀ s | = φ , we have(1+ ε )# φ ≤ Pr [ s ∗ = s ] ≤ 1+ ε # φ .

Original: The tools WAPS (Gupta et al., 2019) and Unigen

Revised: The tools WAPS (Gupta et al., 2019) and Unigen

Original: (Soos et al., 2020) are state-of-the-art perfectly uniform and almost-uniform samplers respectively.
Revised: (Soos et al., 2020) are state-of-the-art perfectly uniform and almost-uniform samplers respectively.

Original: Fidelity.
Revised: Fidelity.

Original: The ﬁdelity ρ of the explainer model g is a measure of how closely the predictions made by g match those of f in the neighborhood around the given point x
Revised: The ﬁdelity ρ of the explainer model g is a measure of how closely the predictions made by g match those of f in the neighborhood around the given point x

Original: (Ribeiro et al., 2016).
Revised: (Ribeiro et al., 2016).

Original: It is calculated as the fraction of the sampled neighbors where the output class of f and g agree, i.e.
Revised: It is calculated as the fraction of the sampled neighbors where the output class of f and g agree, i.e.

Original: [Equation]
Revised: [Equation]

Original: where I is the indicator function.
Revised: where I is the indicator function.

Original:  A NALYSIS OF LIME
Revised:  A NALYSIS OF LIME

Original: LIME sampling procedure.
Revised: LIME sampling procedure.

Original: LIME generates the neighborhood of the data point under explanation differently for text, image and tabular datasets.
Revised: LIME generates the neighborhood of the data point under explanation differently for text, image and tabular datasets.

Original: We consider tabular data here (see Appendix B for omitted details, examples, etc.).
Revised: We consider tabular data here (see Appendix B for omitted details, examples, etc.).

Original: For tabular data, LIME samples points z i ∈
Revised: For tabular data, LIME samples points z i ∈

Original: Z in the neighborhood of x (continuous features are discretized) and then maps them to the interpretable (binary) domain as z (cid:48)
Revised: Z in the neighborhood of x (continuous features are discretized) and then maps them to the interpretable (binary) domain as z (cid:48)

Original: i ∈ Z (cid:48) .
Revised: i ∈ Z (cid:48) .

Original: The distance function π is deﬁned on the binarized space to measure distance between the original x (cid:48) and sampled instances z (cid:48) .
Revised: The distance function π is deﬁned on the binarized space to measure distance between the original x (cid:48) and sampled instances z (cid:48) .

Original: Given ( Z (cid:48) , Z , π ) LIME builds a linear regression model g to ﬁt these data, where the distance measure π deﬁnes weights of samples.
Revised: Given ( Z (cid:48) , Z , π ) LIME builds a linear regression model g to ﬁt these data, where the distance measure π deﬁnes weights of samples.

Original: [Equation]
Revised: [Equation]

Original: Framework characteristics.
Revised: Framework characteristics.

Original: The underlying assumption of LIME is that it is possible to approximate the behavior of the black box classiﬁer accurately around x , so that g can faithfully mimic f in x ’s neighbourhood.
Revised: The underlying assumption of LIME is that it is possible to approximate the behavior of the black box classiﬁer accurately around x , so that g can faithfully mimic f in x ’s neighbourhood.

Original: To fulﬁll this assumption, LIME presumes that it is possible to produce a representation of the neighbourhood Z of x that adheres to the true distribution of the data D (i.e.
Revised: To fulﬁll this assumption, LIME presumes that it is possible to produce a representation of the neighbourhood Z of x that adheres to the true distribution of the data D (i.e.

Original: Z ∼ D ), through its ﬁxed sampling procedure.
Revised: Z ∼ D ), through its ﬁxed sampling procedure.

Original: However, this assumption is quite strong.
Revised: However, this assumption is quite strong.

Original: Even if the classiﬁer f behaves linearly in the neighbourhood of x , the correct distribution of instances can vary depending on the problem being solved, which may not be accounted for by a ﬁxed sampling procedure.
Revised: Even if the classiﬁer f behaves linearly in the neighbourhood of x , the correct distribution of instances can vary depending on the problem being solved, which may not be accounted for by a ﬁxed sampling procedure.

Original: Therefore, the key question is does LIME’s sampling procedure produce a neighbourhood Z s.t.
Revised: Therefore, the key question is does LIME’s sampling procedure produce a neighbourhood Z s.t.

Original: Z ∼ D ? .
Revised: Z ∼ D ? .

Original: Another more practical question is does OOD sampling lead to unreliable explanations?
Revised: Another more practical question is does OOD sampling lead to unreliable explanations?

Original: In fact, Slack et al.
Revised: In fact, Slack et al.

Original: (2020) already gamed LIME’s out-of-distribution (OOD) samples using an additional scaffolded classiﬁer that behaves exactly like f on samples from the distribution and produces arbitrary decisions for other samples.
Revised: (2020) already gamed LIME’s out-of-distribution (OOD) samples using an additional scaffolded classiﬁer that behaves exactly like f on samples from the distribution and produces arbitrary decisions for other samples.

Original: We show that even without wrapping a classiﬁer,
Revised: We show that even without wrapping a classiﬁer,

Original: LIME’s explanations suffer dramatically from OOD sampling.
Revised: LIME’s explanations suffer dramatically from OOD sampling.

Original: To do so, we introduce a simple metric that measures the severity of OOD sampling.
Revised: To do so, we introduce a simple metric that measures the severity of OOD sampling.

Original: The metric is based on the observation that if Z ∼ D (i.e. no OOD sampling occurs), then the fraction of instances with ‘true’ label C 1 should be roughly the same for both Z and D .
Revised: The metric is based on the observation that if Z ∼ D (i.e. no OOD sampling occurs), then the fraction of instances with ‘true’ label C 1 should be roughly the same for both Z and D .

Original: Let W Q be the fraction of intances q ∈ Q that have true label l ( q )
Revised: Let W Q be the fraction of intances q ∈ Q that have true label l ( q )

Original: =
Revised: =

Original: C 1 , i.e. W Q = (cid:80) q ∈ Q I ( l ( q ) =
Revised: C 1 , i.e. W Q = (cid:80) q ∈ Q I ( l ( q ) =

Original: = C 1 ) /
Revised: = C 1 ) /

Original: | Q | .
Revised: | Q | .

Original: Then if Z ∼ D , we should have W Z /W
Revised: Then if Z ∼ D , we should have W Z /W

Original: D
Revised: D

Original: ≈ 1 .
Revised: ≈ 1 .

Original: By construction of the binary space Z (cid:48) , we know that W Z = W Z (cid:48) .
Revised: By construction of the binary space Z (cid:48) , we know that W Z = W Z (cid:48) .

Original: Therefore, we should also get W Z (cid:48) /W
Revised: Therefore, we should also get W Z (cid:48) /W

Original: D
Revised: D

Original: ≈ 1 .
Revised: ≈ 1 .

Original: For an explainer E , we deﬁne
Revised: For an explainer E , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: If the value of SQ ( E ) is far from 1 , it is indicative of OOD sampling.
Revised: If the value of SQ ( E ) is far from 1 , it is indicative of OOD sampling.

Original: However, computing SQ ( E ) directly is infeasible in practice, as we generally do not have access to the true labels of instances generated through sampling.
Revised: However, computing SQ ( E ) directly is infeasible in practice, as we generally do not have access to the true labels of instances generated through sampling.

Original: Therefore we craft a few classiﬁcation tasks where a perfect classiﬁer f can be trained such that f ( x ) ≥ 0 .
Revised: Therefore we craft a few classiﬁcation tasks where a perfect classiﬁer f can be trained such that f ( x ) ≥ 0 .

Original: 5 for ∀ x, x ∼ D such that x belongs to C 1 .
Revised: 5 for ∀ x, x ∼ D such that x belongs to C 1 .

Original: This allows us to obtain the true labels for computing W Z (cid:48) and W D .
Revised: This allows us to obtain the true labels for computing W Z (cid:48) and W D .

Revised: We show such an example in Fig.1, called ‘Hit or miss example’.

Revised: In Fig.1, the squares S 0 and S 1 represent the areas of the input space that are in-distribution, while points outside the squares (white regions) are OOD.

Revised: Intuitively, when perturbing the point with coordinates (8 , 8) , for example, LIME’s distribution-unaware sampling procedure is likely to generate a lot of OOD samples, as the neighborhood of (8 , 8) is largely OOD (white).

Original:  Example B.1 shows such a simple classiﬁcation task.
Revised: In Example B.1 in Appendix B, we precisely deﬁne the classiﬁcation task in this example.

Original: We ﬁnd SQ ( LIME ) = 0 .
Revised: We found SQ ( LIME ) = 0 .

Original: 75for this task, which is much less than 1, and clearly indicates that LIME’s sampling procedure does not satisfy the main underlying assumption Z ∼ D .
Revised: 75 which is much less than 1, and clearly indicates that LIME’s sampling procedure does not satisfy the main underlying assumption Z ∼ D .

Original: Framework capabilities.
Revised: Framework capabilities.

Original: The other limitation of LIME is that the ﬁxed perturbation procedure gives users very little ﬂexibility in tailoring explanations to their needs.
Revised: The other limitation of LIME is that the ﬁxed perturbation procedure gives users very little ﬂexibility in tailoring explanations to their needs.

Original: Users can only control the data point x to be explained, which can be insufﬁcient for different use-cases.
Revised: Users can only control the data point x to be explained, which can be insufﬁcient for different use-cases.

Original: It is natural to ask: Can we generate explanations using a speciﬁc subspace of inputs?
Revised: It is natural to ask: Can we generate explanations using a speciﬁc subspace of inputs?

Original: Namely, can we formally deﬁne Z instead of having LIME deﬁne it for us?
Revised: Namely, can we formally deﬁne Z instead of having LIME deﬁne it for us?

Original: The need for customizing Z arises naturally in the process of reﬁning explanations, or when debugging models.
Revised: The need for customizing Z arises naturally in the process of reﬁning explanations, or when debugging models.

Original: For example, regulators may want to know how insurance applications are decided for people with speciﬁc preexisting conditions.
Revised: For example, regulators may want to know how insurance applications are decided for people with speciﬁc preexisting conditions.

Original: Traders may need to understand the reason behind a decision to sell, in the context of the prevailing market conditions.
Revised: Traders may need to understand the reason behind a decision to sell, in the context of the prevailing market conditions.

Original: Answering such queries is challenging in the existing LIME setting as it requires sampling capabilities from a constrained subspace.
Revised: Answering such queries is challenging in the existing LIME setting as it requires sampling capabilities from a constrained subspace.

Original: In our proposed framework, the user can naturally enforce such requirements through constraints such as ordering relations, arithmetic expressions over bounded integers, Boolean expressions or any logical combinations of these.
Revised: In our proposed framework, the user can naturally enforce such requirements through constraints such as ordering relations, arithmetic expressions over bounded integers, Boolean expressions or any logical combinations of these.

Original:  G ENERATING AND C ERTIFYING C ONSTRAINT -D RIVEN E XPLANATIONS
Revised:  G ENERATING AND C ERTIFYING C ONSTRAINT -D RIVEN E XPLANATIONS

Original: From the preceding discussion, it is clear that the limitations of the LIME framework that remain unaddressed are (a) OOD sampling, (b) ability to work in constrained spaces and (c) providing veriﬁable guarantees on ﬁdelity.
Revised: From the preceding discussion, it is clear that the limitations of the LIME framework that remain unaddressed are (a) OOD sampling, (b) ability to work in constrained spaces and (c) providing veriﬁable guarantees on ﬁdelity.

Original: Towards the goal of remedying this, we ﬁrst present our constraintdriven explanation generation framework called CLIME.
Revised: Towards the goal of remedying this, we ﬁrst present our constraintdriven explanation generation framework called CLIME.

Original: Then, we show how we can obtain certiﬁably accurate measurements of an explanation’s quality through a novel estimation algorithm. 
Revised: Then, we show how we can obtain certiﬁably accurate measurements of an explanation’s quality through a novel estimation algorithm. S S     (8,8)

Original: 4. CLIME E XPLANATION F RAMEWORK
Revised: 4. CLIME E XPLANATION F RAMEWORK

Original: The CLIME framework generates explanations on instances sampled (almost) uniformly from userdeﬁned subspaces which are deﬁned through constraints.
Revised: The CLIME framework generates explanations on instances sampled (almost) uniformly from userdeﬁned subspaces which are deﬁned through constraints.

Original: Boolean constraints are powerful enough to

Original: Output: g : Interpretable linear classiﬁer 1: Z (cid:48) ← getSamples ( φ, ε, N ) ; 2: Z ← {} 3: for z (cid:48)

Original: ∈ Z (cid:48) do 4:

Original: Z ← Z ∪ { z (cid:48) , f ( z ) , π x ( z ) } 5: end for 6: g ← K-L ASSO ( Z, K )

Original: Output: (cid:98) ρ : Estimate of ρ (see Thm. 1) 1: if checkThreshold( ε, δ, γ ) =

Original: =

Original: True then 2: return ⊥ 3: end if 4: (cid:98) ρ ← AA (cid:48) (0 .

Original: 4 ∗ ε, 0 .

Original: 4 ∗ ε, δ ) 5: return (cid:98) ρ Output: True whp if ρ ≤ γ − ε 1: ν ← min ( ε + ε 2 / 2 − γε/ 2 , ( ε − γε/ 2) / (1+ ε/ 2)) 2: N ← 12 ν 2 log( 1 δ ) 3: S ← getSamples ( N, ε/ 2) 4: C ←5: for s ∈ S do 6: c ← checkProperty ( s ) 7: C ← C + c/N 8: end for 9: if C ≤ γ then 10: return True 11: else 12: return False 13: end if represent log-linear family of distributions (Chavira & Darwiche, 2008), yet allow fast sampling of solutions either uniformly or with a user-provided bias (Gupta et al., 2019), thanks to advances in SAT technology.
Revised: Boolean constraints are powerful enough to represent log-linear family of distributions (Chavira & Darwiche, 2008), yet allow fast sampling of solutions either uniformly or with a user-provided bias (Gupta et al., 2019), thanks to advances in SAT technology.

Original: In this work, we focus on explanations based on (almost) uniformly sampling solutions of constraints, but we note that the extension to biased (weighted) sampling is straightforward.
Revised: In this work, we focus on explanations based on (almost) uniformly sampling solutions of constraints, but we note that the extension to biased (weighted) sampling is straightforward.

Original: The pseudo-code of the constrained explanation framework is presented in Alg.
Revised: The pseudo-code of the constrained explanation framework is presented in Alg.

Original: The key difference with respect to LIME is that along with the input instance x CLIME also takes as input a Boolean formula φ .
Revised: The key difference with respect to LIME is that along with the input instance x CLIME also takes as input a Boolean formula φ .

Original: The variables of φ are exactly the Boolean features of the interpretable domain Z (cid:48) .
Revised: The variables of φ are exactly the Boolean features of the interpretable domain Z (cid:48) .

Original: As an example, assume that φ represents the constraint that at least ‘c’ variables must be true for some user-deﬁned ‘c’.
Revised: As an example, assume that φ represents the constraint that at least ‘c’ variables must be true for some user-deﬁned ‘c’.

Original: For image data this constraint enforces the requirement that at least ‘c’ superpixels must be ‘on’, while for text it forces at least ‘c’ words from x to be present in each z .
Revised: For image data this constraint enforces the requirement that at least ‘c’ superpixels must be ‘on’, while for text it forces at least ‘c’ words from x to be present in each z .

Original: This blocks out very sparse data ensuring that only informative instances are used for training the explainer model.
Revised: This blocks out very sparse data ensuring that only informative instances are used for training the explainer model.

Revised: We assume access to a procedure getSamples that returns N independent samples satisfying φ .

Original: The algorithmalso takes  a parameter ε  which represents the tolerance to deviation from perfectlyuniform sampling.
Revised: The algorithm takes as input a parameter ε , which represents the tolerance to deviation from perfectlyuniform sampling.

Original: If ε = 0 , then the call to getSamples in line 1 must be to a perfectly uniform sampler like WAPS (Gupta et al., 2019), otherwise an almost-uniform sampler like Unigen3 (Soos et al., 2020) sufﬁces.
Revised: If ε = 0 , then the call to getSamples in line 1 must be to a perfectly uniform sampler like WAPS (Gupta et al., 2019), otherwise an almost-uniform sampler like Unigen3 (Soos et al., 2020) sufﬁces.

Original: The rest of the algorithm is similar to LIME:
Revised: The rest of the algorithm is similar to LIME:

Original: samples z (cid:48)
Revised: samples z (cid:48)

Original: i are mapped back to the original domain, and the output of f on each z
Revised: i are mapped back to the original domain, and the output of f on each z

Original: i and the distance of each z i to x are used for training g in line 6, where at most K coefﬁcients are allowed to be non-zero to ensure interpretability.
Revised: i and the distance of each z i to x are used for training g in line 6, where at most K coefﬁcients are allowed to be non-zero to ensure interpretability.

Original: In Appendix C, we consider two classes of crafted examples to demonstrate  undesired consequences of OOD sampling for LIME and show that these issues are resolved by CLIME.
Revised: In Appendix C, we consider two classes of crafted examples to demonstrate (a) undesired consequences of OOD sampling for LIME which are resolved in CLIME (Example C.1 and C.2) and (b)

Original: We show that giving the user the ability to deﬁne Z φ ∼ D using constraints mitigates OOD sampling. Also, we show that SQ ( LIME ) can be very low, e.g. 0.43, while SQ ( CLIME ) = 0 . 99 on the same task (see
Revised: CLIME’s capabilities to incorporate user-deﬁned constraints (Example C.3).

Original: Example C.1 and C.2) showing that ODD sampling does not occur. Finally, we demonstrate the inﬂuence of OOD sampling on explanations.
Revised: Our ﬁrst set of examples highlight that LIME does suffer from OOD sampling, leading to poor quality of explanations.

Original: Even if a feature value is indicative of class C 0 , LIME often assigns the feature a positive coefﬁcient indicating that it contributes toward C 1 prediction. Another interesting observation is  that the dataset itself can have a very strong inﬂuence on generated explanations, e.g. unbalanced datasets can introduce a strong bias on explanations (Example C.2).
Revised: In particular, SQ ( LIME ) can be very low and LIME’s explanations signiﬁcantly differ from the true explanations. In contrast, giving the user the ability to deﬁne Z φ ∼ D using constraints as in CLIME, mitigates OOD sampling; SQ ( CLIME ) is almost 1 and CLIME produces correct explanations (we recall that the ground truth is known in these examples).

Original: Our third example (Example C.3) illustrates the explanation debugging process, where we allow the user to specify the subspace Z φ to obtain an explanation from, e.g. to know how decisions are made for speciﬁc classes of people such as those with alcohol dependency.
Revised: Our second class of crafted examples illustrates the explanation debugging process, where we allow the user to specify the subspace Z φ to obtain an explanation from, e.g. to know how decisions are made for speciﬁc classes of people such as those with alcohol dependency.

Original: We show that CLIME can naturally handle these constrained user-spaces, which may be challenging for LIME.
Revised: We show that CLIME can naturally handle these constrained user-spaces, which may be challenging for LIME.

Original: 4. C ERTIFYING E XPLANATION Q UALITY
Revised: Input: f : Opaque classiﬁer φ : Boolean constraints ε : Tolerance N : Number of samples π x : Similarity kernel K : Length of explanation Output: g : Interpretable linear classiﬁer 1: Z (cid:48) ← getSamples ( φ, ε, N ) ; 2: Z ← {} 3: for z (cid:48) ∈ Z (cid:48) do 4:

Revised: Z ← Z ∪ { z (cid:48) , f ( z ) , π x ( z ) } 5: end for 6: g ← K-L ASSO ( Z, K )

Revised: Input: ε : Tolerance δ : Conﬁdence γ : Threshold

Revised: Output: (cid:98) ρ : Estimate of ρ (see Thm. 1) 1: if checkThreshold( ε, δ, γ ) =

Revised: =

Revised: True then 2: return ⊥ 3: end if 4: (cid:98) ρ ← AA (cid:48) (0 .

Revised: 4 ∗ ε, 0 .

Revised: 4 ∗ ε, δ ) 5: return (cid:98)

Revised: ρ

Revised: Algorithm

Revised: Input: ε : Tolerance δ : Conﬁdence γ : Threshold

Revised: Output: True whp if ρ ≤ γ − ε 1: ν ← min ( ε + ε 2 / 2 − γε/ 2 , ( ε − γε/ 2) / (1+ ε/ 2)) 2: N ← 12 ν 2 log( 1 δ ) 3: S ← getSamples ( φ, ε/ 2 , N ) 4: C ←5: for s ∈ S do 6: c ← checkProperty ( s ) 7:

Revised: C ← C + c/N 8: end for 9: if C ≤ γ then 10: return True 11: else 12: return False 13: end if

Revised: 4. C ERTIFYING E XPLANATION Q UALITY

Original: For increasing user trust, it is necessary to provide a measure of the quality of explanations generated.
Revised: For increasing user trust, it is necessary to provide a measure of the quality of explanations generated.

Original: This is especially important for explanations of user deﬁned sub-spaces, as it may be possible that no simple explanation exists for a large subspace, and further reﬁnements to the constraints may be required to get a high quality explanation.
Revised: This is especially important for explanations of user deﬁned sub-spaces, as it may be possible that no simple explanation exists for a large subspace, and further reﬁnements to the constraints may be required to get a high quality explanation.

Original: The ﬁdelity metric, as deﬁned in Eqn.
Revised: The ﬁdelity metric, as deﬁned in Eqn.

Original: 1 measures quality in terms of the fraction of samples on which the prediction made by the explainer model, matches the prediction of the opaque model.
Revised: 1 measures quality in terms of the fraction of samples on which the prediction made by the explainer model, matches the prediction of the opaque model.

Original: A high ﬁdelity score indicates that the explainer model is faithful to the opaque model in Z .
Revised: A high ﬁdelity score indicates that the explainer model is faithful to the opaque model in Z .

Original: A low ﬁdelity score can occur if, for instance, a linear classiﬁer is used for explaining a highly non-linear decision boundary, making the explainer unreliable.
Revised: A low ﬁdelity score can occur if, for instance, a linear classiﬁer is used for explaining a highly non-linear decision boundary, making the explainer unreliable.

Original: Two parameters inﬂuence the accuracy of the ﬁdelity score: the number of samples in Z φ and the quality of these samples, i.e. their uniformity in U Z .
Revised: Two parameters inﬂuence the accuracy of the ﬁdelity score: the number of samples in Z φ and the quality of these samples, i.e. their uniformity in U Z .

Original: Both of these parameters were chosen heuristically in prior works including LIME, which makes the explanations harder to trust.
Revised: Both of these parameters were chosen heuristically in prior works including LIME, which makes the explanations harder to trust.

Original: Intuitively, a score measured on 10 samples will not be as stable as one measured on 10000 due to randomness inherent in any sampling procedure.
Revised: Intuitively, a score measured on 10 samples will not be as stable as one measured on 10000 due to randomness inherent in any sampling procedure.

Original: Ideally one would want to compute the ﬁdelity on all possible instances belonging to a user-deﬁned subspace of inputs.
Revised: Ideally one would want to compute the ﬁdelity on all possible instances belonging to a user-deﬁned subspace of inputs.

Original: Note that Eqn.
Revised: Note that Eqn.

Original: 1 computes an estimate or sample mean of the ‘true’ population mean deﬁned by
Revised: 1 computes an estimate or sample mean of the ‘true’ population mean deﬁned by

Original: [Equation]
Revised: [Equation]

Original:  This observation allows us to compute the estimate (cid:98) ρ in theoretically grounded way, so as to statistically guarantee its closeness to ρ .
Revised: [Equation] This observation allows us to compute the estimate (cid:98) ρ in theoretically grounded way, so as to statistically guarantee its closeness to ρ .

Original: Such guarantees can be indispensable in safety-critical areas such as healthcare, to show adherence to regulatory requirements for AI, for instance.
Revised: Such guarantees can be indispensable in safety-critical areas such as healthcare, to show adherence to regulatory requirements for AI, for instance.

Original: Computing ρ exactly is usually intractable, as the constraint φ is likely to have tens of hundreds of variables and exponentially many solutions.
Revised: Computing ρ exactly is usually intractable, as the constraint φ is likely to have tens of hundreds of variables and exponentially many solutions.

Original: Approximating ρ can be faster, but requires formal guarantees to be meaningful.
Revised: Approximating ρ can be faster, but requires formal guarantees to be meaningful.

Original: A good approximation of ρ is one that is within user-deﬁned tolerance of the true value with high conﬁdence.
Revised: A good approximation of ρ is one that is within user-deﬁned tolerance of the true value with high conﬁdence.

Original: Speciﬁcally, (cid:98) ρ is a good approximation of ρ if
Revised: Speciﬁcally, (cid:98) ρ is a good approximation of ρ if

Original: [Equation]
Revised: [Equation]

Original:  where ε > 0 , δ > 0 are user-deﬁned tolerance and conﬁdence.
Revised: 5) where ε > 0 , δ > 0 are user-deﬁned tolerance and conﬁdence.

Original: To the best of our knowledge, no existing approach is directly applicable for ﬁnding a good approximation of ρ , in a model-agnostic way.
Revised: To the best of our knowledge, no existing approach is directly applicable for ﬁnding a good approximation of ρ , in a model-agnostic way.

Original: The technique presented by (Narodytska et al., 2019), requires the opaque model to be encoded as a Boolean formula, severely limiting types of models that can be explained.
Revised: The technique presented by (Narodytska et al., 2019), requires the opaque model to be encoded as a Boolean formula, severely limiting types of models that can be explained.

Original: On the other hand, algorithms based on Monte Carlo sampling such as the AA algorithm by (Dagum et al., 2000), are known to be fast when ρ is high, but require far too many samples when ρ is low (Meel et al., 2019).
Revised: On the other hand, algorithms based on Monte Carlo sampling such as the AA algorithm by (Dagum et al., 2000), are known to be fast when ρ is high, but require far too many samples when ρ is low (Meel et al., 2019).

Original: They also require perfectly uniform samples, while it may only be feasible to generate almost-uniform samples from the universe U Z φ .
Revised: They also require perfectly uniform samples, while it may only be feasible to generate almost-uniform samples from the universe U Z φ .

Original: In this section, we describe a novel estimation algorithm  that is able to work with almost-uniform samples and also terminate quickly if the quantity being approximated is small.
Revised: In this section, we describe an efﬁcient estimation algorithm based on (Dagum et al., 2000), that is able to work with almost-uniform samples and also terminates quickly if the quantity being approximated is small.

Original: Two key insights inform the design of our approach: we ﬁrst observe that ε -almost uniform sampling can change the value of ρ at most by a factor of (1 + ε ) .
Revised: Two key insights inform the design of our approach: we ﬁrst observe that ε -almost uniform sampling can change the value of ρ at most by a factor of (1 + ε ) .

Original: Secondly, in typical scenarios, users are interested in two-sided bounds on ﬁdelity (as given by Eqn. 5) only if it is high enough.
Revised: Secondly, in typical scenarios, users are interested in two-sided bounds on ﬁdelity (as given by Eqn. 5) only if it is high enough.

Original: If the ﬁdelity is lower than some threshold, say 0.1, then it doesn’t matter if it is 0.05 or 0.01, since the explanation will be unacceptable in either case.
Revised: If the ﬁdelity is lower than some threshold, say 0.1, then it doesn’t matter if it is 0. or 0.01, since the explanation will be unacceptable in either case.

Original: In other words, below a certain threshold, one-sided bounds sufﬁce.
Revised: In other words, below a certain threshold, one-sided bounds sufﬁce.

Original: We present our estimation framework in Algs.
Revised: We present our estimation framework in Algs.

Original: 2 and 3.
Revised: 2 and 3.

Original: Our technique is abstract in the sense that it can estimate the density of samples satisfying any property (not just ﬁdelity), in any given population, so long as it is possible to sample almost-uniformly from the population (encapsulated in a procedure getSamples ).
Revised: Our technique is abstract in the sense that it can estimate the density of samples satisfying any property (not just ﬁdelity), in any given domain (not just Boolean), so long as it is possible to sample (almost) uniformly from the domain (encapsulated in the procedure getSamples ).

Original: We assume access to a procedure checkProperty , that given a sample s , returnsif the property of interest is satisﬁed by s and 0 otherwise.
Revised: We assume access to a procedure checkProperty , that given a sample s , returns 1 if the property of interest is satisﬁed by s and 0 otherwise.

Original: In practice, for measuring ﬁdelity of an explainer model on a subspace deﬁned by some φ , getSamples can invoke WAPS or Unigen3 on φ , while checkProperty is simply the indicator function I [ l f ( z ) =
Revised: In practice, for measuring ﬁdelity of an explainer model on a subspace deﬁned by some φ , getSamples can invoke WAPS or Unigen3 on φ , while checkProperty is simply the indicator function I [ l f ( z ) =

Original: =
Revised: =

Original: l g ( z (cid:48) )] .
Revised: l g ( z (cid:48) )] .

Original: Procedure computeEstimate ﬁrst invokes checkThreshold which returns True with high probability ( 1 − δ ) if the population mean is less than the prescribed threshold γ with tolerance ε .
Revised: Procedure computeEstimate ﬁrst invokes checkThreshold which returns True with high probability ( 1 − δ ) if the population mean is less than the prescribed threshold γ with tolerance ε .

Original: If the check fails, then computeEstimate makes a call to procedure AA (cid:48)
Revised: If the check fails, then computeEstimate makes a call to procedure AA (cid:48)

Original: (line 4), which is an adaptation of the algorithm by (Dagum et al., 2000) that provides guarantees similar to Eqn.
Revised: (line 4), which is an adaptation of the algorithm by (Dagum et al., 2000) that provides guarantees similar to Eqn.

Original: 5 with almost-uniform samples (see Appendix C.2).
Revised: 5 with almost-uniform samples (see Appendix D.1).

Original: Theorem 1 captures the guarantees and the behavior of the framework.
Revised: Theorem 1 captures the guarantees and the behavior of the framework.

Original: Theorem 1.
Revised: Theorem 1.

Original: If ρ ≤ γ − ε , then computeEstimate returns ⊥ with high probability (i.e. at least 1 − δ ).
Revised: If ρ ≤ γ − ε , then computeEstimate returns ⊥ with high probability (i.e. at least 1 − δ ).

Original: If ρ ≥ γ + ε , w.h.p.
Revised: If ρ ≥ γ + ε , w.h.p.

Original: , it returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ≤ (cid:98) ρ ≤ (1 + ε ) ρ ] ≥ (1 − δ ) .
Revised: , it returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ≤ (cid:98) ρ ≤ (1 + ε ) ρ ] ≥ (1 − δ ) .

Revised: The beneﬁt of leveraging the algorithm by (Dagum et al., 2000) is that it was proven to have close-tooptimal sample complexity.

Revised: We demonstrate that this yields a fast certiﬁcation tool in practice.

Revised: We implemented and ran Alg. 2, 3 on 150 benchmarks used in Narodytska et al.

Revised: The benchmarks are CNF formulas that encode the Anchor (Ribeiro et al., 2018) explanations of Binarized Neural

Revised: Networks trained on Adult, Recidivism and Lending datasets.

Original: [Figure]
Revised: The true ﬁdelity of an explanation can be computed from the count of the number of solutions of the corresponding formula. We compared the running-time of our tool to the time taken by the approach of (Narodytska et al., 2019), which utilizes the state-of-the-art approximate model-counting tool called ApproxMC (Soos et al., 2020). Note that both ApproxMC and our tool provide the same probabilistic guarantees on the returned estimate. The results are shown as a scatter-plot in Fig.

Revised: The x-coordinate of a point in blue represents the time taken by ApproxMC on a benchmark, while the y-coordinate represents the time taken by our approach.

Revised: As all the points are far below the diagonal dotted red line, we can infer that ApproxMC takes signiﬁcantly longer than our tool to compute the same estimate.

Revised: In fact, on average (geometric mean), our tool is 7 . 5 × faster than ApproxMC.

Revised: It is clear from Fig.

Revised: 2 that our algorithm scales far better than the alternative, despite being more general and model-agnostic.

Revised: We also experimentally compared the scalability of our tool to ApproxMC for different values of input tolerance ε and conﬁdence δ .

Revised: We found that our tool scales signiﬁcantly better than ApproxMC for tighter tolerance and conﬁdence values.

Revised: Thus, our experiments demonstrate that our tool is efﬁcient in practice.

Revised: We provide more details and results in Appendix D.4.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We experimentally demonstrate how CLIME can be used for obtaining more meaningful explanations on synthetic and real-world datasets, i.e. adult (Kohavi, 1996) and recidivism (Schmidt & Witte, 1988).
Revised: We experimentally demonstrate how CLIME can be used for obtaining more meaningful explanations on synthetic and real-world datasets, i.e. adult (Kohavi, 1996) and recidivism (Schmidt & Witte, 1988).

Original: We also evaluate its ability to detect adversarial attacks.
Revised: We also evaluate its ability to detect adversarial attacks.

Original: We focus on tabular data, but emphasize that CLIME is readily applicable to text and image data as well.
Revised: We focus on tabular data, but emphasize that CLIME is readily applicable to text and image data as well.

Original: Here we present results for recidivism (see extended version of results in Appendix D).
Revised: Here we present results for recidivism (see extended version of results in Appendix E). [Figure]

Original: Recidivism Dataset.
Revised: Recidivism Dataset.

Original: The Recidivism dataset was used to predict recidivism for individuals released from North Carolina prisons.
Revised: The Recidivism dataset was used to predict recidivism for individuals released from North Carolina prisons.

Original: We used the discretized version of dataset as in (Ribeiro et al., 2018).
Revised: We used the discretized version of dataset as in (Ribeiro et al., 2018).

Original: We trained a small MLP that gives 69.1%  on the test set.
Revised: We trained a small Multilayer Perceptron model that gives 69.1% accuracy on the test set.

Original: We consider two constrained spaces:
Revised: We consider two constrained spaces:

Original: • ‘female space’:
Revised: • ‘female space’:

Original: (J UNKY OR A LCOHOL ) AND (C RIME P ROPERTY ) AND G ENDER = F EMALE • ‘male space’:
Revised: (J UNKY OR A LCOHOL ) AND (C RIME P ROPERTY ) AND G ENDER = F EMALE • ‘male space’:

Original: (J UNKY OR A LCOHOL ) AND (C RIME P ROPERTY ) AND G ENDER =
Revised: (J UNKY OR A LCOHOL ) AND (C RIME P ROPERTY ) AND G ENDER =

Original: M ALE .
Revised: M ALE .

Original: We compute explanations for 50 samples within the corresponding constrained space.
Revised: We compute explanations for 50 samples within the corresponding constrained space.

Original: Figure 1 shows results for LIME (a) and CLIME (b), respectively, for both constrained spaces.
Revised: Figure 3 shows results for LIME (a) and CLIME (b), respectively, for both constrained spaces.

Original: At each plot, the blue histogram shows results for ‘female space’ and the orange histogram shows results for ‘male space’, respectively.
Revised: At each plot, the blue histogram shows results for ‘female space’ and the orange histogram shows results for ‘male space’, respectively.

Original: Each bar shows the average rank of a feature in explanations, e.g. if the height of the bar is 0.1 the corresponding feature is mostly top ranked in all explanations.
Revised: Each bar shows the average rank of a feature in explanations, e.g. if the height of the bar is 1 the corresponding feature is mostly top ranked in all explanations.

Original: First, we observe that the ‘gender’ and ‘crimeproperty’ features are ﬁxed in these sub-spaces.
Revised: First, we observe that the ‘Gender’ (‘Gndr’) and ‘Crimeproperty’ (‘CrPr’) features are ﬁxed in these sub-spaces.

Original: Hence, these features are always ranked last by CLIME demonstrating the correct behaviour (any feature with a constant value cannot inﬂuence the decision so it should have the zero coefﬁcient in the explainer model g deﬁned by  2).
Revised: Hence, these features are always ranked last by CLIME demonstrating the correct behaviour (any feature with a constant value cannot inﬂuence the decision so it should have the zero coefﬁcient in the explainer model g deﬁned by Eqn. 2).

Original: Note that LIME is not informative enough for distinguishing between these constrained spaces.
Revised: Note that LIME is not informative enough for distinguishing between these constrained spaces.

Original: However, results of CLIME are very interesting.
Revised: However, results of CLIME are very interesting.

Original: We observe that ‘race’ is one of the top feature to make decision for female prisoners (the ‘race’ blue bar has a small height) and only 4th (on average) for males .
Revised: We observe that ‘Race’ is one of the top feature to make decision for female prisoners (the ‘Race’ blue bar has a small height) and only 4th (on average) for males .

Original: On the other hand, whether a male prisoner is married and has alcohol problems make the most difference which is also an interesting potential bias that we reveal.
Revised: On the other hand, whether a male prisoner is ‘Married’ (‘Mard’) make the most difference which is also an interesting potential bias that we reveal.

Original: Detecting Adversarial Attacks.
Revised: Detecting Adversarial Attacks.

Original: (Slack et al., 2020) presented a way to craft an adversarial attack that seeks to hide the biased predictions made by a ‘bad’ classiﬁer (ex: one that decides credit card applications solely on a sensitive feature like race) from detection by post-hoc explainers like LIME.
Revised: (Slack et al., 2020) presented a way to craft an adversarial attack that seeks to hide the biased predictions made by a ‘bad’ classiﬁer (ex: one that decides credit card applications solely on a sensitive feature like race) from detection by post-hoc explainers like LIME.

Original: Given a biased classiﬁer, their idea was to construct an adversarial classiﬁer that outputs the biased classiﬁer’s prediction if an input instance is detected to be in-distribution, and outputs a prediction based on an innocuous feature if the input instance is OOD.
Revised: Given a biased classiﬁer, their idea was to construct an adversarial classiﬁer that outputs the biased classiﬁer’s prediction if an input instance is detected to be in-distribution, and outputs a prediction based on an innocuous feature if the input instance is OOD.

Original: When LIME is used for explaining a prediction made by the adversarial classiﬁer, almost all of LIME’s perturbed samples are detected to [Figure] OOD, resulting in the innocuous feature appearing as the 
Revised: When LIME is used for explaining a prediction made by the adversarial classiﬁer, almost all of LIME’s perturbed samples are detected to be OOD, resulting in the innocuous feature appearing as the top explanation.

Original: In this  the  successfully hidden
Revised: In this way, the sensitive feature used for deciding in-distribution instances is successfully hidden.

Original: The  on the ability to train a classiﬁer to distinguish between LIME’s perturbed samples and from the original distribution with high accuracy.
Revised: The attack crucially relies on the ability to train a classiﬁer to distinguish between LIME’s perturbed samples and instances

Original: In this experiment, we explore how CLIME can be used for detecting such attacks
Revised: [Figure]

Original: We hypothesized that for discrete data, the distinguishing classiﬁer should have trouble differentiating between instances from the original dataset and instances with low Hamming distances to points in the original dataset.
Revised: We hypothesized that for discrete data, the distinguishing classiﬁer should have trouble differentiating between instances from the original dataset and instances with low Hamming distances to points in the original dataset.

Original: For a given Hamming distance h and a set of instances S , it is possible to construct a formula φ h such that the solutions of φ h are exactly those instances that are at most at a distance h from each instance in S .
Revised: For a given Hamming distance h and a set of instances S , it is possible to construct a formula φ h such that the solutions of φ h are exactly those instances that are at most at a distance h from each instance in S .

Original: We took S to be the set of 3 nearest neighbors of the instance being explained.
Revised: We took S to be the set of 3 nearest neighbors of the instance being explained.

Original: If our hypothesis is correct, the adversarial classiﬁer should have trouble hiding the sensitive features in the subspace Z φ h for low values of h .
Revised: If our hypothesis is correct, the adversarial classiﬁer should have trouble hiding the sensitive features in the subspace Z φ h for low values of h .

Original: Thus, sensitive features should appear in the top CLIME explanations when h is small, while innocuous features should dominate for larger h .
Revised: Thus, sensitive features should appear in the top CLIME explanations when h is small, while innocuous features should dominate for larger h .

Original: To test our hypothesis, we generated 200 CLIME explanations using discretized versions of the three datasets used by (Slack et al., 2020).
Revised: To test our hypothesis, we generated 200 CLIME explanations using discretized versions of the three datasets used by (Slack et al., 2020).

Original: We computed the relative frequency of a feature appearing as the top explanation for different values of h .
Revised: We computed the relative frequency of a feature appearing as the top explanation for different values of h .

Original: The results for the Recidivism dataset with 1 innocuous feature are depicted in Figure 2a (we defer results on other datasets and an extended discussion to
Revised: The results for the Recidivism dataset with 1 innocuous feature are depicted in Figure 4a (we defer results on other datasets and an extended discussion to

Original: Appendix D.3 for lack of space).
Revised: Appendix E.3 for lack of space).

Original: We plot the hamming distance h on the X-axis and the frequency of seeing a feature as the top explanation on the Y-axis.
Revised: We plot the hamming distance h on the X-axis and the frequency of seeing a feature as the top explanation on the Y-axis.

Original: We see the sensitive feature ‘race’ appears as the top explanation for 60% of the instances explained when h
Revised: We see the sensitive feature ‘race’ appears as the top explanation for 60% of the instances explained when h

Original: = 2 .
Revised: = 2 .

Original: The innocuous feature (1st
Revised: The innocuous feature (1st

Original: Unrelated Feature), appears almost exclusively as the top explanation when h ≥ 5 .
Revised: Unrelated Feature), appears almost exclusively as the top explanation when h ≥ 5 .

Original: For comparison, we depict results on CLIME explanations for a non-adversarial classiﬁer in Fig. 2b.
Revised: For comparison, we depict results on CLIME explanations for a non-adversarial classiﬁer in Fig. 4b.

Original: We see that the relative frequency with which a feature appears as the top explanation changes very little for different hamming distances.
Revised: We see that the relative frequency with which a feature appears as the top explanation changes very little for different hamming distances.

Original: Note that for LIME explanations (not shown), the innocuous feature appears as the top explanation for all of the instances explained.
Revised: Note that for LIME explanations (not shown), the innocuous feature appears as the top explanation for all of the instances explained.

Original: This clearly demonstrates CLIME’s ability to detect adversarial attacks.
Revised: While it may be possible to craft even more sophisticated attacks, these results clearly demonstrates CLIME’s ability to detect adversarial attacks that exploit OOD sampling.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: We presented a model-agnostic explanation framework CLIME that is able to operate on constrained subspaces of inputs.
Revised: We presented a model-agnostic explanation framework CLIME that is able to operate on constrained subspaces of inputs.

Original: We introduced a new metric for quantifying the severity of the OOD sampling problem and empirically demonstrated CLIME’s ability to mitigate it.
Revised: We introduced a new metric for quantifying the severity of the OOD sampling problem and empirically demonstrated CLIME’s ability to mitigate it.

Original: Additionally, our new estimation algorithm enables computation of an explanation’s quality up to any desired accuracy.
Revised: Additionally, our new estimation algorithm enables computation of an explanation’s quality up to any desired accuracy.

Original: The need for making XAI more rigorous and evidence-based has been highlighted in the past (Doshi-Velez &
Revised: The need for making XAI more rigorous and evidence-based has been highlighted in the past (Doshi-Velez &

Original: Kim, 2017), and we believe our framework takes a concrete step in this direction.
Revised: Kim, 2017), and we believe our framework takes a concrete step in this direction.

Original: CLIME can also be readily extended in numerous ways.
Revised: CLIME can also be readily extended in numerous ways.

Original: Helping the user with deﬁning relevant subspaces by mining constraints from data is an interesting direction.
Revised: Helping the user with deﬁning relevant subspaces by mining constraints from data is an interesting direction.

Original: Once sampling tools improve, richer constraint languages like SMT (Barrett & Tinelli, 2018) can provide even more ﬂexibility.
Revised: Once sampling tools improve, richer constraint languages like SMT (Barrett & Tinelli, 2018) can provide even more ﬂexibility.

Original: Construction of
Revised: Construction of

Original: CLIME’s explainer model can also incorporate Shapley values as in (Lundberg & Lee, 2017).
Revised: CLIME’s explainer model can also incorporate Shapley values as in (Lundberg & Lee, 2017).

Original: A R ELATED WORK
Revised: A R ELATED WORK

Original: Model explainability is one of the most important problems in machine learning.
Revised: Model explainability is one of the most important problems in machine learning.

Original: Therefore, there are a large number of recent surveys on the topic, e.g. Hoffman & Klein (2017); Hoffman et al.
Revised: Therefore, there are a large number of recent surveys on the topic, e.g. Hoffman & Klein (2017); Hoffman et al.

Original: Lipton (2018); Adadi & Berrada (2018); Guidotti et al.
Revised: Lipton (2018); Adadi & Berrada (2018); Guidotti et al.

Original: ; Rudin (2019).
Revised: ; Rudin (2019).

Original: To overview, we partition approaches to generate explanations for ML models into two groups based on whether they provide theoretical guarantees on the quality of the generated explanations.
Revised: To overview, we partition approaches to generate explanations for ML models into two groups based on whether they provide theoretical guarantees on the quality of the generated explanations.

Original: Explanations without theoretical guarantees.
Revised: Explanations without theoretical guarantees.

Original: There were a number of approaches proposed to compute (model-agnostic) local explanations.
Revised: There were a number of approaches proposed to compute (model-agnostic) local explanations.

Original: We have overviewed LIME (Ribeiro et al., 2016) in
Revised: We have overviewed LIME (Ribeiro et al., 2016) in

Original: Section 2.
Revised: Section 2.

Original: Anchor is a successor of LIME (Ribeiro et al., 2018).
Revised: Anchor is a successor of LIME (Ribeiro et al., 2018).

Original: The main contribution of Anchor is to produce explanations that hold globally, for the entire distribution of inputs.
Revised: The main contribution of Anchor is to produce explanations that hold globally, for the entire distribution of inputs.

Original: SHAP (Lundberg &
Revised: SHAP (Lundberg &

Original: Lee, 2017) is another popular model-agnostic explainer to produce local explanations.
Revised: Lee, 2017) is another popular model-agnostic explainer to produce local explanations.

Original: Similar to other explainers, SHAP does not provide any theoretical justiﬁcation for the sampling procedure.
Revised: Similar to other explainers, SHAP does not provide any theoretical justiﬁcation for the sampling procedure.

Original: However, SHAP employs game theoretic principles to produce an explainable model.
Revised: However, SHAP employs game theoretic principles to produce an explainable model.

Original: Our work focuses on model-agnostic, local explanations, however, we produce explanations with provable guarantees.
Revised: Our work focuses on model-agnostic, local explanations, however, we produce explanations with provable guarantees.

Original: CXPlain proposes to train an additional ‘explanation model’ to provide explanations for a given ML model (Schwab & Karlen, 2019).
Revised: CXPlain proposes to train an additional ‘explanation model’ to provide explanations for a given ML model (Schwab & Karlen, 2019).

Original: Learning of the explanation model is involve an estimation of feature importance using a causal objective.
Revised: Learning of the explanation model involves an estimation of feature importance using a causal objective.

Original: The causal objective captures how input features cause a marginal improvement in the predictive performance of the ML model.
Revised: The causal objective captures how input features cause a marginal improvement in the predictive performance of the ML model.

Original: In our work, we do not consider each feature individually and reason about the space of features as a whole.
Revised: In our work, we do not consider each feature individually and reason about the space of features as a whole.

Original: Moreover, our framework allows us to work with constrained spaces.
Revised: Moreover, our framework allows us to work with constrained spaces.

Revised: Finally, works such as (Lakkaraju et al., 2019) provide limited capabilities for customizing global explanations by letting the user supply a set of features that the they deem important.

Revised: Similar to (Björklund et al., 2019), they avoid sampling a neighbourhood around a given point by using original data points to construct an explainer.

Revised: While avoiding sampling helps scalability, it also undermines applicability.

Revised: For instance, dealing with user-deﬁned constraints, as well as unbalanced or skewed input datasets can be problematic.

Revised: In both cases, the input data may be too sparse to yield meaningful explanations.

Original:  Another line of work in on gradient-based explainers, for example, saliency maps (Zeiler & Fergus, 2014), Integrated Gradient (Sundararajan et al., 2017), DeepLIFT (Shrikumar et al., 2017).
Revised: Recently, Lakkaraju et al. (2020) demonstrated that these explanations are less robust compared to LIME, for example. Another line of work in on gradient-based explainers, for example, saliency maps (Zeiler & Fergus, 2014), Integrated Gradient (Sundararajan et al., 2017), DeepLIFT (Shrikumar et al., 2017).

Original: Gradientbased methods assume full knowledge about the ML model and, also, require these models to be differentiable.
Revised: Gradientbased methods assume full knowledge about the ML model and, also, require these models to be differentiable.

Original: While these methods are very efﬁcient, they do not provide theoretical guarantees on the produced explanations.
Revised: While these methods are very efﬁcient, they do not provide theoretical guarantees on the produced explanations.

Original: On top of that these approaches are not model-agnostic.
Revised: On top of that these approaches are not model-agnostic.

Original: Explanations with theoretical guarantees.
Revised: Explanations with theoretical guarantees.

Original: Recently, a formal approach to analyze explanation of
Revised: Recently, a formal approach to analyze explanation of

Original: ML models was proposed.
Revised: ML models was proposed.

Original: If an ML model allows a formal representation in restricted fragment of the ﬁrst order logic, then one can (a) deﬁne a formal notion of an explanation and (b) design an algorithm to produce these explanations (Shih et al., 2018; 2019; Ignatiev et al., 2019; Darwiche & Hirth, 2020; Darwiche, 2020).
Revised: If an ML model allows a formal representation in restricted fragment of the ﬁrst order logic, then one can (a) deﬁne a formal notion of an explanation and (b) design an algorithm to produce these explanations (Shih et al., 2018; 2019; Ignatiev et al., 2019; Darwiche & Hirth, 2020; Darwiche, 2020).

Original: One of the formal approaches is built on powerful knowledge compilation techniques, e.g. (Shih et al., 2018; 2019).
Revised: One of the formal approaches is built on powerful knowledge compilation techniques, e.g. (Shih et al., 2018; 2019).

Original: The other approach employees very efﬁcient formal reasoners, like SAT, SMT or ILP solvers, as a part of explanation generation algorithms (Ignatiev et al., 2019; Narodytska et al., 2019).
Revised: The other approach employees very efﬁcient formal reasoners, like SAT, SMT or ILP solvers, as a part of explanation generation algorithms (Ignatiev et al., 2019; Narodytska et al., 2019).

Original: If the process of ML model compilation into a tractable structure is feasible then the ﬁrst approach is very effective and allows the user to analyse the ML model efﬁciently.
Revised: If the process of ML model compilation into a tractable structure is feasible then the ﬁrst approach is very effective and allows the user to analyse the ML model efﬁciently.

Original: However, the compilation can be computationally expensive and resource demanding, so the second approach is more efﬁcient in some applications.
Revised: However, the compilation can be computationally expensive and resource demanding, so the second approach is more efﬁcient in some applications.

Original: There are some limitations of these approaches.
Revised: There are some limitations of these approaches.

Original: First, similar to gradient-based methods, they require full knowledge of the original ML model.
Revised: First, similar to gradient-based methods, they require full knowledge of the original ML model.

Original: Second, in practice, these approaches face scalability issues as reasoning about ML models formally is computationally expensive.
Revised: Second, in practice, these approaches face scalability issues as reasoning about ML models formally is computationally expensive.

Original: Quality of the explanations.
Revised: Quality of the explanations.

Original: Finally, we consider a recent line of work on analysis of the quality of explanations.
Revised: Finally, we consider a recent line of work on analysis of the quality of explanations.

Original: Ribeiro et al.
Revised: Ribeiro et al.

Original: (2018)observed that a small perturbation of the input instance might leads to changes in the explanation which are not expected by the user. Moreover, the authors proposed several heuristic measures to evaluate quality of explanations, including the estimated precision of the explanation, i.e. the percentage of instances where the explanation applies and the prediction matches, andthe estimated coverage, i.e. the fraction of the input space where this explanation applies.
Revised: (2018) proposed several heuristic measures to evaluate quality of explanations including ﬁdelity and coverage, but do not provide a way to estimate the true value of these metrics.

Original: It was shown  that explanations are susceptible to adversarial attacks and lack robustness property using perturbations-based methods (Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018).
Revised: In (Ghorbani et al., 2019; Alvarez-Melis & Jaakkola, 2018), it was shown using perturbation-based methods that explanations are susceptible to adversarial attacks and lack robustness property. For example, Zhang et al.

Revised: (2019) investigated several sources of uncertainty in LIME, like sampling variance in explaining a sample.

Revised: The authors experimentally demonstrated that LIME often fails to capture the most important features locally.

Revised: However, the paper does not propose a solution to remedy identiﬁed issues.

Original: Moreover, Slack et al.
Revised: Moreover, Slack et al.

Original: (2020) showed that it is easy to fool an explainer, like LIME and SHAP, as we discussed in details in Section 5.
Revised: (2020) showed that it is easy to fool an explainer, like LIME and SHAP, as we discussed in detail in Section 5.

Original: Narodytska et al.
Revised: Narodytska et al.

Original: (2019) presented a technique for evaluation quality of explanations based on model counting with theoretical guarantees on the quality of the estimate.
Revised: (2019) presented a technique for evaluation quality of explanations based on model counting, but their approach suffers from scalability issues (as shown in Sec. 4.2) and is only applicable to BNNs.

Original: Finally, Lakkaraju et al.
Revised: Lakkaraju et al.

Original: (2020) proposed to usethe adversarial training (Madry et al., 2018) to improve robustness of the explanations.
Revised: (2020) proposed to use adversarial training (Madry et al., 2018) to improve robustness of the explanations.

Original: While the proposed approach improves robustness to adversarial attacks it cannot be easily extended to work in constraint environments and do not provide theoretical guarantees on the ﬁdelity of the explanations.
Revised: While the proposed approach improves robustness to adversarial attacks it cannot be easily extended to work in constraint environments and does not provide theoretical guarantees on the ﬁdelity of the explanations.

Revised: A related line of work on probabilistic veriﬁcation of ML models has seen a surge in interest.

Revised: (Albarghouthi et al., 2017) encoded the underlying model and fairness properties as formulas in SMT over real arithmentic, and relied on symbolic integration techniques.

Revised: However, this approach is known not to scale, eg.

Revised: it can only handle neural networks with a single hidden layer containing just three hidden units.

Original:  B A NALYSIS OF LIME (A DDITIONAL MATERIALS ) B. P ERTURBATION P ROCEDURE
Revised: (Bastani et al., 2019) present an alternative approach that uses Monte Carlo sampling and adaptive concentration inequalities. However, unlike Alg. 2, their method only returns a yes/no answer and does not provide a quantitative estimate. Further, their algorithm may fail to terminate on some inputs, and the sample complexity is not proven to be close-to-optimal. B A NALYSIS OF LIME (A DDITIONAL MATERIALS ) B. P ERTURBATION P ROCEDURE

Original: LIME generates the neighborhood of the data point under explanation differently, depending on whether the given dataset is text, image or tabular.
Revised: LIME generates the neighborhood of the data point under explanation differently, depending on whether the given dataset is text, image or tabular.

Original: For text and images, LIME ﬁrst maps a given point x ∈ R d to x (cid:48) ∈ { 0 , 1 } d (cid:48) and then samples z (cid:48) ∈ { 0 , 1 } d (cid:48) in the neighborhood of x (cid:48) .
Revised: For text and images, LIME ﬁrst maps a given point x ∈ R d to x (cid:48) ∈ { 0 , 1 } d (cid:48) and then samples z (cid:48) ∈ { 0 , 1 } d (cid:48) in the neighborhood of x (cid:48) .

Original: For tabular data, LIME ﬁrst samples points z i in the neighborhood of x and then maps them to the interpretable domain as z (cid:48) i .
Revised: For tabular data, LIME ﬁrst samples points z i in the neighborhood of x and then maps them to the interpretable domain as z (cid:48) i .

Original: The distance function is also deﬁned on the binarized space instead of the original space.
Revised: The distance function is also deﬁned on the binarized space instead of the original space.

Original: Text and Images For text and image data, the given point x is ﬁrst mapped to the Boolean domain.
Revised: Text and Images For text and image data, the given point x is ﬁrst mapped to the Boolean domain.

Original: For text, the Boolean domain is simply the presence or absence of each word in x , while for images, x is ﬁrst divided into superpixels, and the Boolean domain is represented by the presence or absence of each superpixel.
Revised: For text, the Boolean domain is simply the presence or absence of each word in x , while for images, x is ﬁrst divided into superpixels, and the Boolean domain is represented by the presence or absence of each superpixel.

Original: Points z (cid:48) in the neighborhood of x (cid:48) are sampled by ﬂipping a few 1s to 0s where both the number and positions of 1s to ﬂip are chosen randomly.
Revised: Points z (cid:48) in the neighborhood of x (cid:48) are sampled by ﬂipping a few 1s to 0s where both the number and positions of 1s to ﬂip are chosen randomly.

Original: Each z (cid:48)
Revised: Each z (cid:48)

Original: i is mapped back to the original domain to obtain z i which are then input to f for obtaining the ‘true’ labels.
Revised: i is mapped back to the original domain to obtain z i which are then input to f for obtaining the ‘true’ labels.

Original: Interestingly,
Revised: Interestingly,

Original: LIME’s sampling procedure for text and images does not depend on the training set which is not the case for tabular data.
Revised: LIME’s sampling procedure for text and images does not depend on the training set which is not the case for tabular data.

Original: Tabular Tabular data can consist of both continuous and categorical features.
Revised: Tabular Tabular data can consist of both continuous and categorical features.

Original: Although LIME can work with continuous data natively, the recommended way is to perform quantization ﬁrst using statistics of the original dataset.
Revised: Although LIME can work with continuous data natively, the recommended way is to perform quantization ﬁrst using statistics of the original dataset.

Original: We denote I j = ∪ M j k
Revised: We denote I j = ∪ M j k

Original: =

Revised: =

Original: { I jk } the set of all intervals that we obtain due to discretization of the j th feature.
Revised: { I jk } the set of all intervals that we obtain due to discretization of the j th feature.

Original: To construct a neighborhood Z LIME uniformly samples features values from each interval in I i for the i th feature.
Revised: To construct a neighborhood Z LIME uniformly samples features values from each interval in I i for the i th feature.

Original: For categorical features, LIME builds a distribution of feature values and samples values for each feature in proportion to their frequencies.
Revised: For categorical features, LIME builds a distribution of feature values and samples values for each feature in proportion to their frequencies.

Original: Construction of an interpretable domain.
Revised: Construction of an interpretable domain.

Original: Next, we consider how an interpretable binary domain is produced.
Revised: Next, we consider how an interpretable binary domain is produced.

Original: For each sample z in the neighborhood Z , the corresponding z (cid:48) is constructed as
Revised: For each sample z in the neighborhood Z , the corresponding z (cid:48) is constructed as

Original: Discretized features: 
Revised: Discretized features: z (cid:48)

Original: [Equation]
Revised: [Equation]

Original:  Categorical 
Revised: [Equation] [Equation] [Equation] Categorical features: z (cid:48)

Original: [Equation]
Revised: [Equation]

Original:  We denote bin ( z ) the mapping function from the original feature vector z to a binary vector z (cid:48) , z (cid:48)
Revised: 0 , otherwise [Equation] We denote bin ( z ) the mapping function from the original feature vector z to a binary vector z (cid:48) , z (cid:48)

Original: =
Revised: =

Original: bin ( z ) .
Revised: bin ( z ) .

Original: Thus Z (cid:48) =
Revised: Thus Z (cid:48) =

Original: bin ( Z ) .
Revised: bin ( Z ) .

Original: As this mapping is many-to-one, we denote pre-image ( z (cid:48) )
Revised: As this mapping is many-to-one, we denote pre-image ( z (cid:48) )

Original: the set of samples that are mapped to z (cid:48) :
Revised: the set of samples that are mapped to z (cid:48) :

Original: [Equation]
Revised: [Equation]

Original: Interpretation of an explanation.
Revised: Interpretation of an explanation.

Original: We recall the meaning of explanation for a vector z given a linear regression model g deﬁned by Eqn. 2.
Revised: We recall the meaning of explanation for a vector z given a linear regression model g deﬁned by Eqn. 2.

Original: Suppose the i th coefﬁcient is great than zero, c i > 0 .
Revised: Suppose the i th coefﬁcient is great than zero, c i > 0 .

Original: Then the value c i is interpreted as a contribution toward the class 1.
Revised: Then the value c i is interpreted as a contribution toward the class 1.

Original: Namely, if the i th feature f i is not equal to z i then on average the value of g ( z ) decreases by c i .
Revised: Namely, if the i th feature f i is not equal to z i then on average the value of g ( z ) decreases by c i .

Original: To see why it is so, we consider the binary space z (cid:48)
Revised: To see why it is so, we consider the binary space z (cid:48)

Original: =
Revised: =

Original: bin ( z ) .
Revised: bin ( z ) .

Original: If f i (cid:54) =
Revised: If f i (cid:54) =

Original: z i then z (cid:48) i
Revised: z i then z (cid:48) i

Original: = 0 .
Revised: = 0 .

Original: Hence, contribution of the term c
Revised: Hence, contribution of the term c

Original: i z (cid:48)
Revised: i z (cid:48)

Original: i to the value of g ( z (cid:48) ) is decreased by c i .
Revised: i to the value of g ( z (cid:48) ) is decreased by c i .

Original: Similarly, if c i < 0 then the value c i is interpreted as a contribution to the class 0.
Revised: Similarly, if c i < 0 then the value c i is interpreted as a contribution to the class 0.

Original: If f i (cid:54) =
Revised: If f i (cid:54) =

Original: z i than the value of g ( z (cid:48) ) is increased by c i .
Revised: z i than the value of g ( z (cid:48) ) is increased by c i .

Original: B. D ISCUSSION ON THE QUALITY OF THE PERTURBATION PROCEDURE
Revised: B. D ISCUSSION ON THE QUALITY OF THE PERTURBATION PROCEDURE

Original: The quality of explanations depends on the closeness of samples Z to samples from the original distribution D .
Revised: The quality of explanations depends on the closeness of samples Z to samples from the original distribution D .

Original: In Section 3, we introduce a simple metric SQ ( E ) =
Revised: In Section 3, we introduce a simple metric SQ ( E ) =

Original: W Z (cid:48) /W
Revised: W Z (cid:48) /W

Original: D (see Eqn. 3) to measure the quality of Z .
Revised: D (see Eqn. 3) to measure the quality of Z .

Original: Please note that the SQ metric might look similar to the ﬁdelity metric that is deﬁned in Eqn. 1.
Revised: Please note that the SQ metric might look similar to the ﬁdelity metric that is deﬁned in Eqn. 1.

Original: However, these metrics are different.
Revised: However, these metrics are different.

Original: SQ measures quality of the sampling procedure based on the behavior of an ideal classiﬁer on the generated samples, while (cid:98) ρ evaluates the closeness of two models g and f .S S S
Revised: SQ measures quality of the sampling procedure based on the behavior of an ideal classiﬁer on the generated samples, while (cid:98) ρ evaluates the closeness of two models g and f .

Original: S [Figure] [Figure] We recall that if the sampling procedure is correct then SQ ( E ) should be close to 1 for a perfect classiﬁer.
Revised: We recall that if the sampling procedure is correct then SQ ( E ) should be close to 1 for a perfect classiﬁer.

Original: While we can always experimentally measure W Z (cid:48) , the value W D cannot be compute in general.
Revised: While we can always experimentally measure W Z (cid:48) , the value W D cannot be compute in general.

Original: However, we will use few crafted classiﬁcation tasks where W D can be computed exactly to highlight the difference in behaviour of LIME and our proposed procedure.
Revised: However, we will use few crafted classiﬁcation tasks where W D can be computed exactly to highlight the difference in behaviour of LIME and our proposed procedure.

Original: The following crafted example is designed to highlight that OOD sampling occurs and our SQ ( E ) metric is able to detect it.
Revised: The following crafted example is designed to highlight that OOD sampling occurs and our SQ ( E ) metric is able to detect it.

Original: Moreover, we show that the quality as explanations suffer due to the OOD sampling.
Revised: Moreover, we show that the quality as explanations suffer due to the OOD sampling.

Original: Example B.1 (‘Hit or miss’) .
Revised: Example B.1 (‘Hit or miss’) .

Original: We deﬁne the classiﬁcation problem to illustrate issues with the OOD sampling.
Revised: We deﬁne the classiﬁcation problem to illustrate issues with the OOD sampling.

Original: This problem models a situation when the classiﬁer relies on seeing speciﬁc feature values in the input sample to make its decision.
Revised: This problem models a situation when the classiﬁer relies on seeing speciﬁc feature values in the input sample to make its decision.

Original: As we mentioned above, the reason we work with a crafted problem is that we can compute the value W D exactly, which is not possible for real world datasets.
Revised: As we mentioned above, the reason we work with a crafted problem is that we can compute the value W D exactly, which is not possible for real world datasets.

Original: Deﬁnition of a task.
Revised: Deﬁnition of a task.

Original: We consider two areas S 0 and S 1 on the 2D plane (see Figure 3(a)).
Revised: We consider two areas S 0 and S 1 on the 2D plane (see Figure 1(a)).

Original: A sample z , z ∼ D , represents integer coordinates of three points p 1 , p 2 and p 3 drawn from S 0 ∪ S 1 .
Revised: A sample x , x ∼ D , represents integer coordinates of three points p 1 , p 2 and p 3 drawn from S 0 ∪ S 1 .

Original: Hence, we have six features vector x =
Revised: Hence, we have six features vector x =

Original: [ x 1 , x 2 , . . .
Revised: [ x 1 , x 2 , . . .

Original: , x 6 ] , where p i =
Revised: , x 6 ] , where p i =

Original: ( x 2 i − 1 , x 2 i ) , i = 1 , 2 , 3 , is a 2D point, or x = [ p 1 , p 2 , p 3 ] =
Revised: ( x 2 i − 1 , x 2 i ) , i = 1 , 2 , 3 , is a 2D point, or x = [ p 1 , p 2 , p 3 ] =

Original: [( x 1 , x 2 ) , ( x 3 , x 4 ) , ( x 5 , x 6 )] .
Revised: [( x 1 , x 2 ) , ( x 3 , x 4 ) , ( x 5 , x 6 )] .

Original: If there is p i ∈ x s.t.
Revised: If there is p i ∈ x s.t.

Original: p i / ∈ S 

Revised: p i / ∈ S

Original: ∩ S 1 then x is an out-of-distribution sample.
Revised: ∩ S 1 then x is an out-of-distribution sample.

Original: For example, [(7 , 7) , (7 , 7) , (10 , 10)] is an in-distribution sample, while [(1 , 10) , (7 , 7) , (10 , 10)] is an out-of-distribution sample as the ﬁrst point (1 , 10) / ∈ S 

Revised: For example, [(7 , 7) , (7 , 7) , (10 , 10)] is an in-distribution sample, while [(1 , 10) , (7 , 7) , (10 , 10)] is an out-of-distribution sample as the ﬁrst point (1 , 10) / ∈ S

Original: ∩ S 1 .
Revised: ∩ S 1 .

Original: Given a sample x ∼ D , we say that x belongs to C 1 iff there exists i s.t.
Revised: Given a sample x ∼ D , we say that x belongs to C 1 iff there exists i s.t.

Original: p i ∈ S 1 , i ∈
Revised: p i ∈ S 1 , i ∈

Original: [1 , 3] , and belongs to C 0 otherwise.
Revised: [1 , 3] , and belongs to C 0 otherwise.

Original: For example, [(7 , 7) , (7 , 7) , (10 , 10)] belongs to C 1 as the third point (10 , 10) ∈ S 1 .
Revised: For example, [(7 , 7) , (7 , 7) , (10 , 10)] belongs to C 1 as the third point (10 , 10) ∈ S 1 .

Original: So, if one of three points in x hits S 1 then x is classiﬁed as C 1 .
Revised: So, if one of three points in x hits S 1 then x is classiﬁed as C 1 .

Original: In this example, an explanation for a sample x is well-deﬁned as points in the S 0 (resp.
Revised: In this example, an explanation for a sample x is well-deﬁned as points in the S 0 (resp.

Original: S 1 ) area contribute to the C(resp.
Revised: S 1 ) area contribute to the C(resp.

Original: C 1 ) class, respectively.
Revised: C 1 ) class, respectively.

Original: LIME performance.
Revised: LIME performance.

Original: We train an almost perfect neural network-based classiﬁer for this simple example (See more in Section D.1).
Revised: We train an almost perfect neural network-based classiﬁer for this simple example (See more in Section E.1).

Original: We compute our metric SQ for LIME and get that SQ ( LIME )
Revised: We compute our metric SQ for LIME and get that SQ ( LIME )

Original: = 0 .
Revised: = 0 .

Original: 75 which is less than 1 hinting that the sampling procedure is biased.
Revised: 75 which is less than 1 hinting that the sampling procedure is biased.

Original: To see how SQ affects explanations, we generate LIME explanations for a samples x
Revised: To see how SQ affects explanations, we generate LIME explanations for a samples x

Original: = [(7 , 7) , (7 , 7) , (7 , 7)] that is close to the corner (8 , 8) .
Revised: = [(7 , 7) , (7 , 7) , (7 , 7)] that is close to the corner (8 , 8) .

Original: As all points in x are in S 0 , x belongs to class C 0 .
Revised: As all points in x are in S 0 , x belongs to class C 0 .

Original: Table 1 shows explanations for this point in the column ‘Balanced dataset’.
Revised: Table 1 shows explanations for this point in the column ‘Balanced dataset’.

Original: The ﬁrst row shows explanations produced by LIME.
Revised: The ﬁrst row shows explanations produced by LIME.

Original: The red horizontal bar indicates that the corresponding feature has a negative coefﬁcient in g
Revised: The red horizontal bar indicates that the corresponding feature has a negative coefﬁcient in g

Original: and it contributes toward C 0 .
Revised: and it contributes toward C 0 .

Original: Symmetrically, the green bar indicates that the feature contributes toward C 1 .
Revised: Symmetrically, the green bar indicates that the feature contributes toward C 1 .

Original: We can see that an explanation of x is incorrect as some features with values less than 8 have positive contributions, e.g. features x 4 and x 5 have green bars, so they contribute toward the wrong class C 1 .
Revised: We can see that an explanation of x is incorrect as some features with values less than 8 have positive contributions, e.g. features x 4 and x 5 have green bars, so they contribute toward the wrong class C 1 .

Original: In our experimental evaluation we sampled instances around the corner (8,8) with p i ∈ S 0 , i ∈ [1 , 3] .
Revised: In our experimental evaluation we sampled instances around the corner (8,8) with p i ∈ S 0 , i ∈ [1 , 3] .

Original: So, all these samples belong to the class
Revised: So, all these samples belong to the class

Original: C 0 , hence, all features should have negative coefﬁcients the liner model g .
Revised: C 0 , hence, all features should have negative coefﬁcients the liner model g .

Original: We found that LIME explanations have 25% of features with positive coefﬁcients in g that contribute to the wrong class
Revised: We found that LIME explanations have 25% of features with positive coefﬁcients in g that contribute to the wrong class

Original: C 1 which is an incorrect behaviour.
Revised: C 1 which is an incorrect behaviour.

Original: [Table]
Revised: [Table]

Original: x = [
Revised: (Example C.3)

Original: [Table] [Table] [Table]
Revised: (Examples B.1, C.1)

Original: ] x= [Table]
Revised: (Example C.2) sample x

Original: [Table] ] x = [ [Table] [Table]
Revised: = [7 ,..., 7] x = [7 ,..., 7] x = [1 , 1 , 1 , 1 , 12 ..., 12]

Original: [Table] [Table] [Table]
Revised: LIME

Original: [Table]
Revised: CLIME

Original: C G ENERATINGAND C ERTIFYING C ONSTRAINT -D RIVEN E XPLANATIONS (A DDITIONAL MATERIALS )C. C ASE STUDIES MANIFESTING COMPLICATIONS DUE TO OOD.
Revised: C G ENERATING C ONSTRAINT -D RIVEN E XPLANATIONS (A DDITIONAL MATERIALS )

Original: In this section we consider several examples to demonstrate undesired consequences of OOD sampling for LIME and how these are resolved by CLIME.
Revised: In this section we consider several examples to demonstrate undesired consequences of OOD sampling for LIME and how these are resolved by CLIME.

Original: Each of these examples is designed to model an abstraction of the real user-case.
Revised: Each of these examples is designed to model an abstraction of the real user-case.

Original: Moreover, for these simpliﬁed tasks, we can estimate quality of the sampling procedure using our metric SQ as we can compute W D analytically.
Revised: Moreover, for these simpliﬁed tasks, we can estimate quality of the sampling procedure using our metric SQ as we can compute W D analytically.

Original: Example C.1 (‘Hit and miss continued’) .
Revised: Example C.1 (‘Hit and miss continued’) .

Original: We come back to Example B.1 to evaluate performance of
Revised: We come back to Example B.1 to evaluate performance of

Original: CLIME.
Revised: CLIME.

Original: First, we estimate our metric SQ : SQ ( CLIME ) = 0 .
Revised: First, we estimate our metric SQ : SQ ( CLIME ) = 0 .

Original: As it is close to 1, we conclude that CLIME samples from the true distribution.
Revised: As it is close to 1, we conclude that CLIME samples from the true distribution.

Original: Then we generated explanations produced by CLIME for the same sample x
Revised: Then we generated explanations produced by CLIME for the same sample x

Original: = [(7 , 7) , (7 , 7) , (7 , 7)] .
Revised: = [(7 , 7) , (7 , 7) , (7 , 7)] .

Original: The second row (Table 1, ‘Balanced dataset’ column) shows explanations produced by CLIME.
Revised: The second row (Table 1, ‘Balanced dataset’ column) shows explanations produced by CLIME.

Original: Note that CLIME produces valid explanations – all features with non-negligible coefﬁcients have negative contributions (the corresponding horizontal bars are red).
Revised: Note that CLIME produces valid explanations – all features with non-negligible coefﬁcients have negative contributions (the corresponding horizontal bars are red).

Original: The same holds for all samples x
Revised: The same holds for all samples x

Original: = [ p 1 , p 2 , p 3 ] with p i ∈ S 0 that are in near the corner (8,8).
Revised: = [ p 1 , p 2 , p 3 ] with p i ∈ S 0 that are in near the corner (8,8).

Original: We recall that these samples were problematic for LIME to produce correct explanations.
Revised: We recall that these samples were problematic for LIME to produce correct explanations.

Original: Example C.2 (‘Hit and miss’.
Revised: Example C.2 (‘Hit and miss’.

Original: Sensitivity to the training set) .
Revised: Sensitivity to the training set) .

Original: As we described in Example B.1, LIME is sensitive to OOD samples even if we have a near to perfect classiﬁer f .
Revised: As we described in Example B.1, LIME is sensitive to OOD samples even if we have a near to perfect classiﬁer f .

Original: Moreover, we observed that the choice of the dataset can severely impact explanations.
Revised: Moreover, we observed that the choice of the dataset can severely impact explanations.

Original: We created a new dataset for the same classiﬁcation task as in Example B.1{+B.
+} (Figure 3).
Revised: We created a new dataset for the same classiﬁcation task as in Example {+B.
+} (Figure 1).

Original: The only difference is that we have twice more C 0 samples compared to C 1 samples in the training set.
Revised: The only difference is that we have twice more C 0 samples compared to C 1 samples in the training set.

Original: Unbalanced datasets are often occur in practice if samples for one class are much cheaper to obtained compared to the other class.
Revised: Unbalanced datasets are often occur in practice if samples for one class are much cheaper to obtained compared to the other class.

Original: We compute SQ : SQ ( LIME )
Revised: We compute SQ : SQ ( LIME )

Original: = 0 .
Revised: = 0 .

Original: 43 , which is much less than 1 , and SQ ( CLIME )
Revised: 43 , which is much less than 1 , and SQ ( CLIME )

Original: = 0 .
Revised: = 0 .

Original: Table(‘Unbalanced dataset’) shows our results for x in the second column .
Revised: Table(‘Unbalanced dataset’) shows our results for x in the second column .

Original: Based on our metrics, it is expected that CLIME produces better explanations as almost all features contribute to C 0 which is not the case for LIME.
Revised: Based on our metrics, it is expected that CLIME produces better explanations as almost all features contribute to C 0 which is not the case for LIME.

Original: Example C.3 (‘Projected explanations’) .
Revised: Example C.3 (‘Projected explanations’) .

Original: Let us consider a scenario where the user asks for an explanation constrained on a subspace of the original distribution D .
Revised: Let us consider a scenario where the user asks for an explanation constrained on a subspace of the original distribution D .

Original: For example, for the recidivism dataset, a user might want to know how decisions are made for the subgroup of people who are junkie or have problems with alcohol.
Revised: For example, for the recidivism dataset, a user might want to know how decisions are made for the subgroup of people who are junkie or have problems with alcohol.

Original: Here, we craft a synthetic example to show how projecting on such subsets of features can affect explanations.
Revised: Here, we craft a synthetic example to show how projecting on such subsets of features can affect explanations.

Original: Deﬁnition of a task.
Revised: Deﬁnition of a task.

Original: We consider two areas S 0 and S 1 on the 2D plane (see Figure 3(b)).
Revised: We consider two areas S 0 and S 1 on the 2D plane (see Figure 5).

Original: A sample x from D represents ﬁve 2D integer points from S 0 ∪ S 1 .
Revised: A sample x from D represents ﬁve 2D integer points from S 0 ∪ S 1 .

Original: Hence, we have ten features vector x =
Revised: Hence, we have ten features vector x =

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: . , p 5 ] =
Revised: . , p 5 ] =

Original: [( x 1 , x 2 ) , . .
Revised: [( x 1 , x 2 ) , . .

Original: , ( x 9 , x 10 )] , where ( x 2 i − 1 , x 2 i ) ) , i = 1 , . .
Revised: , ( x 9 , x 10 )] , where ( x 2 i − 1 , x 2 i ) ) , i = 1 , . .

Original: , 5 is a 2D point.
Revised: , 5 is a 2D point.

Original: Given a sample x = [ p 1 , . . .
Revised: Given a sample x = [ p 1 , . . .

Original: , p 5 ] , we say that x belongs to C 1 iff there exist i, j ∈ [1 , 5] s.t.
Revised: , p 5 ] , we say that x belongs to C 1 iff there exist i, j ∈ [1 , 5] s.t.

Original: p
Revised: p

Original: i , p j ∈ S 1 and to C 0 otherwise.
Revised: i , p j ∈ S 1 and to C 0 otherwise.

Original: So, if two out of ﬁve points in x are in S 1 then x is classiﬁed as 1.
Revised: So, if two out of ﬁve points in x are in S 1 then x is classiﬁed as 1.

Original: As in Example B. points in the area S 0 (resp.
Revised: As in Example B. points in the area S 0 (resp.

Original: S 1 ) contribute to the C 0 (resp.
Revised: S 1 ) contribute to the C 0 (resp.

Original: C 1 ) class, respectively. 
Revised: C 1 ) class, respectively. S

Revised: S     (4,4)

Original: Projection constraints.
Revised: Projection constraints.

Original: Next, we assume that the user adds constraints that the ﬁrst two points must be in S 0 : p 1 , p 2 ∈ S 0 .
Revised: Next, we assume that the user adds constraints that the ﬁrst two points must be in S 0 : p 1 , p 2 ∈ S 0 .

Original: As we have a constrained distribution, namely, p 0 , p 1 ∈ S 0 , we know that the ﬁrst four features, x 1 , x 2 , x 3 , and x 4 , should not appear in any explanation as no matter how we vary these features values in the constrained space, the corresponding points p 1 , p 2 are always in S 0 .
Revised: As we have a constrained distribution, namely, p 0 , p 1 ∈ S 0 , we know that the ﬁrst four features, x 1 , x 2 , x 3 , and x 4 , should not appear in any explanation as no matter how we vary these features values in the constrained space, the corresponding points p 1 , p 2 are always in S 0 .

Original: Hence, the corresponding coefﬁcients of features x 1 , . .
Revised: Hence, the corresponding coefﬁcients of features x 1 , . .

Original: , x 4 , should be close to zero as changing these features in the constrained space does not have an effect on the prediction.
Revised: , x 4 , should be close to zero as changing these features in the constrained space does not have an effect on the prediction.

Original: Explainers performance.
Revised: Explainers performance.

Original: We train an almost perfect neural network-based classiﬁer for this simple task .
Revised: We train an almost perfect neural network-based classiﬁer for this simple task .

Original: We consider explanations of length six.
Revised: We consider explanations of length six.

Original: We compute our sampling quality metric SQ .
Revised: We compute our sampling quality metric SQ .

Original: We get that SQ ( LIME )
Revised: We get that SQ ( LIME )

Original: = 1 . 38 and SQ ( CLIME )
Revised: = 1 . 38 and SQ ( CLIME )

Original: = 0 .
Revised: = 0 .

Original: Hence, these estimates signal that LIME’s sampling procedure does not sample from the true distribution.
Revised: Hence, these estimates signal that LIME’s sampling procedure does not sample from the true distribution.

Original: Table 1 shows results for LIME and CLIME in the third column (‘Projected explanations’) for another sample x
Revised: Table 1 shows results for LIME and CLIME in the third column (‘Projected explanations’) for another sample x

Original: = [(1 , 1) , (1 , 1) , (12 , 12) , (12 , 12) , (12 , 12)] .
Revised: = [(1 , 1) , (1 , 1) , (12 , 12) , (12 , 12) , (12 , 12)] .

Original: As you can see, x 3 and x 4 occur in the explanation for LIME which is incorrect.
Revised: As you can see, x 3 and x 4 occur in the explanation for LIME which is incorrect.

Original: CLIME produces correct explanations as constrained features x 1 , x 2 , x 3 , and x 4 do not occur in CLIME’s explanations.
Revised: CLIME produces correct explanations as constrained features x 1 , x 2 , x 3 , and x 4 do not occur in CLIME’s explanations.

Original: (See more in Section D.1).
Revised: (See more in Section E.1).

Original: C. AA’ A LGORITHM
Revised: D C ERTIFYING C ONSTRAINT -D RIVEN E XPLANATIONS (A DDITIONAL MATERIALS ) D. AA’ A LGORITHM

Original: For completeness, we present the AA (cid:48) algorithm in full, which is a simple adaptation of AA algorithm by (Dagum et al., 2000) which uses almost-uniform samples instead of perfectly uniform.
Revised: For completeness, we present the AA (cid:48) algorithm in full, which is a simple adaptation of AA algorithm by (Dagum et al., 2000) which uses almost-uniform samples instead of perfectly uniform.

Original: AA (cid:48) , takes as input 3 parameters, ε 1 , ε 2 and δ .
Revised: AA (cid:48) , takes as input 3 parameters, ε 1 , ε 2 and δ .

Original: It uses ε 2 as the tolerance parameter in calls to an almost-uniform sampler (encapsulated in procedure getSamples ).
Revised: It uses ε 2 as the tolerance parameter in calls to an almost-uniform sampler (encapsulated in procedure getSamples ).

Revised: For ease of exposition, we use φ to denote the subspace that getSamples generates samples from.

Revised: We emphasize that the estimation algorithm is not limited to Boolean domains; it is applicable to any population so long as it is possible to sample (almost) uniformly from that population.

Original: As before, we assume access to a procedure checkProperty which returns 1 if the property of interest is satisﬁed by the sample and 0 otherwise.
Revised: As before, we assume access to a procedure checkProperty which returns 1 if the property of interest is satisﬁed by the sample and 0 otherwise.

Original: [Equation]
Revised: [Equation]

Original: The guarantees provided by AA (cid:48) are similar to Eqn.
Revised: The guarantees provided by AA (cid:48) are similar to Eqn.

Original: and are precisely captured in Eqn. 9.
Revised: and are precisely captured in Eqn. 9.

Original: See
Revised: See

Original: Lemmas 1 and 4 for the proof.
Revised: Lemmas 1 and 4 for the proof.

Original: C. P ROOF OF T HEOREMMean deviation due to almost uniform sampling
Revised: D. P ROOF OF T HEOREMMean deviation due to almost uniform sampling

Original: Lemma 1.
Revised: Lemma 1.

Original: Let ρ be the density of instances that satisfy some property P in a universe U Z , that is,
Revised: Let ρ be the density of instances that satisfy some property P in a universe U Z , that is,

Original: [Equation]
Revised: [Equation]

Revised: Output: (cid:98) ρ : Estimate of ρ satisfying Eqn.1: τ ← 4( e − 2) ln(2 /δ ) /ε2:

Revised: τ 2 ← 1 .

Revised: 1 τ 3: ˚ ρ ← stoppingRule (1 / 2 , ε 2 , δ/ 3 , τ ) 4:

Revised: N ← τ 2 · ε 1 / ˚ ρ 5: a ←6: for i ∈ { 1 , . . .

Revised: , N } do 7: s 1 ← getSamples ( φ, ε 2 , 1) 8: c 1 ← checkProperty ( s 1 ) 9: s 2 ← getSamples ( φ, ε 2 , 1) 10: c 2 ← checkProperty ( s 2 ) 11: a ← a + ( c 1 − c 2 ) 2 /12: end for 13: ξ ← max ( a/N, ˚ ρ · ε 1 ) 14: N ← τ 2 · ξ/ ˚ ρ15: a ←16: for i ∈ { 1 , . .

Revised: , N } do 17: s ← getSamples ( φ, ε 2 , 1) 18: c

Revised: ← checkProperty ( s ) 19: a ← a + c 20: end for 21: (cid:98) ρ ← a/N 22: return (cid:98) ρ

Revised: Output: ˚ ρ (weak estimate) 1: τ 1 ← 1 + (1 + ε 1 ) τ 2: N ←3: a ←4: while a < τ 1 do 5: s ← getSamples ( φ, ε 2 , 1) 6: a ← a + checkProperty ( s ) 7: end while 8: ˚

Revised: ρ ← τ 1 /N 9: return ˚ ρ

Original: Suppose we sample each instance z ∈ U Z almost-uniformly, that is
Revised: Suppose we sample each instance z ∈ U Z almost-uniformly, that is

Original: [Equation]
Revised: [Equation]

Original:  Then we have ρ
Revised: | Then we have ρ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: In the worst cases, each sample z s.t.
Revised: In the worst cases, each sample z s.t.

Original: I [ P ( z )]
Revised: I [ P ( z )]

Original: = 1 will be sampled with
Revised: = 1 will be sampled with

Original: probability 1+ ε |U Z |, in which case (cid:80)z ∈U Z
Revised: probability 1+ ε |U Z | in which case (cid:80)

Original:  I [ P ( z )] · Pr[ z ∗ =
Revised: z ∈U Z I [ P ( z )] · Pr[ z ∗ =

Original: z ] = (1 + ε ) · ρ 2.
Revised: z ] = (1 + ε ) · ρ 2.

Original: probability 1 (1+ ε ) 
Revised: probability 1+ ε ) |U

Original: |U Z | in which case (cid:80) z∈U Z
Revised: Z | in which case (cid:80) z

Original:  I [ P ( z )] · Pr[ z ∗=
Revised: ∈U Z I [ P ( [Equation] ∗

Original: z ] = ρ (1+ ε )
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: Let ρ ≤ γ − ε .
Revised: Let ρ ≤ γ − ε .

Original: Then the probability that checkThreshold returns True is at-least 1 − δ .
Revised: Then the probability that checkThreshold returns True is at-least 1 − δ .

Original: Output: (cid:98) ρ : Estimate of ρ satisfying Eqn.1: τ ← 4( e − 2) ln(2 /δ ) /ε2:

Original: τ 2 ← 1 .

Original: 1 τ 3: ˚ ρ ← stoppingRule (1 / 2 , ε 2 , δ/ 3 , τ ) 4:

Original: N ← τ 2 · ε 1 / ˚ ρ 5: a ←6: for i ∈ { 1 , . . .

Original: , N } do 7: s 1 ← getSamples (1 , ε 2 ) 8: c 1 ← checkProperty ( s 1 ) 9: s 2 ← getSamples (1 , ε 2 ) 10: c 2 ← checkProperty ( s 2 ) 11: a ← a + ( c 1 − c 2 ) 2 /12: end for 13: ξ ← max ( a/N, ˚ ρ · ε 1 ) 14: N ← τ 2 · ξ/ ˚ ρ15: a ←16: for i ∈ { 1 , . .

Original: , N } do 17: s ← getSamples (1 , ε 2 ) 18: c

Original: ← checkProperty ( s ) 19: a ← a + c 20: end for 21: (cid:98) ρ ← a/N 22: return (cid:98) ρ

Original: Output: ˚ ρ (weak estimate) 1: τ 1 ← 1 + (1 + ε 1 ) τ 2: N ←3: a ←4: while a < τ 1 do 5: s ← getSamples (1 , ε 2 ) 6: a ← a + checkProperty ( s ) 7: end while 8: ˚

Original: ρ ← τ 1 /N 9: return ˚ ρ

Original: Proof.
Revised: Proof.

Original: Note that µ C ≤ (1 + ε/ 2) ρ as getSamples may return almost uniformly distributed samples in line 3 of checkThreshold .
Revised: Note that µ C ≤ (1 + ε/ 2) ρ as getSamples may return almost uniformly distributed samples in line 3 of checkThreshold .

Original: We will ﬁrst prove that µ C + ν ≤ γ .
Revised: We will ﬁrst prove that µ C + ν ≤ γ .

Original: We haveγ ≤ γ = ⇒ γ [Equation] 1 + ε/
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: But we have γ ≥ ρ + ε .
Revised: But we have γ ≥ ρ + ε .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: The last equation follows from the fact that µ C ≤ (1+ ε/ 2) ρ and ν ≤ ε + ε 2 / 2 − γε/ 2 from line 1 of checkThreshold .
Revised: The last equation follows from the fact that µ C ≤ (1+ ε/ 2) ρ and ν ≤ ε + ε 2 / 2 − γε/ 2 from line 1 of checkThreshold .

Original: Now since µ C + ν ≤ γ , we have Pr[ C ≤ γ ] ≥ Pr[ C ≤ µ C
Revised: Now since µ C + ν ≤ γ , we have Pr[ C ≤ γ ] ≥ Pr[ C ≤ µ C

Original: + ν ] = 1 − Pr
Revised: + ν ] = 1 − Pr

Original: [ C ≥ µ C + ν ] .
Revised: [ C ≥ µ C + ν ] .

Original: C is the average of N independent 0 / 1 random variables
Revised: C is the average of N independent 0 / 1 random variables

Original: c (line 6).
Revised: c (line 6).

Original: Therefore applying
Revised: Therefore applying

Original: Chernoff bound,
Revised: Chernoff bound,

Original: [Equation]
Revised: [Equation]

Original: But N = 12 ν 2 log( 1 δ ) .
Revised: But N = 12 ν 2 log( 1 δ ) .

Original: Therefore, Pr[ C ≥ µ C + ν ] ≤ δ .
Revised: Therefore, Pr[ C ≥ µ C + ν ] ≤ δ .

Original: Substituting back, we get Pr[ C ≤ γ ] ≥ 1 − δ .
Revised: Substituting back, we get Pr[ C ≤ γ ] ≥ 1 − δ .

Original: Therefore, in line 8, the probability that checkThreshold returns True, is at least 1 − δ .
Revised: Therefore, in line 8, the probability that checkThreshold returns True, is at least 1 − δ .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let ρ ≥ γ + ε .
Revised: Let ρ ≥ γ + ε .

Original: Then the probability that checkThreshold returns False is at-least 1 − δ .
Revised: Then the probability that checkThreshold returns False is at-least 1 − δ .

Original: Proof.
Revised: Proof.

Original: Similar to preceding Lemma.
Revised: Similar to preceding Lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: Algorithm AA (cid:48) returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ψ ≤ (cid:98) ρ ψ ≤ (1+ ε ) ρ ψ ] ≥ (1 − δ )
Revised: Algorithm AA (cid:48) returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ψ ≤ (cid:98) ρ ψ ≤ (1+ ε ) ρ ψ ] ≥ (1 − δ )

Original: Proof.
Revised: Proof.

Original: Algorithm AA (cid:48) takes as input 3 parameters, ε 1 , ε 2 and δ .
Revised: Algorithm AA (cid:48) takes as input 3 parameters, ε 1 , ε 2 and δ .

Original: It invokes the AA Algorithm by Dagum et al.
Revised: It invokes the AA Algorithm by Dagum et al.

Original: with parameters ε 

Revised: with parameters ε

Original: and δ on samples generated by an almost-uniform sampler with parameter
Revised: and δ on samples generated by an almost-uniform sampler with parameter

Original: ε 2 .
Revised: ε 2 .

Original: By Lemma 1, the population mean can shift at most by a factor of (1 + ε 2 ) due to almost-uniform sampling (instead of perfectly uniform).
Revised: By Lemma 1, the population mean can shift at most by a factor of (1 + ε 2 ) due to almost-uniform sampling (instead of perfectly uniform).

Original: Combined with the approximation guarantees of AA algorithm, the resulting tolerance has an upper-bound of (1 + ε 1 ) ×
Revised: Combined with the approximation guarantees of AA algorithm, the resulting tolerance has an upper-bound of (1 + ε 1 ) ×

Original: (1 + ε 2 ) and a lower bound of (1 − ε 1 ) /
Revised: (1 + ε 2 ) and a lower bound of (1 − ε 1 ) /

Original: (1 + ε 2 ) .
Revised: (1 + ε 2 ) .

Original: In line 4 of Alg. 2, the AA (cid:48) algorithm is invoked with ε 1 =
Revised: In line 4 of Alg. 2, the AA (cid:48) algorithm is invoked with ε 1 =

Original: ε 2 = 0 .
Revised: ε 2 = 0 .

Original: 4 ∗ ε .
Revised: 4 ∗ ε .

Original: Substituting these values in the expressions for the upper and lower bounds on tolerance, we get the result.
Revised: Substituting these values in the expressions for the upper and lower bounds on tolerance, we get the result.

Original: Theorem 1.
Revised: Theorem 1.

Original: If ρ ≤ γ − ε , then computeEstimate returns ⊥ with high probability (i.e. at least 1 − δ ).
Revised: If ρ ≤ γ − ε , then computeEstimate returns ⊥ with high probability (i.e. at least 1 − δ ).

Original: If ρ ≥ γ + ε , w.h.p.
Revised: If ρ ≥ γ + ε , w.h.p.

Original: , it returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ≤ (cid:98) ρ ≤ (1 + ε ) ρ ] ≥ (1 − δ ) .
Revised: , it returns an estimate (cid:98) ρ such that Pr[(1 − ε ) ρ ≤ (cid:98) ρ ≤ (1 + ε ) ρ ] ≥ (1 − δ ) .

Original: Proof.
Revised: Proof.

Original: Follows from preceding lemmas.
Revised: Follows from preceding lemmas.

Original: Note that the bounds and number of samples used for proving the preceding theorem were computed assuming we only have access to Almost-Uniform samples.
Revised: Note that the bounds and number of samples used for proving the preceding theorem were computed assuming we only have access to Almost-Uniform samples.

Original: The bounds can be made signiﬁcantly tighter or the number of samples can be reduced, if we have access to perfectly uniform samples.
Revised: The bounds can be made signiﬁcantly tighter or the number of samples can be reduced, if we have access to perfectly uniform samples.

Original: C. A PPLICABILITY OF THE E STIMATION FRAMEWORK
Revised: D. A PPLICABILITY OF THE E STIMATION FRAMEWORK

Original: We emphasize that our estimation framework is general enough to compute any metric for any universe (so long as one can sample from it almost uniformly) according to the guarantees provided by Thm. 1 in any setting (not just explainability).
Revised: We emphasize that our estimation framework is general enough to compute any metric for any universe (so long as one can sample from it almost uniformly) according to the guarantees provided by Thm. 1 in any setting (not just explainability).

Original: Further, for the speciﬁc application explainability, our estimation framework can be used for measuring properties like ﬁdelity of any explainer model (not just the ones crafted by CLIME), and on any subspace of inputs (not just the one that the explainer was trained on).
Revised: Further, for the speciﬁc application explainability, our estimation framework can be used for measuring properties like ﬁdelity of any explainer model (not just the ones crafted by CLIME), and on any subspace of inputs (not just the one that the explainer was trained on).

Original: For example, the ﬁdelity of a CLIME explainer model trained on a subspace deﬁned by one set of constraints (say φ ) maybe evaluated on another subspace deﬁned by ψ .
Revised: For example, the ﬁdelity of a CLIME explainer model trained on a subspace deﬁned by one set of constraints (say φ ) maybe evaluated on another subspace deﬁned by ψ .

Original: If the ﬁdelity on ψ is found to be high enough, it can save the cost of having to generate a separate explanation for ψ .
Revised: If the ﬁdelity on ψ is found to be high enough, it can save the cost of having to generate a separate explanation for ψ .

Original: This can be especially useful in model debugging where users may reﬁne constraints frequently.
Revised: This can be especially useful in model debugging where users may reﬁne constraints frequently.

Original: D E XPERIMENTS (A DDITIONAL EXPERIMENTS )
Revised: D. E MPIRICAL E VALUATION OF E STIMATION A LGORITHM

Revised: In order to test the scalability of our estimation algorithm (Algs. 2, we evaluated its performance on the same set of benchmarks used by (Narodytska et al., 2019). Benchmarks The benchmarks are CNF formulas that encode ﬁdelity of Anchor (Ribeiro et al., 2018) explanations for Binarized Neural Networks with upto 3 hidden layers and 100 neurons in total, and are generated from Adult, Recidivism and Lending datasets. There were 50 CNF formulas from each dataset, for a total of 150 benchmarks, with number of variables ranging between 20, to 80,000 and number of clauses ranging between 80,000 and 290,000.

Revised: The projected model count of each formula represents the number of inputs on which the class-label for Anchor’s explanation matches the true label of the instance being explained.

Revised: The ﬁdelity of an explanation can thus be computed as the ratio of the solution-count of the formula, to the size of the universe.

Revised: Parameters We used the same tolerance ( ε = 0 . 8 ) and conﬁdence ( δ = 0 . 2 ) used in (Narodytska et al., 2019), for the main experiment.

Revised: Additionally, we set the threshold to γ = 0 .

Revised: We also compared the running times for tighter tolerance and conﬁdence (see Discussion below).

Revised: Experimental Setup We set a time out of 3 minutes (180 seconds), and ran each experiment on

Revised: Intel Xeon E5-2650 CPU running at 2.20GHz, with 4GB main memory.

Revised: We compiled our code using

Revised: GCC 6.4 with O3 ﬂag enabled.

Revised: For ApproxMC we used the latest publicly available version (4.01).

Revised: Results The results are presented in Fig.

Revised: Each point in blue corresponds to one benchmark, and the x-coordinate represents the time taken by ApproxMC while the y-coordinate represents the time taken by our approach.

Revised: It can be seen that our approach completes all benchmarks in underseconds with majority taking less than 10 seconds.

Revised: In contrast, ApproxMC is able to ﬁnish onlybenchmarks out of 150 in under 25 seconds, with a majority taking around 75 seconds.

Revised: The average (geometric mean) speedup factor offered by our tool relative to ApproxMC was 7.5.

Revised: Discussion Fig.

Revised: 2 conclusively demonstrates the efﬁciency of our approach as compared to hashing and SAT based approaches like ApproxMC.

Revised: Our tool was able to return estimates with two-sided bounds for all benchmarks.

Revised: Our tool offers an average (geometric mean) speedup by a factor of 7.

Revised: relative to ApproxMC.

Revised: In addition, our approach is able to scale far better than ApproxMC for tighter tolerance and conﬁdence parameters.

Revised: For a representative benchmark, we evaluated the time taken by our tool and ApproxMC for the default tolerance ( ε = 0 . 8 ) and conﬁdence ( δ = 0 . 2 ).

Revised: We then computed the slow-down in both tools after setting ε = 0 . 05 and conﬁdence δ = 0 .

Revised: We found that our tool slowed down by a factor of 10 while ApproxMC slowed down by a factor of 30.

Revised: Lastly, we computed the error in the estimate returned by our algorithm (with default tolerance and conﬁdence) on small benchmarks where it was possible to compute the true count.

Revised: We consistently observed that the error was less than 0.1, which is much smaller than the speciﬁed tolerance of 0.8.

Revised: Thus our approach is both sound and efﬁcient in practice.

Revised: E E XPERIMENTS (A DDITIONAL EXPERIMENTS )

Original: We perform an extensive evaluation of our explainer CLIME.
Revised: We perform an extensive evaluation of our explainer CLIME.

Original: First, we consider explanations for a classiﬁcation task using synthetic and real-world datasets.
Revised: First, we consider explanations for a classiﬁcation task using synthetic and real-world datasets.

Original: Second, we demonstrate how our framework can be used to detect adversarial attacks on an explainer.
Revised: Second, we demonstrate how our framework can be used to detect adversarial attacks on an explainer.

Original: D. S YNTHETIC EXPERIMENTS
Revised: E. S YNTHETIC EXPERIMENTS

Original: For each synthetic dataset, we generated train and test datasets with 1000 samples per class.
Revised: For each synthetic dataset, we generated train and test datasets with 1000 samples per class.

Original: We trained a MLP with three internal layers of ﬁfty neurons that gives 99.9% on the test set.
Revised: We trained a MLP with three internal layers of ﬁfty neurons that gives 99.9% on the test set.

Original: So, we have an almost perfect classiﬁer.
Revised: So, we have an almost perfect classiﬁer.

Original: D.1. ’H IT OR MISS ’ (E XAMPLE B.1)
Revised: E.1. ’H IT OR MISS ’ (E XAMPLE B.1)

Original: We consider a binary classiﬁcation task from Example B.1.
Revised: We consider a binary classiﬁcation task from Example B.1.

Original: We recall that a sample x , x ∼ D , represents integer coordinates of three points p 1 , p 2 and p 3 drawn from S 0 ∪ S 1 .
Revised: We recall that a sample x , x ∼ D , represents integer coordinates of three points p 1 , p 2 and p 3 drawn from S 0 ∪ S 1 .

Original: Hence, we have six features vector x =
Revised: Hence, we have six features vector x =

Original: [ x 1 , x 2 , . . .
Revised: [ x 1 , x 2 , . . .

Original: , x 6 ] , where p i =
Revised: , x 6 ] , where p i =

Original: ( x 2 i − 1 , x 2 i ) , i = 1 , 2 , 3 , is a 2D point.
Revised: ( x 2 i − 1 , x 2 i ) , i = 1 , 2 , 3 , is a 2D point.

Original: Computation of SQ .
Revised: Computation of SQ .

Original: As we know the true distribution, we can compute W D .
Revised: As we know the true distribution, we can compute W D .

Original: Let hit ( x ) =
Revised: Let hit ( x ) =

Original: |{ p i | p i ∈ S 1 , p i ∈ x }| be a function that computes the number of points p i
Revised: |{ p i | p i ∈ S 1 , p i ∈ x }| be a function that computes the number of points p i

Original: ∈
Revised: ∈

Original: x
Revised: x

Original: ∩ S 1 .
Revised: ∩ S 1 .

Original: For example, hit ([(1 , 1) , (10 , 10) , (10 , 9)])
Revised: For example, hit ([(1 , 1) , (10 , 10) , (10 , 9)])

Original: = 2 as (10 , 10) ∈ S 1 and (10 , 9) ∈ S 1 .
Revised: = 2 as (10 , 10) ∈ S 1 and (10 , 9) ∈ S 1 .

Original: We have that
Revised: We have that

Original: P ( p i ∈ S 1 ) =
Revised: P ( p i ∈ S 1 ) =

Original: | S 1 | | S 0 | + | S 1 | = 25 / (25 + 49) = 0 .
Revised: | S 1 | | S 0 | + | S 1 | = 25 / (25 + 49) = 0 .

Original: Hence,
Revised: Hence,

Original: W D = 3 P ( hit ( x ) = 1) + 3 P ( hit ( x ) = 2) + P ( hit ( x ) = 3) = 3 × 0 .
Revised: W D = 3 P ( hit ( x ) = 1) + 3 P ( hit ( x ) = 2) + P ( hit ( x ) = 3) = 3 × 0 .

Original: 15 + 3 × 0 .
Revised: 15 + 3 × 0 .

Original: 07 + 0 .
Revised: 07 + 0 .

Original: 04 = 0 .
Revised: 04 = 0 .

Original: We generate 100 samples such that x i ∈ [5 , 7] , i = [1 , 6] .
Revised: We generate 100 samples such that x i ∈ [5 , 7] , i = [1 , 6] .

Original: Therefore, each sample consists of points that are close to the corner (8,8) and it is classiﬁed as C 0 .
Revised: Therefore, each sample consists of points that are close to the corner (8,8) and it is classiﬁed as C 0 .

Original: We compute explanations by
Revised: We compute explanations by

Original: LIME and CLIME on these samples.
Revised: LIME and CLIME on these samples.

Original: We empirically compute W LIME Z (cid:48) and W CLIME Z (cid:48) for the neighborhood Z (cid:48) .
Revised: We empirically compute W LIME Z (cid:48) and W CLIME Z (cid:48) for the neighborhood Z (cid:48) .

Original: We found that W LIME Z (cid:48)
Revised: We found that W LIME Z (cid:48)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: 53 and W CLIME Z (cid:48)
Revised: 53 and W CLIME Z (cid:48)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: So, SQ ( LIME )
Revised: So, SQ ( LIME )

Original: ≈
Revised: ≈

Original: 75 < 1 and SQ ( CLIME ) ≈ 1 .
Revised: 75 < 1 and SQ ( CLIME ) ≈ 1 .

Original: These results hint that LIME may produce invalid explanations.
Revised: These results hint that LIME may produce invalid explanations.

Original: We experimentally verify this hypothesis next.
Revised: We experimentally verify this hypothesis next.

Original: Explainers performance.
Revised: Explainers performance.

Original: We recall that each sample consist of points that are close to the corner (8,8) and it is classiﬁed as C 0 .
Revised: We recall that each sample consist of points that are close to the corner (8,8) and it is classiﬁed as C 0 .

Original: Hence, we expect that coefﬁcients in the explanations are less than 0 as all features contribute to C 0 .
Revised: Hence, we expect that coefﬁcients in the explanations are less than 0 as all features contribute to C 0 .

Original: To evaluate this expected behavior, we computed
Revised: To evaluate this expected behavior, we computed

Original: [Equation]
Revised: [Equation]

Original: where c i is the i th normalised (between 0 and 1) coefﬁcient in an explanation (Eqn. 2).
Revised: where c i is the i th normalised (between 0 and 1) coefﬁcient in an explanation (Eqn. 2).

Original: We average results over 100 samples.
Revised: We average results over 100 samples.

Original: Ideally, C + has to be equal to zero, as we should not have positive coefﬁcients in an explanation in any of these samples.
Revised: Ideally, C + has to be equal to zero, as we should not have positive coefﬁcients in an explanation in any of these samples.

Original: We get that C + LIME = 0 .
Revised: We get that C + LIME = 0 .

Original: 18 and C + CLIME
Revised: 18 and C + CLIME

Original: = 0 .
Revised: = 0 .

Original: As you can see, C + LIME is signiﬁcantly greater than zero showing that many explanations are incorrect.
Revised: As you can see, C + LIME is signiﬁcantly greater than zero showing that many explanations are incorrect.

Original: Therefore, explanations produced by LIME suffer from ODD sampling and result in incorrect explanations.
Revised: Therefore, explanations produced by LIME suffer from ODD sampling and result in incorrect explanations.

Original: D.1. P ROJECTED EXPLANATIONS (E XAMPLE C.3)
Revised: E.1. P ROJECTED EXPLANATIONS (E XAMPLE C.3)

Original: We consider a binary classiﬁcation task from Example C.3.
Revised: We consider a binary classiﬁcation task from Example C.3.

Original: We recall that a sample x from D represents ﬁve 2D integer points from S 0 ∪ S 1 .
Revised: We recall that a sample x from D represents ﬁve 2D integer points from S 0 ∪ S 1 .

Original: Hence, we have ten features vector x =
Revised: Hence, we have ten features vector x =

Original: [ p 1 , . .
Revised: [ p 1 , . .

Original: . , p 5 ] =
Revised: . , p 5 ] =

Original: [( x 1 , x 2 ) , . .
Revised: [( x 1 , x 2 ) , . .

Original: , ( x 9 , x 10 )] , where p i =
Revised: , ( x 9 , x 10 )] , where p i =

Original: ( x 2 i − 1 , x 2 i ) ) , i = 1 , . .
Revised: ( x 2 i − 1 , x 2 i ) ) , i = 1 , . .

Original: , 5 is a 2D point.
Revised: , 5 is a 2D point.

Original: We recall that the user constraint is that p 1 , p 2 ∈ S 0 .
Revised: We recall that the user constraint is that p 1 , p 2 ∈ S 0 .

Original: Computation of SQ .
Revised: Computation of SQ .

Original: We deﬁne hit ( x ) the same way as above.
Revised: We deﬁne hit ( x ) the same way as above.

Original: We have that P ( p i ∈ S 1 ) =
Revised: We have that P ( p i ∈ S 1 ) =

Original: | S 1 | | S 0 | + | S 1 | = 64 / 144 = 0 .
Revised: | S 1 | | S 0 | + | S 1 | = 64 / 144 = 0 .

Original: 44 for this task.
Revised: 44 for this task.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: 3 × P ( hit ( x ) = 2) + P ( hit ( x ) = 3) = 0 .
Revised: 3 × P ( hit ( x ) = 2) + P ( hit ( x ) = 3) = 0 .

Original: 33 + 0 .
Revised: 33 + 0 .

Original: = 0 .
Revised: = 0 .

Original: We compute explanations by LIME and CLIME on 100 samples that satisfy the user constraint.
Revised: We compute explanations by LIME and CLIME on 100 samples that satisfy the user constraint.

Original: We ask explainers to give us the top six features per sample.
Revised: We ask explainers to give us the top six features per sample.

Original: Moreover, to help LIME to capture constraints, we provide it with a dataset where p 1 , p 2 ∈ S 0 for all samples.
Revised: Moreover, to help LIME to capture constraints, we provide it with a dataset where p 1 , p 2 ∈ S 0 for all samples.

Original: Then we run LIME and CLIME sampling procedures and compute W LIME Z (cid:48) and W CLIME Z (cid:48) for the neighborhood Z (cid:48) .
Revised: Then we run LIME and CLIME sampling procedures and compute W LIME Z (cid:48) and W CLIME Z (cid:48) for the neighborhood Z (cid:48) .

Original: We found that
Revised: We found that

Original: W LIME Z (cid:48)
Revised: W LIME Z (cid:48)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: 56 and W CLIME Z (cid:48)
Revised: 56 and W CLIME Z (cid:48)

Original: ≈ 0 .
Revised: ≈ 0 .

Original: So, SQ ( LIME )
Revised: So, SQ ( LIME )

Original: ≈ 1 . 38 > 1 and SQ ( CLIME )
Revised: ≈ 1 . 38 > 1 and SQ ( CLIME )

Original: ≈ 1 .
Revised: ≈ 1 .

Original: Again, we can see that SQ ( LIME ) is signiﬁcantly differ from 1 indicating ODD sampling.
Revised: Again, we can see that SQ ( LIME ) is signiﬁcantly differ from 1 indicating ODD sampling.

Original: Explainers performance.
Revised: Explainers performance.

Original: We run LIME and CLIME on 100 samples and compute the number of times each feature occurs in an explanation (an explanation contains 6 features).
Revised: We run LIME and CLIME on 100 samples and compute the number of times each feature occurs in an explanation (an explanation contains 6 features).

Original: Figure 2 shows our results.
Revised: Figure 6 shows our results.

Original: For each feature, we show the number of occurrences of this feature in produced explanations.
Revised: For each feature, we show the number of occurrences of this feature in produced explanations.

Original: Namely, the blue histogram shows results for LIME and the orange histogram shows results for
Revised: Namely, the blue histogram shows results for LIME and the orange histogram shows results for

Original: CLIME.
Revised: CLIME.

Original: As we discussed before, as p 1 and p 2 are in S 0 , so the ﬁrst 4 features should not appear in the explanation as changing their values never affects the prediction.
Revised: As we discussed before, as p 1 and p 2 are in S 0 , so the ﬁrst 4 features should not appear in the explanation as changing their values never affects the prediction.

Original: Indeed, CLIME almost never choose features x 1 , . . .
Revised: Indeed, CLIME almost never choose features x 1 , . . .

Original: , x 4 in the explanation (almost zero height orange bars for these features).
Revised: , x 4 in the explanation (almost zero height orange bars for these features).

Original: However, LIME often picks these constrained features in its explanations.
Revised: However, LIME often picks these constrained features in its explanations.

Original: D. A DULT DATASET
Revised: E. A DULT DATASET

Original: The Adult dataset (Kohavi, 1996) is originally taken from the Census bureau and targets predicting whether or not a given adult person earns more than $50K a year depending on various attributes, e.g. race, sex, education, hours of work, etc.
Revised: The Adult dataset (Kohavi, 1996) is originally taken from the Census bureau and targets predicting whether or not a given adult person earns more than $50K a year depending on various attributes, e.g. race, sex, education, hours of work, etc.

Original: We pre-processed columns with continuous features, e.g. the pre-processor discretizes the capital gain and capital loss features into categorical features, e.g.
Revised: We pre-processed columns with continuous features, e.g. the pre-processor discretizes the capital gain and capital loss features into categorical features, e.g.

Original: ’Unknown’, ’Low’ and ’High’(Ribeiro et al., 2018).
Revised: ’Unknown’, ’Low’ and ’High’(Ribeiro et al., 2018).

Original: We trained a MLP with three internal layers of 64, 32, and 32 neurons that achieves 84.2% on the test set.
Revised: We trained a MLP with three internal layers of 64, 32, and 32 neurons that achieves 84.2% on the test set.

Original: We consider the two constrained spaces:
Revised: We consider the two constrained spaces:

Original: • ‘female space’ :
Revised: • ‘female space’ :

Original: = (C APITAL L OSS = ‘U NKNOWN ’) AND (C APITAL G AIN = ‘U NKNOWN ’) AND (E DUCATION = 10 TH OR 11 TH GRADE ) AND (S EX = F EMALE ) • ‘male space’ :
Revised: = (C APITAL L OSS = ‘U NKNOWN ’) AND (C APITAL G AIN = ‘U NKNOWN ’) AND (E DUCATION = 10 TH OR 11 TH GRADE ) AND (S EX = F EMALE ) • ‘male space’ :

Original: = (C APITAL L OSS = ‘U NKNOWN ’) AND (C APITAL G AIN = ‘U NKNOWN ’) AND (E DUCATION = 10 TH OR 11 TH GRADE ) AND (S EX = M ALE ).
Revised: = (C APITAL L OSS = ‘U NKNOWN ’) AND (C APITAL G AIN = ‘U NKNOWN ’) AND (E DUCATION = 10 TH OR 11 TH GRADE ) AND (S EX = M ALE ).

Original: We compute explanations for 100 samples within the corresponding constrained space.
Revised: We compute explanations for 100 samples within the corresponding constrained space.

Original: Figureshows results for LIME (a) and CLIME (b), respectively, for both constrained spaces.
Revised: Figureshows results for LIME (a) and CLIME (b), respectively, for both constrained spaces.

Original: At each plot, the blue histogram shows results for ‘female space’ and the orange histogram shows results for ‘male space’.
Revised: At each plot, the blue histogram shows results for ‘female space’ and the orange histogram shows results for ‘male space’.

Original: Each bar shows the average rank of a feature in explanations, e.g. if the height of the bar is 0.1 than the corresponding feature is mostly top ranked in all explanations.
Revised: Each bar shows the average rank of a feature in explanations, e.g. if the height of the bar is 0.1 than the corresponding feature is mostly top ranked in all explanations.

Revised: We used the following shortcuts of features names in Figure 7 : ‘Age’ (‘Age’), ‘Workclass’ (‘WrkC’), ‘Education’ (‘Edu’), ‘Marital Status’ (‘Mard’), ‘Occupation’ (‘Occp’), ‘Relationship’ (‘Relp’), ‘Race’ (‘Race’), ‘Sex’ (‘Sex’), ‘Capital Gain’ (‘CapG’), ‘Capital Loss’ (‘CapL’), ‘Hours/week’ (‘Hrs’), and ‘Country’ (‘Cntr’).

Original: First, we observe that as features Capital Loss, Capital Gain and Sex are ﬁxed in both constrained sub-spaces so their values should not inﬂuence the outcome.
Revised: First, we observe that as features Capital Loss, Capital Gain and Sex are ﬁxed in both constrained sub-spaces so their values should not inﬂuence the outcome.

Original: Note that these features are always ranked among last by CLIME.
Revised: Note that these features are always ranked among last by CLIME.

Original: However, this is not the case for LIME.
Revised: However, this is not the case for LIME.

Original: We observed exactly the same behavior in our synthetic example from Example C.3.
Revised: We observed exactly the same behavior in our synthetic example from Example C.3.

Original: Second, we can see that LIME explanations are not so much different for both spaces.
Revised: Second, we can see that LIME explanations are not so much different for both spaces.

Original: While we do not have the ground truth to check against in this example, we can contrast LIME results with CLIME results.
Revised: While we do not have the ground truth to check against in this example, we can contrast LIME results with CLIME results.

Original: CLIME shows that ‘Education’ is more important for males than for females (see in Figure 4(b), the orange bar is lower than the blue bar for the ‘Education’ feature).
Revised: CLIME shows that ‘Education’ is more important for males than for females (see in Figure 7(b), the orange bar is lower than the blue bar for the ‘Education’ feature).

Original: ‘Relationship’ seems to be more important for females than for males (see in Figure 4(b), the orange bar is higher than the blue bar for the ‘Relationship’ feature).
Revised: ‘Relationship’ seems to be more important for females than for males (see in Figure 7(b), the orange bar is higher than the blue bar for the ‘Relationship’ feature).

Original: These results indicate that there might be a potential bias in these spaces.
Revised: These results indicate that there might be a potential bias in these spaces.

Original: D. D ETECTING A DVERSARIALA TTACKS (E XTENDED )
Revised: E. D ETECTING A DVERSARIAL

Original: Suppose a biased classiﬁer makes decisions solely based on some sensitive feature; for ex: a classiﬁer for accepting / rejecting loan applications that decides purely based on an individual’s race.
Revised: Suppose a biased classiﬁer makes decisions solely based on some sensitive feature; for ex: a classiﬁer for accepting / rejecting loan applications that decides purely based on an individual’s race.

Original: LIME’s explanations for the predictions made by such a classiﬁer would clearly indicate race to be a major deciding factor.
Revised: LIME’s explanations for the predictions made by such a classiﬁer would clearly indicate race to be a major deciding factor.

Original: (Slack et al., 2020) devised a technique that takes such an arbitrary biased classiﬁer and constructs another classiﬁer that closely mimics the biased classiﬁer on in-distribution data, yet is able to evade bias-detection by preventing the sensitive feature from appearing in post-hoc explanations.
Revised: (Slack et al., 2020) devised a technique that takes such an arbitrary biased classiﬁer and constructs another classiﬁer that closely mimics the biased classiﬁer on in-distribution data, yet is able to evade bias-detection by preventing the sensitive feature from appearing in post-hoc explanations.

Original: Note that accurately measuring the ﬁdelity of the explainer model alone is not sufﬁcient to detect such an attack as the problem lies in the way the opaque classiﬁer is constructed and not in the resulting explanation.
Revised: Note that accurately measuring the ﬁdelity of the explainer model alone is not sufﬁcient to detect such an attack as the problem lies in the way the opaque classiﬁer is constructed and not in the resulting explanation.

Original: We ﬁrst discuss the construction of the adversarial classiﬁer, and then see how constraints can help in foiling it.
Revised: We ﬁrst discuss the construction of the adversarial classiﬁer, and then see how constraints can help in foiling it.

Original: D.3. B ACKGROUND
Revised: E.3. B ACKGROUND

Original: The attack makes use of the fact that LIME’s perturbation procedure creates samples that are distinguishable from those of the original distribution.
Revised: The attack makes use of the fact that LIME’s perturbation procedure creates samples that are distinguishable from those of the original distribution.

Original: The ﬁrst step is to train a classiﬁer such as a
Revised: The ﬁrst step is to train a classiﬁer such as a

Original: Random Forest to detect whether a given sample is in or out of distribution.
Revised: Random Forest to detect whether a given sample is in or out of distribution.

Original: This classiﬁer is trained on a synthetic data-set consisting of instances from the original data-set labeled as in-distribution and LIMEs perturbations of each of those instances as OOD.
Revised: This classiﬁer is trained on a synthetic data-set consisting of instances from the original data-set labeled as in-distribution and LIMEs perturbations of each of those instances as OOD.

Original: This OOD detecting classiﬁer is able to achieve very good accuracy, which is a testament to the severity of the OOD problem of LIME. 
Revised: This OOD detecting classiﬁer is able to achieve very good accuracy, which is a testament to the severity of the OOD problem of LIME. [Figure]

Original: The next step in the adversarial attack is to create an innocuous classiﬁer that makes decisions based on some unrelated feature.
Revised: The next step in the adversarial attack is to create an innocuous classiﬁer that makes decisions based on some unrelated feature.

Original: For example, in case of credit card applications, the innocuous classiﬁer could make decisions based on either an existing unimportant feature such as card number or on a[Figure] new feature added to the dataset whose values are randomly sampled.
Revised: For example, in case of credit card applications, the innocuous classiﬁer could make decisions based on either an existing unimportant feature such as card number or on a new feature added to the dataset whose values are randomly sampled.

Original: This unrelated feature will be made to appear in place of the sensitive feature in explanations.
Revised: This unrelated feature will be made to appear in place of the sensitive feature in explanations.

Original: The last step of the attack is construct the adversarial classiﬁer which ﬁrst applies the OOD-detecting classiﬁer on a given instance x .
Revised: The last step of the attack is construct the adversarial classiﬁer which ﬁrst applies the OOD-detecting classiﬁer on a given instance x .

Original: If the OOD-detecting classiﬁer predicts x to be in-distribution, the the adversarial classiﬁer outputs the prediction of the biased classiﬁer on x .
Revised: If the OOD-detecting classiﬁer predicts x to be in-distribution, the the adversarial classiﬁer outputs the prediction of the biased classiﬁer on x .

Original: If x is predicted to be
Revised: If x is predicted to be

Original: OOD, then the adversarial classiﬁer outputs the prediction of the innocuous classiﬁer on x .
Revised: OOD, then the adversarial classiﬁer outputs the prediction of the innocuous classiﬁer on x .

Original: Clearly, the predictions made by the adversarial classiﬁer on in-distribution instances are biased, as the adversarial classiﬁer simply outputs the biased classiﬁers prediction.
Revised: Clearly, the predictions made by the adversarial classiﬁer on in-distribution instances are biased, as the adversarial classiﬁer simply outputs the biased classiﬁers prediction.

Original: Suppose LIME is used for explaining the prediction made by the adversarial classiﬁer on some in-distribution instance x .
Revised: Suppose LIME is used for explaining the prediction made by the adversarial classiﬁer on some in-distribution instance x .

Original: LIME ﬁrst perturbs x .
Revised: LIME ﬁrst perturbs x .

Original: However, due to LIME’s
Revised: However, due to LIME’s

Original: OOD sampling problem
Revised: OOD sampling problem

Original: , most of the perturbed samples are OOD and are detected by the OOD-detecting classiﬁer as such.
Revised: , most of the perturbed samples are OOD and are detected by the OOD-detecting classiﬁer as such.

Original: Therefore, the output of the adversarial model on the perturbed samples is that of the innocuous classiﬁer.
Revised: Therefore, the output of the adversarial model on the perturbed samples is that of the innocuous classiﬁer.

Original: Thus for vast majority of the samples on which LIME’s explainer model is trained, the unrelated feature appears as the deciding factor for the adversarial model’s prediction, whereas the sensitive feature is successfully hidden.
Revised: Thus for vast majority of the samples on which LIME’s explainer model is trained, the unrelated feature appears as the deciding factor for the adversarial model’s prediction, whereas the sensitive feature is successfully hidden.

Original: Experimentally, the authors generated explanations for the adversarial model on multiple test instances belonging to Recidivism, Communities and Crime and German datasets.
Revised: Experimentally, the authors generated explanations for the adversarial model on multiple test instances belonging to Recidivism, Communities and Crime and German datasets.

Original: They showed that the sensitive features appeared with very low frequency among the top 3 explanations for all the test points, which indicates that the attack was successful.
Revised: They showed that the sensitive features appeared with very low frequency among the top 3 explanations for all the test points, which indicates that the attack was successful.

Original: D.3. E XPERIMENTAL S ETUP
Revised: E.3. E XPERIMENTAL S ETUP

Original: We hypothesized that for discrete data, instances with low Hamming distance to points in the given dataset should tend to be in-distribution, whereas instances with a high hamming distance should tend to be OOD.
Revised: We hypothesized that for discrete data, instances with low Hamming distance to points in the given dataset should tend to be in-distribution, whereas instances with a high hamming distance should tend to be OOD.

Original: Note that the recommended way for using LIME is on data which is discretized.
Revised: Note that the recommended way for using LIME is on data which is discretized.

Original: LIME itself uses Hamming distance as a distance metric for weighting the samples it generates.
Revised: LIME itself uses Hamming distance as a distance metric for weighting the samples it generates.

Original: However, it is unable to speciﬁcally sample proportionately from high-weight regions which leads to a lot of samples with low weights.
Revised: However, it is unable to speciﬁcally sample proportionately from high-weight regions which leads to a lot of samples with low weights.

Original: With CLIME one can, however, use constraints to generate samples that would be weighted highly by LIME.
Revised: With CLIME one can, however, use constraints to generate samples that would be weighted highly by LIME.

Original: For a given Hamming distance h and a set of instances S , a formula φ h can be constructed such that the the solutions of φ h are exactly those instances that are at most at a distance h from each instance in S .
Revised: For a given Hamming distance h and a set of instances S , a formula φ h can be constructed such that the the solutions of φ h are exactly those instances that are at most at a distance h from each instance in S .

Original: If our hypothesis is correct, the sensitive features should appear as the top CLIME explanation of the adversarial classiﬁer for low values of h , while innocuous features should dominate for larger h .
Revised: If our hypothesis is correct, the sensitive features should appear as the top CLIME explanation of the adversarial classiﬁer for low values of h , while innocuous features should dominate for larger h .

Original: Note that the Hamming distance between two instances doesn’t take into account the degree to which the value of a feature in one instance differs from the other.
Revised: Note that the Hamming distance between two instances doesn’t take into account the degree to which the value of a feature in one instance differs from the other.

Original: Speciﬁcally, for discrete features that are the quantized versions of continuous features, values of adjacent quantiles are clearly ‘closer’
Revised: Speciﬁcally, for discrete features that are the quantized versions of continuous features, values of adjacent quantiles are clearly ‘closer’

Original: that those of non-adjacent quantiles.
Revised: that those of non-adjacent quantiles.

Original: This is also true for natively discrete features like the star rating of a product.
Revised: This is also true for natively discrete features like the star rating of a product.

Original: One can design metrics smarter than the hamming distance, that also take into account these differences in values and are even more robust against adversarial attacks.
Revised: One can design metrics smarter than the hamming distance, that also take into account these differences in values and are even more robust against adversarial attacks.

Original: In this work however, we focus on the Hamming distance as a proof of concept and leave experimentation with other metrics to future work.
Revised: In this work however, we focus on the Hamming distance as a proof of concept and leave experimentation with other metrics to future work.

Original: D. P RACTICAL C ONSIDERATIONS
Revised: E. P RACTICAL C ONSIDERATIONS

Original: The tool by (Slack et al., 2020), as of the time of writing this paper, was only able to create an adversarial attack on continuous features (for LIME), and did not have the ability to train the OODdetecting classiﬁer on discrete data.
Revised: The tool by (Slack et al., 2020), as of the time of writing this paper, was only able to create an adversarial attack on continuous features (for LIME), and did not have the ability to train the OODdetecting classiﬁer on discrete data.

Original: For our experiments, we augmented their tool to handle purely discrete data by using LIME’s native perturbation procedure for generating training data for the
Revised: For our experiments, we augmented their tool to handle purely discrete data by using LIME’s native perturbation procedure for generating training data for the

Original: OOD-detecting classiﬁer.
Revised: OOD-detecting classiﬁer.

Original: For discrete data, the parameter ‘perturbation_multiplier’ in their tool plays an important role.
Revised: For discrete data, the parameter ‘perturbation_multiplier’ in their tool plays an important role.

Original: Its default value of 30 creates a very imbalanced dataset and the accuracy of the
Revised: Its default value of 30 creates a very imbalanced dataset and the accuracy of the

Original: OOD-detector is dramatically affected and it classiﬁes most instances as OOD.
Revised: OOD-detector is dramatically affected and it classiﬁes most instances as OOD.

Original: We found that a value of 2 led to the best test accuracy for the OOD-detector.
Revised: We found that a value of 2 led to the best test accuracy for the OOD-detector.

Original: With discrete data, we also observed that the efﬁcacy of the adversarial attack on LIME was drastically reduced.
Revised: With discrete data, we also observed that the efﬁcacy of the adversarial attack on LIME was drastically reduced.

Original: The sensitive feature appeared as the 2nd from top explanation almost 100% of the time.
Revised: The sensitive feature appeared as the 2nd from top explanation almost 100% of the time.

Original: Additionally, for attacks that used 2 uncorrelated features, the sensitive feature also appeared as the top explanation with a considerable frequency.
Revised: Additionally, for attacks that used 2 uncorrelated features, the sensitive feature also appeared as the top explanation with a considerable frequency.

Original: However, for attacks that used 1 uncorrelated feature, the top explanation was the uncorrelated feature with almost 100% frequency.
Revised: However, for attacks that used 1 uncorrelated feature, the top explanation was the uncorrelated feature with almost 100% frequency.

Original: Hence in our experiments, we focus on the top explanation ((Slack et al., 2020) considered top 3 explanations) and use only one unrelated feature.
Revised: Hence in our experiments, we focus on the top explanation ((Slack et al., 2020) considered top 3 explanations) and use only one unrelated feature.

Original: For CC and German experiments, we selected the top 12 most important features of the 3 datasets using scikit-learn’s Random Forest classiﬁer, as there were far too many features in the original datasets (100 for CC and 28 for German).
Revised: For CC and German experiments, we selected the top 12 most important features of the 3 datasets using scikit-learn’s Random Forest classiﬁer, as there were far too many features in the original datasets (100 for CC and 28 for German).

Original: The test accuracy of the
Revised: The test accuracy of the

Original: Random Forest with 100 estimators was 0.83 and 0.75 for CC and German respectively even after restriction to 12 features, down from 0.875 and 0.82, respectively, before removing features.
Revised: Random Forest with 100 estimators was 0.83 and 0.75 for CC and German respectively even after restriction to 12 features, down from 0.875 and 0.82, respectively, before removing features.

Original: D. R ESULTS
Revised: E. R ESULTS

Original: The results for the Recidivism dataset were depicted in Figure 2.
Revised: The results for the Recidivism dataset were depicted in Figure 4.

Original: The results for CC dataset are depicted in Fig.
Revised: The results for CC dataset are depicted in Fig.

Original: 5 and those for German in Fig.
Revised: 8 and those for German in Fig.

Original: It can be seen that the sensitive feature appears prominently for low hamming distance while the unrelated feature dominates higher for higher distance, which conﬁrms the efﬁcacy of CLIME.
Revised: It can be seen that the sensitive feature appears prominently for low hamming distance while the unrelated feature dominates higher for higher distance, which conﬁrms the efﬁcacy of CLIME.

Original: In general, it is harder to craft attacks similar to (Slack et al., 2020) against CLIME explanations, as the sampling procedure is dependent on user-provided constraints and is not ﬁxed like LIME.
Revised: In general, it is harder to craft attacks similar to (Slack et al., 2020) against CLIME explanations, as the sampling procedure is dependent on user-provided constraints and is not ﬁxed like LIME.

Original: However, for the current experiment, an adversary could conceivably train the OOD-detector on instances with low hamming distance from the original dataset to speciﬁcally foil our method.
Revised: However, for the current experiment, an adversary could conceivably train the OOD-detector on instances with low hamming distance from the original dataset to speciﬁcally foil our method.

Original: We tested whether the sensitive feature still appeared in the top CLIME explanation when the OODdetector was trained in this manner (instead of being trained on LIME generated perturbations).
Revised: We tested whether the sensitive feature still appeared in the top CLIME explanation when the OODdetector was trained in this manner (instead of being trained on LIME generated perturbations).

Original: We compare the frequency of seeing the sensitive feature as the top CLIME explanation of the adversarial classiﬁer in the two scenarios in Table 3.
Revised: We compare the frequency of seeing the sensitive feature as the top CLIME explanation of the adversarial classiﬁer in the two scenarios in Table 2.

Original: While the frequency of seeing the sensitive feature drops as compared to the case when the OOD detector is trained on LIME perturbations, the drop is not very steep and the sensitive feature still ﬁgures prominently as the top explanation.
Revised: While the frequency of seeing the sensitive feature drops as compared to the case when the OOD detector is trained on LIME perturbations, the drop is not very steep and the sensitive feature still ﬁgures prominently as the top explanation.

Original: This shows that our hamming distance based technique is not only able to detect attacks, but is also robust against attacks itself.
Revised: This shows that our hamming distance based technique is not only able to detect attacks, but is also robust against attacks itself.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Recidivism
Revised: Recidivism

Original: CC
Revised: CC

Original: German
Revised: German

Original: OOD detector trained on
Revised: OOD detector trained on

Original: LIME’s samples 0.
Revised: LIME’s samples 0.

Original: OOD detector trained on samples from Hamming-constraints 0.
Revised: OOD detector trained on samples from Hamming-constraints 0.