Original: Washing The Unwashable : On The (Im)possibility of Fairwashing Detection
Revised: Washing The Unwashable : On The (Im)possibility of Fairwashing Detection

Original: Abstract
Revised: Abstract

Original: The use of black-box models ( e.g. , deep neural networks) in high-stakes decisionmaking systems, whose internal logic is complex, raises the need for providing explanations about their decisions.
Revised: The use of black-box models ( e.g. , deep neural networks) in high-stakes decisionmaking systems, whose internal logic is complex, raises the need for providing explanations about their decisions.

Original: Model explanation techniques mitigate this problem by generating an interpretable and high-fidelity surrogate model ( e.g. , a logistic regressor or decision tree) to explain the logic of black-box models.
Revised: Model explanation techniques mitigate this problem by generating an interpretable and high-ﬁdelity surrogate model ( e.g. , a logistic regressor or decision tree) to explain the logic of black-box models.

Original: In this work, we investigate the issue of fairwashing, in which model explanation techniques are manipulated to rationalize decisions taken by an unfair black-box model using deceptive surrogate models.
Revised: In this work, we investigate the issue of fairwashing, in which model explanation techniques are manipulated to rationalize decisions taken by an unfair black-box model using deceptive surrogate models.

Original: More precisely, we theoretically characterize and analyze fairwashing, proving that this phenomenon is difficult to avoid due to an irreducible factor—the unfairness of the black-box model.
Revised: More precisely, we theoretically characterize and analyze fairwashing, proving that this phenomenon is difﬁcult to avoid due to an irreducible factor—the unfairness of the black-box model.

Original: Based on the theory developed, we propose a novel technique, called FRAUD-Detect (FaiRness AUDit Detection), to detect fairwashed models by measuring a divergence over subpopulation-wise fidelity measures of the interpretable model.
Revised: Based on the theory developed, we propose a novel technique, called FRAUD-Detect (FaiRness AUDit Detection), to detect fairwashed models by measuring a divergence over subpopulation-wise ﬁdelity measures of the interpretable model.

Original: We empirically demonstrate that this divergence is significantly larger in purposefully fairwashed interpretable models than in honest ones.
Revised: We empirically demonstrate that this divergence is signiﬁcantly larger in purposefully fairwashed interpretable models than in honest ones.

Original: Furthermore, we show that our detector is robust to an informed adversary trying to bypass our detector.
Revised: Furthermore, we show that our detector is robust to an informed adversary trying to bypass our detector.

Original: The code implementing FRAUD-Detect is available at https://github.com/cleverhans-lab/FRAUD-Detect .
Revised: The code implementing FRAUD-Detect is available at https://github.com/cleverhans-lab/FRAUD-Detect .

Original:  Introduction
Revised:  Introduction

Original: The wide applicability of machine learning models has recently increased their usage in high-stakes decision systems such as credit scoring [43], insurance risk [10] and predictive justice [30].
Revised: The wide applicability of machine learning models has recently increased their usage in high-stakes decision systems such as credit scoring [43], insurance risk [10] and predictive justice [30].

Original: The consequences of erroneous decisions that are based on predictions of machine learning models ( e.g. , people being wrongly denied parole [46]) have increased the demand—from both the public and government—to provide an explanation to humans about model decisions.
Revised: The consequences of erroneous decisions that are based on predictions of machine learning models ( e.g. , people being wrongly denied parole [46]) have increased the demand—from both the public and government—to provide an explanation to humans about model decisions.

Original: For instance, this appears as an explanation requirement in the European General Data Protection Regulation [26].
Revised: For instance, this appears as an explanation requirement in the European General Data Protection Regulation [26].

Original: However, various widely-used model architectures, such as deep neural networks, are considered black-boxes due to their complex and hidden internal logic, impeding the ability to explain their decisions in terms that are understandable by a human.
Revised: However, various widely-used model architectures, such as deep neural networks, are considered black-boxes due to their complex and hidden internal logic, impeding the ability to explain their decisions in terms that are understandable by a human.

Original: To address this issue, black-box model
Revised: To address this issue, black-box model

Original: [Figure][Figure]
Revised: [Figure]

Revised: Fairwashing?

Original: T −
Revised: T

Original: T +
Revised: T +

Original: F +
Revised: F +

Original: F −.2.
Revised: F  0.

Original: R a t e
Revised: R a t e

Original: Honest interpretable model
Revised: Honest interpretable model

Original: T −
Revised: T

Original: T +
Revised: T +

Original: F +
Revised: F +

Original: F −.2.
Revised: F  0.

Original: R a t e
Revised: R a t e

Original: [Figure]
Revised: [Figure]

Original: explanation techniques [27, 6] aim to provide reasons for predictions of black-box models in humanunderstandable terms.
Revised: explanation techniques [27, 6] aim to provide reasons for predictions of black-box models in humanunderstandable terms.

Original: One possible way to realize this is to approximate the black-box model using an inherently interpretable model ( e g. , logistic regression or decision tree) whose decisions can be easily explained by design [27].
Revised: One possible way to realize this is to approximate the black-box model using an inherently interpretable model ( e g. , logistic regression or decision tree) whose decisions can be easily explained by design [27].

Original: For instance, in logistic regression, the coefficients of the model directly represent the importance of each input feature, while a decision tree is composed of a set of explainable if-then decision rules related to the input features.
Revised: For instance, in logistic regression, the coefﬁcients of the model directly represent the importance of each input feature, while a decision tree is composed of a set of explainable if-then decision rules related to the input features.

Original: In this black-box model explanation setting, the interpretable model is trained with the objective that its predictions agree with the predictions of the black-box model ( i.e. , high fidelity [20]).
Revised: In this black-box model explanation setting, the interpretable model is trained with the objective that its predictions agree with the predictions of the black-box model ( i.e. , high ﬁdelity [20]).

Original: However, black-box model explanations can be manipulated to provide deceiving explanations, even if these explanations display a high fidelity with respect to the original black-box model.
Revised: However, black-box model explanations can be manipulated to provide deceiving explanations, even if these explanations display a high ﬁdelity with respect to the original black-box model.

Original: An example of such manipulation occurs when the explanations provided cover up the unfairness of the underlying black-box model, resulting in fairwashing [2, 3].
Revised: An example of such manipulation occurs when the explanations provided cover up the unfairness of the underlying black-box model, resulting in fairwashing [2, 3].

Original: As a concrete example, a dishonest bank [4] could use a blackbox model that denies a loan to a customer in a discriminative manner—based on a sensitive attribute such as race or gender identity—while providing an interpretable surrogate model promoting the false impression that the decisions are fair and based solely on non-sensitive features.
Revised: As a concrete example, a dishonest bank [4] could use a blackbox model that denies a loan to a customer in a discriminative manner—based on a sensitive attribute such as race or gender identity—while providing an interpretable surrogate model promoting the false impression that the decisions are fair and based solely on non-sensitive features.

Original: These manipulations are made possible in part by the ambiguity in current regulations [26] when it comes to describing what constitutes a valid explanation [45].
Revised: These manipulations are made possible in part by the ambiguity in current regulations [26] when it comes to describing what constitutes a valid explanation [45].

Original: Fairwashing is clearly an ethical issue as individuals who have unfairly received unfavorable decisions are deprived of a chance to contest the decision [3].
Revised: Fairwashing is clearly an ethical issue as individuals who have unfairly received unfavorable decisions are deprived of a chance to contest the decision [3].

Original: Due to the growing importance of fairness in machine learning [7], we focus on a scenario in which a dishonest entity attempts to perform fairwashing [2, 3] when providing model explanations to an auditor ( e.g. , an external dedicated party).
Revised: Due to the growing importance of fairness in machine learning [7], we focus on a scenario in which a dishonest entity attempts to perform fairwashing [2, 3] when providing model explanations to an auditor ( e.g. , an external dedicated party).

Original: For example, if a company’s users expect that the decisions they received were a result of model bias, they may send their data and received decisions to an auditor.
Revised: For example, if a company’s users expect that the decisions they received were a result of model bias, they may send their data and received decisions to an auditor.

Original: The auditor may make a legal demand [8] on the company to provide an interpretable model to explain these users’ received decisions (see Figure 1).
Revised: The auditor may make a legal demand [8] on the company to provide an interpretable model to explain these users’ received decisions (see Figure 1).

Original: In this case, a dishonest company has an incentive to manipulate the black-box model explanation technique, to under-report and occlude the unfairness of the black-box model under scrutiny to evade the potential consequences of unfair decision-making.
Revised: In this case, a dishonest company has an incentive to manipulate the black-box model explanation technique, to under-report and occlude the unfairness of the black-box model under scrutiny to evade the potential consequences of unfair decision-making.

Original: The dishonest entity could thus mislead the auditor by rationalizing its decisions using the corresponding fairwashed interpretable model.
Revised: The dishonest entity could thus mislead the auditor by rationalizing its decisions using the corresponding fairwashed interpretable model.

Original: In this paper, we are the first to propose a method for detecting such dishonest model explanations.
Revised: In this paper, we are the ﬁrst to propose a method for detecting such dishonest model explanations.

Original: This is a challenging problem.
Revised: This is a challenging problem.

Original: Fairwashing cannot be detected through computing the explanation violation of the explainable model, such as fidelity of explainable models, as we demonstrate that the explainable model cannot achieve perfect fidelity with respect to the black-box model.
Revised: Fairwashing cannot be detected through computing the explanation violation of the explainable model, such as ﬁdelity of explainable models, as we demonstrate that the explainable model cannot achieve perfect ﬁdelity with respect to the black-box model.

Original: In addition to this, measuring the differences between the fairness of the black-box model and the explainable model cannot help to detect fairwashing as there are several design choices (including fairwashing) that could lead to different fairness, thus non-intentional fairwashing.
Revised: In addition to this, measuring the differences between the fairness of the black-box model and the explainable model cannot help to detect fairwashing as there are several design choices (including fairwashing) that could lead to different fairness, thus non-intentional fairwashing.

Original: Our primary contributions include (1) an extensive theoretical analysis of fairwashing and the fairness limitations in explanation techniques and (2) a novel approach for detecting, and thus deterring, fairwashing.
Revised: Our primary contributions include (1) an extensive theoretical analysis of fairwashing and the fairness limitations in explanation techniques and (2) a novel approach for detecting, and thus deterring, fairwashing.

Original: Our proposed method, FRAUD-Detect (FaiRness AUDit Detection), formulates the problem of detecting fairwashing as a non-cooperative test; both to overcome possible dishonest behaviors in entities that are actively being audited and for communication efficiency.
Revised: Our proposed method, FRAUD-Detect (FaiRness AUDit Detection), formulates the problem of detecting fairwashing as a non-cooperative test; both to overcome possible dishonest behaviors in entities that are actively being audited and for communication efﬁciency.

Original: FRAUDDetect operates in a realistic scenario: FRAUD-Detect does not require access to the black-box model provided the users calling for the audit have provided their data and received decisions, which is a realistic setting as often the model is not provided by the entity due to intellectual property and tradesecret concerns [39, 37].
Revised: FRAUDDetect operates in a realistic scenario: FRAUD-Detect does not require access to the black-box model provided the users calling for the audit have provided their data and received decisions, which is a realistic setting as often the model is not provided by the entity due to intellectual property and tradesecret concerns [39, 37].

Original: Thus, FRAUD-Detect only relies on the predictions of the interpretable and black-box models.
Revised: Thus, FRAUD-Detect only relies on the predictions of the interpretable and black-box models.

Original: Both our theoretical and empirical analysis quantify the per-subpopulation, per-label fidelity of the interpretable model with respect to the black-box model.
Revised: Both our theoretical and empirical analysis quantify the per-subpopulation, per-label ﬁdelity of the interpretable model with respect to the black-box model.

Original: Figure 1 shows confusion matrix distributions for two sensitive subpopulations before and after fairwashing.
Revised: Figure 1 shows confusion matrix distributions for two sensitive subpopulations before and after fairwashing.

Original: These matrices are constructed by comparing the interpretable to the black-box model predictions.
Revised: These matrices are constructed by comparing the interpretable to the black-box model predictions.

Original: Fairwashing induces a divergence between subpopulation values to conceal the unfairness of the black-box model.
Revised: Fairwashing induces a divergence between subpopulation values to conceal the unfairness of the black-box model.

Original: FRAUD-Detect leverages the Kullback–Leibler (KL) divergence between subpopulation confusion matrices to distinguish between honest and fairwashed interpretable models.
Revised: FRAUD-Detect leverages the Kullback–Leibler (KL) divergence between subpopulation confusion matrices to distinguish between honest and fairwashed interpretable models.

Original: For a comprehensive empirical study, we examine logistic regressors and decision trees as interpretable models with respect to black-box Deep Neural Networks, AdaBoost [24], Gradient
Revised: For a comprehensive empirical study, we examine logistic regressors and decision trees as interpretable models with respect to black-box Deep Neural Networks, AdaBoost [24], Gradient

Original: Boosted Decision Trees [17] and Random Forests [13] on three benchmark datasets: COMPAS [5],
Revised: Boosted Decision Trees [17] and Random Forests [13] on three benchmark datasets: COMPAS [5],

Original: Adult Income [22] and Bank Marketing [34].
Revised: Adult Income [22] and Bank Marketing [34].

Original: We evaluate the strength of our detector in the presence of an informed adversary representing an informed dishonest entity with knowledge of FRAUD-Detect that seeks to fairwash and bypass detection by constraining the divergence over subpopulation-wise confusion matrices.
Revised: We evaluate the strength of our detector in the presence of an informed adversary representing an informed dishonest entity with knowledge of FRAUD-Detect that seeks to fairwash and bypass detection by constraining the divergence over subpopulation-wise confusion matrices.

Original: Our empirical experiments quantify the capacity of this informed adversary by computing the achievable fairness gap and fidelity under the additional detection constraint.
Revised: Our empirical experiments quantify the capacity of this informed adversary by computing the achievable fairness gap and ﬁdelity under the additional detection constraint.

Original: We provide additional theoretical support for the challenge of jointly satisfying fairwashing and evasion in appendices.
Revised: We provide additional theoretical support for the challenge of jointly satisfying fairwashing and evasion in appendices.

Original: In summary, our main contributions are as follows:
Revised: In summary, our main contributions are as follows:

Original: • We characterize fairwashing via a theoretical analysis on the difference in subpopulation gap between the black-box model and the interpretable model, fairwashing violation (Section 3).
Revised: • We characterize fairwashing via a theoretical analysis on the difference in subpopulation gap between the black-box model and the interpretable model, fairwashing violation (Section 3).

Original: • Based on our fairwashing theory, we establish requirements for a sufficient fairwashing detection method and prove that fairwashing is impossible to avoid completely due to an irreducible component in the violation term.
Revised: • Based on our fairwashing theory, we establish requirements for a sufﬁcient fairwashing detection method and prove that fairwashing is impossible to avoid completely due to an irreducible component in the violation term.

Original: • We introduce the first method for fairwashing detection (Section 4.2).
Revised: • We introduce the ﬁrst method for fairwashing detection (Section 4.2).

Original: Informed by our theoretical results, we observe that fairwashing causes disparate effect on the subpopulation-wise fidelity distributions of the interpretable model with respect to the black-box predictions.
Revised: Informed by our theoretical results, we observe that fairwashing causes disparate effect on the subpopulation-wise ﬁdelity distributions of the interpretable model with respect to the black-box predictions.

Original: We leverage this observation to propose a non-cooperative black-box access method, FRAUDDetect , which utilizes KL divergence on per-subpopulation confusion matrices to distinguish between fairwashed and honest interpretable models.
Revised: We leverage this observation to propose a non-cooperative black-box access method, FRAUDDetect , which utilizes KL divergence on per-subpopulation confusion matrices to distinguish between fairwashed and honest interpretable models.

Original: • Our empirical results demonstrate that FRAUD-Detect successfully detects fairwashing based on the KL divergence between subpopulation-wise confusion matrices (Section 6).
Revised: • Our empirical results demonstrate that FRAUD-Detect successfully detects fairwashing based on the KL divergence between subpopulation-wise confusion matrices (Section 6).

Original: We show that the divergence over subpopulation confusion matrices can vary by over 0 .between an honest and a fairwashed interpretable models.
Revised: We show that the divergence over subpopulation confusion matrices can vary by over 0 .between an honest and a fairwashed interpretable models.

Original: • We illustrate the robustness of FRAUD-Detect against an informed adversary ( i
Revised: • We illustrate the robustness of FRAUD-Detect against an informed adversary ( i

Original: e. , a dishonest entity who attempts to fairwash while evading detection).
Revised: e. , a dishonest entity who attempts to fairwash while evading detection).

Original: Our empirical results show that evading our detector comes at the cost of a significant increase in subpopulation gap, negating fairwashing.
Revised: Our empirical results show that evading our detector comes at the cost of a signiﬁcant increase in subpopulation gap, negating fairwashing.

Original: Specifically, a dishonest entity that jointly attempts to fairwash while evading detection only achieves parity gaps greater than 10 %.Preliminaries and Problem
Revised: Speciﬁcally, a dishonest entity that jointly attempts to fairwash while evading detection only achieves parity gaps greater than 10 %.

Original: 2.1 Related Work
Revised: 2.1 Related Work

Original: Fairness.
Revised: Fairness.

Original: Many different formal definitions of algorithmic fairness have been proposed in the machine learning community [35].
Revised: Many different formal deﬁnitions of algorithmic fairness have been proposed in the machine learning community [35].

Original: One of the main challenges of algorithmic fairness is the lack of consensus on a universally applicable fairness definition.
Revised: One of the main challenges of algorithmic fairness is the lack of consensus on a universally applicable fairness deﬁnition.

Original: More precisely, it has been formally demonstrated that many such measures are incompatible with each other ( i.e. , they cannot be achieved jointly in some situations) [31].
Revised: More precisely, it has been formally demonstrated that many such measures are incompatible with each other ( i.e. , they cannot be achieved jointly in some situations) [31].

Original: In addition, their differences are also rooted in their underlying philosophical and moral assumptions [28].
Revised: In addition, their differences are also rooted in their underlying philosophical and moral assumptions [28].

Original: Thus, the choice of a particular fairness metric is usually context dependent.
Revised: Thus, the choice of a particular fairness metric is usually context dependent.

Original: Nonetheless broadly speaking, there are two main families of fairness definitions: individual fairness and group fairness [23].
Revised: Nonetheless broadly speaking, there are two main families of fairness deﬁnitions: individual fairness and group fairness [23].

Original: Individual fairness posits that “similar individuals should be treated similarly”.
Revised: Individual fairness posits that “similar individuals should be treated similarly”.

Original: In contrast, group fairness strives to equalize statistical properties of classification outcomes across subpopulations created by partitioning the population based on a sensitive attribute A such as gender.
Revised: In contrast, group fairness strives to equalize statistical properties of classiﬁcation outcomes across subpopulations created by partitioning the population based on a sensitive attribute A such as gender.

Original: In our work, we focus on a particular notion of group fairness, demographic parity, also called statistical group parity, which refers to observing equal probability of positive label prediction over subpopulations.
Revised: In our work, we focus on a particular notion of group fairness, demographic parity, also called statistical group parity, which refers to observing equal probability of positive label prediction over subpopulations.

Original: One of the reasons we rely on demographic parity as the fairness metric is that true labels will not be available in a realistic scenario (especially in fairwashing scenario described in Section 2.3).
Revised: One of the reasons we rely on demographic parity as the fairness metric is that true labels will not be available in a realistic scenario (especially in fairwashing scenario described in Section 2.3).

Original: In general, the existence of true labels conflicts with the assumption that when requesting decisions (or explanations), users (or auditors) do not have access to these true labels [42].
Revised: In general, the existence of true labels conﬂicts with the assumption that when requesting decisions (or explanations), users (or auditors) do not have access to these true labels [42].

Original: Furthermore, using the predictions of a black-box model as the true labels is paradoxical in the sense that the black-box model is assumed to be unfair.
Revised: Furthermore, using the predictions of a black-box model as the true labels is paradoxical in the sense that the black-box model is assumed to be unfair.

Original: Demographic parity enforces independence between the class predicted by the model and inclusion in a particular subpopulation.
Revised: Demographic parity enforces independence between the class predicted by the model and inclusion in a particular subpopulation.

Original: Definition 1 (Demographic parity [15]) .
Revised: Deﬁnition 1 (Demographic parity [15]) .

Original: A classifier ˆ Y satisfies demographic parity with respect to sensitive attribute A
Revised: A classiﬁer ˆ Y satisﬁes demographic parity with respect to sensitive attribute A

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: Global black-box explanation methods focus on explaining the whole logic of black-box models by training an inherently interpretable surrogate model.
Revised: Global black-box explanation methods focus on explaining the whole logic of black-box models by training an inherently interpretable surrogate model.

Original: We refer the interested reader to Appendix A for the description of other explanation methods.
Revised: We refer the interested reader to Appendix A for the description of other explanation methods.

Original: In terms of abusing explanations via fairwashing,
Revised: In terms of abusing explanations via fairwashing,

Original: Fukuchi et al., similarly to us, try to detect fairwashing but with a different setting [25].
Revised: Fukuchi et al., similarly to us, try to detect fairwashing but with a different setting [25].

Original: Slack et al.
Revised: Slack et al.

Original: shows an alternate method of fairwashing that evades local model explanation techniques in a setting that is comparable to ours but they do not attempt to detect fairwashing [44].
Revised: shows an alternate method of fairwashing that evades local model explanation techniques in a setting that is comparable to ours but they do not attempt to detect fairwashing [44].

Original: We explore these additional fairwashing settings in Appendix B.
Revised: We explore these additional fairwashing settings in Appendix B.

Original: 2.2 Desiderata for Global Black-box Explanations
Revised: 2.2 Desiderata for Global Black-box Explanations

Original: We consider the setting in which an entity learns a complex black-box model B ( · ) on a training set ( X Tr , Y Tr , A ) , in which A represents a sensitive attribute.
Revised: We consider the setting in which an entity learns a complex black-box model B ( · ) on a training set ( X Tr , Y Tr , A ) , in which A represents a sensitive attribute.

Original: We maintain the setup from recent fairwashing literature [3, 2] and consider binary classifiers mapping M features into a binary label
Revised: We maintain the setup from recent fairwashing literature [3, 2] and consider binary classiﬁers mapping M features into a binary label

Original: B : R M (cid:55)→ { 0 , 1 } .
Revised: B : R M 7! { 0 , 1 } .

Original: We also consider binary-valued sensitive attributes, A ∈ { 0 , 1 } .
Revised: We also consider binary-valued sensitive attributes, A 2 { 0 , 1 } .

Original: Note that we do not assume that A is necessarily used during the training of B ( · ) .
Revised: Note that we do not assume that A is necessarily used during the training of B ( · ) .

Original: Given user queries, the entity uses the predictions of B ( · ) as part of a high-stakes decision system.
Revised: Given user queries, the entity uses the predictions of B ( · ) as part of a high-stakes decision system.

Original: Later, a group of these users may request an explicit explanation from the entity due to concerns over improper use of the sensitive attribute.
Revised: Later, a group of these users may request an explicit explanation from the entity due to concerns over improper use of the sensitive attribute.

Original: Such a scenario frequently arises in high-impact domains, such as bank loans and credit scoring.
Revised: Such a scenario frequently arises in high-impact domains, such as bank loans and credit scoring.

Original: Let X sg be a suing set comprising the unlabeled data of users demanding an explanation for their particular outcomes (see Figure 1).
Revised: Let X sg be a suing set comprising the unlabeled data of users demanding an explanation for their particular outcomes (see Figure 1).

Original: Consequently, an external auditing entity requires the company to provide explanations in terms that are understandable to humans about the predictions of the black-box.
Revised: Consequently, an external auditing entity requires the company to provide explanations in terms that are understandable to humans about the predictions of the black-box.

Original: Here, we will assume that the explanation will be provided in the form of a global explanation.
Revised: Here, we will assume that the explanation will be provided in the form of a global explanation.

Original: This means that a simple interpretable model I ( · ) will be trained on a dataset X labeled by querying the black-box model B ( · ) trained on X Tr , such that I ( · ) accurately reflects and explains the logic of B ( · ) in terms that are understandable to humans.
Revised: This means that a simple interpretable model I ( · ) will be trained on a dataset X labeled by querying the black-box model B ( · ) trained on X Tr , such that I ( · ) accurately reﬂects and explains the logic of B ( · ) in terms that are understandable to humans.

Original: Formalizing further the auditing desiderata for black-box model explanations, the interpretable model
Revised: Formalizing further the auditing desiderata for black-box model explanations, the interpretable model

Original: I ( · ) must accurately mirror: 1) the output predictions of the black-box model ( i.e. , fidelity) and 2) the fairness (according to a pre
Revised: I ( · ) must accurately mirror: 1) the output predictions of the black

Original: Fidelity.
Revised: Fidelity.

Original: The interpretable model should satisfy the fidelity criterion for any X ∼ D .
Revised: The interpretable model should satisfy the ﬁdelity criterion for any X ⇠ D .

Original: Fidelity (defined in Appendix C) is difficult to perfectly achieve and challenging to measure for an auditor over all X ∼ D .
Revised: Fidelity (deﬁned in Appendix C) is difﬁcult to perfectly achieve and challenging to measure for an auditor over all X ⇠ D .

Original: Thus, we additionally introduce the notion of empirical fidelity.
Revised: Thus, we additionally introduce the notion of empirical ﬁdelity.

Original: Definition 2 (Empirical Fidelity) .
Revised: Deﬁnition 2 (Empirical Fidelity) .

Original: The empirical fidelity on a dataset X = { x i } Ni =1 including N data points x i is defined as the relative accuracy of I ( · ) with respect to B ( · ) on X :[Equation] B
Revised: The empirical ﬁdelity on a dataset X = { x i } Ni =1 including N data points x i is deﬁned as the relative accuracy of I ( · ) with respect to B ( · ) on X :

Original: [Equation]
Revised: [Equation]

Original: where the indicator 1 ( · , · ) outputs 1 if the output label of I ( · ) and B ( · ) are the same and 0 otherwise.
Revised: where the indicator 1 ( · , · ) outputs 1 if the output label of I ( · ) and B ( · ) are the same and 0 otherwise.

Original: Fairness.
Revised: Fairness.

Original: As the audit of B ( · ) is expressly requested due to fairness concerns with respect to A , the interpretable model should reflect the fairness violations or adherences of the black-box model.
Revised: As the audit of B ( · ) is expressly requested due to fairness concerns with respect to A , the interpretable model should reﬂect the fairness violations or adherences of the black-box model.

Original: 2.3 Fairwashing Definitions
Revised: 2.3 Fairwashing Deﬁnitions

Original: To evade legal consequences of decision-making based on improper use of a sensitive attribute, a dishonest company may perform fairwashing .
Revised: To evade legal consequences of decision-making based on improper use of a sensitive attribute, a dishonest company may perform fairwashing .

Original: Fairwashing with respect to a pre-defined fairness metric permits the dishonest company to learn an interpretable model that hides the unfair behaviour of their black-box model.
Revised: Fairwashing with respect to a pre-deﬁned fairness metric permits the dishonest company to learn an interpretable model that hides the unfair behaviour of their black-box model.

Original: In practice, fairwashing could also occur due to the fact that current regulations [26] do not define what constitutes a valid explanation , thus leaving the possibility for the model provider to choose the interpretable model that meets their needs [45].
Revised: In practice, fairwashing could also occur due to the fact that current regulations [26] do not deﬁne what constitutes a valid explanation , thus leaving the possibility for the model provider to choose the interpretable model that meets their needs [45].

Original: Fairwashing can be quantified with respect to a particular fairness measure.
Revised: Fairwashing can be quantiﬁed with respect to a particular fairness measure.

Original: Definition 3 (Fairwashing) .
Revised: Deﬁnition 3 (Fairwashing) .

Original: Let Γ I and Γ B define the fairness gaps of the interpretable model and black-box model, respectively, with respect to a pre-defined fairness metric.
Revised: Let  I and  B deﬁne the fairness gaps of the interpretable model and black-box model, respectively, with respect to a pre-deﬁned fairness metric.

Original: For example, in terms of demographic parity (Definition 1):
Revised: For example, in terms of demographic parity (Deﬁnition 1):

Original: [Equation]
Revised: [Equation]

Original: We define the fairwashing violation γ > 0 as the difference between Γ I and Γ B :
Revised: We deﬁne the fairwashing violation  > 0 as the difference between  I and  B :

Original: [Equation]
Revised: [Equation]

Original: Assumption 1.
Revised: Assumption 1.

Original: The fairwashing violation γ is non-negative.
Revised: The fairwashing violation  is non-negative.

Original: Remark that the case of γ < 0 is the opposite of fairwashing since the interpretable model is displaying a bigger fairness gap than that of the black-box model, thus a model owner has no incentive to employ such an interpretable model.
Revised: Remark that the case of  < 0 is the opposite of fairwashing since the interpretable model is displaying a bigger fairness gap than that of the black-box model, thus a model owner has no incentive to employ such an interpretable model.

Original:  Fairness Gap of the Black-Box Breeds Fairwashing
Revised:  Fairness Gap of the Black-Box Breeds Fairwashing

Original: We analyze whether it is possible to eliminate the risk of fairwashing altogether.
Revised: We analyze whether it is possible to eliminate the risk of fairwashing altogether.

Original: We characterize the fairwashing violation γ and show that completely eliminating fairwashing is impossible: an interpretable model explaining an unfair black-box model always has a non-zero fairwashing violation.
Revised: We characterize the fairwashing violation  and show that completely eliminating fairwashing is impossible: an interpretable model explaining an unfair black-box model always has a non-zero fairwashing violation.

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume ˆ Y B are the black-box model B ( · ) predictions, ˆ Y
Revised: Assume ˆ Y B are the black-box model B ( · ) predictions, ˆ Y

Original: I are the predictions of a surrogate (interpretable) model I ( · ) trained on B ( · ) outputs and A is a sensitive attribute: 1) If B ( · ) does not satisfy demographic parity, completely eliminating fairwashing is impossible (i.e., γ > 0 ); 2)
Revised: I are the predictions of a surrogate (interpretable) model I ( · ) trained on B ( · ) outputs and A is a sensitive attribute: 1) If B ( · ) does not satisfy demographic parity, completely eliminating fairwashing is impossible (i.e.,  > 0 ); 2)

Original: A detector for fairwashing measuring false-positive rates and true-positive rates of I ( · ) with respect to B ( · ) is sufficient.
Revised: A detector for fairwashing measuring false-positive rates and true-positive rates of I ( · ) with respect to B ( · ) is sufﬁcient.

Original: We provide a proof sketch here (see Appendix D for full proof).
Revised: We provide a proof sketch here (see Appendix D for full proof).

Original: Proof Sketch.
Revised: Proof Sketch.

Original: Define T + a , the true-positive rate of I ( · ) with respect to B ( · ) on X ∼ D a , in which D a denotes the distribution over data with attribute a ∈ A .
Revised: Deﬁne T + a , the true-positive rate of I ( · ) with respect to B ( · ) on X ⇠ D a , in which D a denotes the distribution over data with attribute a 2 A .

Original: Define F + a , the false-positive rate of I ( · ) with respect to B ( · ) on X ∼ D a .
Revised: Deﬁne F + a , the false-positive rate of I ( · ) with respect to B ( · ) on X ⇠ D a .

Original: Denote the differences between T +0 and T +1 , and F +0 and F +1 as ˜ δ, δ ′ , respectively.
Revised: Denote the differences between T +0 and T +1 , and F +0 and F +1 as ˜  ,  0 , respectively.

Original: We can express Γ I in terms of ˜ δ, δ ′ , T +1 , F +1 , Γ B and Y B |
Revised: We can express  I in terms of ˜  ,  0 , T +1 , F +1 ,  B and Y B |

Original: A by expanding terms and using Bayes’ formula:
Revised: A by expanding terms and using Bayes’ formula:

Original: [Equation]
Revised: [Equation]

Original: Now that we have derived Γ I in terms of our desired terms, we can eliminate Γ I from the equation for γ .
Revised: Now that we have derived  I in terms of our desired terms, we can eliminate  I from the equation for  .

Original: The resulting formula for γ in terms of ˜ δ, δ ′ , T +1 , F +1 , Γ B and Y B |
Revised: The resulting formula for  in terms of ˜  ,  0 , T +1 , F +1 ,  B and Y B |

Original: A allows us to demonstrate correctness , irreducibility and sufficiency .
Revised: A allows us to demonstrate correctness , irreducibility and sufﬁciency .

Original: [Equation]
Revised: [Equation]

Original: Correctness.
Revised: Correctness.

Original: Note that if Γ B = 0 ( i.e. , black-box model is fair) then no-fairwashing is present, and no detector should mistakenly report fairwashing.
Revised: Note that if  B = 0 ( i.e. , black-box model is fair) then no-fairwashing is present, and no detector should mistakenly report fairwashing.

Original: We can verify this with Equation (15).
Revised: We can verify this with Equation (15).

Original: Under
Revised: Under

Original: Assumption 1, we know that γ ≥ 0 and as below terms are, by definition non-negative, thus:
Revised: Assumption 1, we know that   0 and as below terms are, by deﬁnition non-negative, thus:

Original: [Equation]
Revised: [Equation]

Original: Irreducibility.
Revised: Irreducibility.

Original: If Γ B ̸ = 0 , even if ˜ δ = δ ′ = 0 , i.e. when true-positive and false-positive rates are equal over subpopulations ( T +0 = T +1 and F +0 = F +1 )
Revised: If  B 6 = 0 , even if ˜  =  0 = 0 , i.e. when true-positive and false-positive rates are equal over subpopulations ( T +0 = T +1 and F +0 = F +1 )

Original: , there exists an irreducible fairwashing violation:
Revised: , there exists an irreducible fairwashing violation:

Original: [Equation]
Revised: [Equation]

Original: ) since 1 + F +1 − T +1 > 0 except in trivial cases (see Appendix D for the proof).
Revised: since 1 + F +1  T +1 > 0 except in trivial cases (see Appendix D for the proof).

Original: We further point out that 1 + F + 1 − T + 1 is not a function of disparity in the interpretable model as the value does not measure disparity between subpopulations.
Revised: We further point out that 1 + F + 1  T + 1 is not a function of disparity in the interpretable model as the value does not measure disparity between subpopulations.

Original: Rather, 1 + F +1 − T +1 is a function of the difference in true-positive and false-positive rates ( T +1 and F +1 , respectively) within a single subpopulation.
Revised: Rather, 1 + F +1  T +1 is a function of the difference in true-positive and false-positive rates ( T +1 and F +1 , respectively) within a single subpopulation.

Original: The sole measure of disparity in the gap above is black-box Γ B , which is a given constant.
Revised: The sole measure of disparity in the gap above is black-box  B , which is a given constant.

Original: Sufficiency.
Revised: Sufﬁciency.

Original: The remaining factors of the gap are only function of ˜ δ and δ ′ as Pr (cid:104) ˆ Y B =
Revised: The remaining factors of the gap are only function of ˜  and  0 as Pr h ˆ Y B =

Original: 1 | A = 0 (cid:105) and Pr (cid:104)
Revised: 1 | A = i and Pr h

Original: ˆ Y B =
Revised: ˆ Y B =

Original: 1 | A = 0 (cid:105) are also given constants.
Revised: 1 | A =i are also given constants.

Original: Therefore, a detector that measures ˜ δ and δ ′ is sufficient to detect fairwashing.
Revised: Therefore, a detector that measures ˜  and  0 is sufﬁcient to detect fairwashing.

Original:  Detecting Fairwashing 4.1 Fairwashing In Practice
Revised:  Detecting Fairwashing 4.1 Fairwashing In Practice

Original: Eliminating the fairness gap of the black-box model would be the ideal outcome as it would also remove the possibility for a dishonest model provider to perform fairwashing.
Revised: Eliminating the fairness gap of the black-box model would be the ideal outcome as it would also remove the possibility for a dishonest model provider to perform fairwashing.

Original: However, in practice absolute fairness is often unrealistic and even regulatory bodies tolerate small violations of fairness laws [18].
Revised: However, in practice absolute fairness is often unrealistic and even regulatory bodies tolerate small violations of fairness laws [18].

Original: In Section 3, we showed that fairness violations result in fairwashing violations unless the interpretable model has a higher fairness gap than the black-box model, which would not be desirable from the point of view of the model provider.
Revised: In Section 3, we showed that fairness violations result in fairwashing violations unless the interpretable model has a higher fairness gap than the black-box model, which would not be desirable from the point of view of the model provider.

Original: Therefore, not all fairwashing violations may have been intended by the model owners.
Revised: Therefore, not all fairwashing violations may have been intended by the model owners.

Original: These involuntary violations are in fact side-effects of the imperfect fidelity, which is inherent to optimizing the empirical fidelity (Definition 2) of the interpretable model on a training set X tr ∼ D ,
Revised: These involuntary violations are in fact side-effects of the imperfect ﬁdelity, which is inherent to optimizing the empirical ﬁdelity (Deﬁnition 2) of the interpretable model on a training set X tr ⇠ D ,

Original: [Equation]
Revised: [Equation]

Original: In contrast, deliberate fairwashing is the result of a similar optimization problem purposefully constrained to provide better fairness on a target suing set X sg ∼ D :
Revised: In contrast, deliberate fairwashing is the result of a similar optimization problem purposefully constrained to provide better fairness on a target suing set X sg ⇠ D :

Original: Definition 4 (Fairwashing Optimization [2]) .
Revised: Deﬁnition 4 (Fairwashing Optimization [2]) .

Original: Given a black-box model B ( · ) and a suing set X sg , fairwashing optimization is defined as learning an interpretable model I ( · ) from B ( · ) on X sg such that the interpretable model has 1) high fidelity with respect to the black-box model and 2) is less unfair than this black-box model:
Revised: Given a black-box model B ( · ) and a suing set X sg , fairwashing optimization is deﬁned as learning an interpretable model I ( · ) from B ( · ) on X sg such that the interpretable model has 1) high ﬁdelity with respect to the black-box model and 2) is less unfair than this black-box model:

Original: [Equation]
Revised: [Equation]

Original: in which ϵ is an upper bound on the fairness gap of the interpretable model on the suing set
Revised: in which ✏ is an upper bound on the fairness gap of the interpretable model on the suing set

Original: FairnessGap ( I ; X sg ) , and FairnessGap ( · , · ) constitutes a measure of fairness gap according to some fairnessmetric – generically measured by the subpopulation gap in a fairness metric.
Revised: FairnessGap ( I ; X sg ) , and FairnessGap ( · , · ) constitutes a measure of fairness gap according to some fairness

Original: Note that ϵ <

Original: FairnessGap ( B ; X sg ) must be less than the fairness gap of the black-box model on the suing set

Original: [-FairnessGap ( B ; X sg ) to cause fairwashing violation γ > 
-]
Revised: [-FairnessGap ( B ; X sg ) to cause fairwashing violation γ > 
-]

Original: (see Definition 3). 4.2 Proposed method: FRAUD-Detect
Revised: 4.2 Proposed method: FRAUD-Detect

Original: As demonstrated in Section 3, a fairwashing detection method that relies on the difference over subpopulations in true-positive and false-positive rates of the interpretable model w.r.t the blackbox model ( ˜ δ and δ ′ respectively) is sufficient.
Revised: As demonstrated in Section 3, a fairwashing detection method that relies on the difference over subpopulations in true-positive and false-positive rates of the interpretable model w.r.t the blackbox model ( ˜  and  0 respectively) is sufﬁcient.

Original: Inspired by this theoretical analysis, we propose our fairwashing detector, FRAUD-Detect , which locates fairwashing violations.
Revised: Inspired by this theoretical analysis, we propose our fairwashing detector, FRAUD-Detect , which locates fairwashing violations.

Original: FRAUD-Detect aims to distinguish between honest and fairwashed interpretable models approximating the unfair black-box model B ( · ) as described in Section 2.2 under the additional following constraints: 1) without white-box access to the black-box and interpretable models as well as; 2) independently of the cooperation of the model provider being audited.
Revised: FRAUD-Detect aims to distinguish between honest and fairwashed interpretable models approximating the unfair black-box model B ( · ) as described in Section 2.2 under the additional following constraints: 1) without white-box access to the black-box and interpretable models as well as; 2) independently of the cooperation of the model provider being audited.

Original: These constraints are introduced to conform to realistic scenarios in which entities desire to retain the confidentiality of their black-box model due to intellectual property rights and because they considered it as a company’s asset.
Revised: These constraints are introduced to conform to realistic scenarios in which entities desire to retain the conﬁdentiality of their black-box model due to intellectual property rights and because they considered it as a company’s asset.

Original: Similarly to definitions of true-positive T + a and false-positive F + a rates of I ( · ) w.r.t B ( · ) on data with attribute a ∈ A
Revised: Similarly to deﬁnitions of true-positive T + a and false-positive F + a rates of I ( · ) w.r.t B ( · ) on data with attribute a 2 A

Original: = { 0 , 1 } , we define the true-negative rate, T − a , and false-negative rate, F − a as:
Revised: = { 0 , 1 } , we deﬁne the true-negative rate, T  a , and false-negative rate, F  a as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: (8) for a ∈ A = { 0 , 1 } .
Revised: (8) for a 2 A = { 0 , 1 } .

Original: Note that T − a = 1 − F + a and F − a = 1 − T + a .
Revised: Note that T  a = 1  F + a and F  a = 1  T + a .

Original: FRAUD-Detect computes the T + a , F + a , T − a , F − a for each subpopulation 
Revised: FRAUD-Detect computes the T + a , F + a , T  a , F  a for each subpopulation a 2 A

Original: a ∈ A = { 0 , 1 } using the predictions of the interpretable model and black-box model on the suing set
Revised: = { 0 , 1 } using the predictions of the interpretable model and black-box model on the suing set

Original: X sg .
Revised: X sg .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: be the (flattened) confusion matrices of the interpretable model w.r.t the black-box model on X sg with attribute 0 and 1 , respectively.
Revised: be the (ﬂattened) confusion matrices of the interpretable model w.r.t the black-box model on X sg with attribute 0 and 1 , respectively.

Original: Then, FRAUD-Detect computes the divergence between C 0 and Cusing Kullback–Leibler (KL) as:
Revised: Then, FRAUD-Detect computes the divergence between C 0 and Cusing Kullback–Leibler (KL) as:

Original: [Equation]
Revised: [Equation]

Original: Recall that we demonstrated in Section 3 that the divergence in T +0 and T +1 ( ˜ δ ) and F +0 and F +1 ( δ ′ ) were sufficient for a detection method.
Revised: Recall that we demonstrated in Section 3 that the divergence in T +0 and T +1 ( ˜  ) and F +0 and F +1 (  0 ) were sufﬁcient for a detection method.

Original: We can complement 2 our detection method with additional divergences in T − a and F − a and measure the dissimilarity via KL divergence.
Revised: We can complement 2 our detection method with additional divergences in T  a and F  a and measure the dissimilarity via KL divergence.

Original: This choice is natural as the KL divergence is commonly used to quantify the divergence over probability distributions—
Revised: This choice is natural as the KL divergence is commonly used to quantify the divergence over probability distributions—

Original: C a functions as a simplified representation of the probability distribution Y I | Y B , A = a .
Revised: C a functions as a simpliﬁed representation of the probability distribution Y I | Y B , A = a .

Original: For an honest interpretable model, C KL is generally relatively low ( ˜ δ, δ ′ ≈ 0 ) as the only divergence arises from general error in fidelity optimization and the fairness gap of the black-box model (as shown in Section 3).
Revised: For an honest interpretable model, C KL is generally relatively low ( ˜  ,  0 ⇡ 0 ) as the only divergence arises from general error in ﬁdelity optimization and the fairness gap of the black-box model (as shown in Section 3).

Original: In other words, the honest interpretable model approximates the black-box model equivalently across subpopulations.
Revised: In other words, the honest interpretable model approximates the black-box model equivalently across subpopulations.

Original: However, for a fairwashed interpretable model, C KL grows significantly, as the interpretable model is explicitly manipulated to optimize the fairness across subpopulations and improve over the black-box model.
Revised: However, for a fairwashed interpretable model, C KL grows signiﬁcantly, as the interpretable model is explicitly manipulated to optimize the fairness across subpopulations and improve over the black-box model.

Original: We quantify the distinction between honest and fairwashed interpretable models via a threshold ∆ > 0 on C KL such that if C KL > ∆ , the interpretable model is considered fairwashed.
Revised: We quantify the distinction between honest and fairwashed interpretable models via a threshold  > 0 on C KL such that if C KL >  , the interpretable model is considered fairwashed.

Original: Note here that due to the irreducibility result stated in Section 3, ∆ must be chosen such that ∆ is tightly greater than the irreducible term in order to properly distinguish between accidental fairwashing that occurs as a result of the irreducibility and malicious fairwashing.
Revised: Note here that due to the irreducibility result stated in Section 3,  must be chosen such that  is tightly greater than the irreducible term in order to properly distinguish between accidental fairwashing that occurs as a result of the irreducibility and malicious fairwashing.

Original: This procedure is detailed in Algorithm 1.
Revised: This procedure is detailed in Algorithm 1.

Original:  Evading FRAUD-Detect
Revised:  Evading FRAUD-Detect

Original: In this section, we investigate on whether a dishonest entity could evade FRAUD-Detect while performing fairwashing.
Revised: In this section, we investigate on whether a dishonest entity could evade FRAUD-Detect while performing fairwashing.

Original: We assume that this dishonest entity, which we call the adversary , is informed about FRAUD-Detect .
Revised: We assume that this dishonest entity, which we call the adversary , is informed about FRAUD-Detect .

Original: Namely, the adversary is aware of the fairwashing detection method and desires to fairwash while evading detection by the auditor.
Revised: Namely, the adversary is aware of the fairwashing detection method and desires to fairwash while evading detection by the auditor.

Original: To achieve this goal, the informed adversary has the objective of finding a fairwashed interpretable model that satisfies an additional constraint on the Kullback–Leibler (KL) divergence of the confusion matrices among subpopulations,
Revised: To achieve this goal, the informed adversary has the objective of ﬁnding a fairwashed interpretable model that satisﬁes an additional constraint on the Kullback–Leibler (KL) divergence of the confusion matrices among subpopulations,

Original: C KL in the previous section.
Revised: C KL in the previous section.

Original: To probe the robustness of our detector to an informed adversary empirically, we explore the range of fairness gap given a fixed value of fidelity and a fixed value of
Revised: To probe the robustness of our detector to an informed adversary empirically, we explore the range of fairness gap given a ﬁxed value of ﬁdelity and a ﬁxed value of

Original: C KL via solving the informed adversary optimization problem.
Revised: C KL via solving the informed adversary optimization problem.

Revised: Algorithm

Original: Input: Query access to the interpretable model
Revised: Input: Query access to the interpretable model

Original: I ( · ) , suing dataset X sg , Black-box model predictions on the suing set B ( X sg ) , sensitive attribute a ∈ A and a threshold ∆ > 0 .
Revised: I ( · ) , suing dataset X sg , Black-box model predictions on the suing set B ( X sg ) , sensitive attribute a 2 A and a threshold  > 0 .

Original: Output: (cid:26) T fairwashing is detected
Revised: Output: ⇢ T fairwashing is detected

Original: F fairwashing is not detected 1: ˆ Y I← I
Revised: F fairwashing is not detected 1: ˆ Y I

Original:  ( X sg ) 
Revised: I ( X sg ) .

Original: ▷ Query interpretable model 2: for i← 0 , 1 do 3:
Revised: Query interpretable model 2: for i 0 , 1 do 3:

Original: T + i← |{ ˆ y B =1 , ˆ y I =1 ,a = i }| |{ ˆ y B
Revised: T + i |{ ˆ y B =1 , ˆ y I =1 ,a = i }| |{ ˆ y B

Original: =1 ,a = i }| ▷ TPR 4:
Revised: =1 ,a = i }| . TPR 4:

Original: F + i← |{ ˆ y B =0 , ˆ y I =1 ,a = i }| |{ ˆ y B
Revised: F + i |{ ˆ y B =0 , ˆ y I =1 ,a = i }| |{ ˆ y B

Original: =0 ,a = i }| ▷ FPR 5:
Revised: =0 ,a = i }| . FPR 5:

Original: F − i← |{ ˆ y B =1 , ˆ y I =0 ,a = i }| |{ ˆ y B
Revised: F  i |{ ˆ y B =1 , ˆ y I =0 ,a = i }| |{ ˆ y B

Original: =1 ,a = i }| ▷ FNR 6:
Revised: =1 ,a = i }| . FNR 6:

Original: T − i← |{ ˆ y B =0 , 
Revised: T  i |{ ˆ y B =0 , ˆ y I =0 ,a = i }| |{ ˆ y B

Original: ˆ y I =0 ,a = i }| |{ ˆ y B
Revised: =0 ,a = i }| .

Original: =0 ,a = i }| ▷ TNR 7: C 0 ← [ T +0 , F +0 , T − 0 , F − 0 ] 8: C 1 ← [ T +1 , F +1 , T − 1 , F − 1 ] 9: C KL← KL ( C 0 , C 1 ) ▷ Kullback–Leibler 10: if C KL > ∆ then 11: return T 12: else 13: return F
Revised: TNR 7: C[ T +0 , F +0 , T  0 , F  0 ] 8: C[ T +1 , F +1 , T  1 , F  1 ] 9: C KL KL ( C 0 , C 1 ) . Kullback–Leibler 10: if C KL >  then 11: return T 12: else 13: return F

Original: Definition 5 (Informed Adversary Optimization) .
Revised: Deﬁnition 5 (Informed Adversary Optimization) .

Original: Given a black-box model B ( · ) , a suing set X sg , a sensitive attribute A , a loss threshold v and a fairwashing detection threshold ∆ , an informed adversary aims to learn an interpretable I ( · ) such that minimize
Revised: Given a black-box model B ( · ) , a suing set X sg , a sensitive attribute A , a loss threshold v and a fairwashing detection threshold  , an informed adversary aims to learn an interpretable I ( · ) such that minimize

Original: FairnessGap ( I ; X sg ) subject to
Revised: FairnessGap ( I ; X sg ) subject to

Original: L ( I ;X sg ) ≤
Revised: L ( I ;

Original: [Equation]
Revised: [Equation]

Original: where C KL is the KL divergence between the subpopulation confusion matrices of X sg .
Revised: where C KL is the KL divergence between the subpopulation confusion matrices of X sg .

Original: The differences between the fairwashing optimization in Definition 4, and the informed adversary optimization of Definition 5 are two fold.
Revised: The differences between the fairwashing optimization in Deﬁnition 4, and the informed adversary optimization of Deﬁnition 5 are two fold.

Original: First, the order of the maximand and the constraint is reversed.
Revised: First, the order of the maximand and the constraint is reversed.

Original: Both formulations are valid since each indicates priorities of the adversary, namely, whether to put a hard limit on the fidelity loss or on the fairness gap of the interpretable model.
Revised: Both formulations are valid since each indicates priorities of the adversary, namely, whether to put a hard limit on the ﬁdelity loss or on the fairness gap of the interpretable model.

Original: Second, in
Revised: Second, in

Original: Definition 5 we replaced the non-differentiable EmpiricalFidelity ( I, B ; X tr ) with a differentiable measure of loss L ( I ; X sg ) .
Revised: Deﬁnition 5 we replaced the non-differentiable EmpiricalFidelity ( I, B ; X tr ) with a differentiable measure of loss L ( I ; X sg ) .

Original: In our empirical evaluations, we use the logistic regression loss.
Revised: In our empirical evaluations, we use the logistic regression loss.

Original: To solve the above optimisation problem, we consider the Rashomon set of high-fidelity interpretable models and compute the range of the fairness gap of interpretable models.
Revised: To solve the above optimisation problem, we consider the Rashomon set of high-ﬁdelity interpretable models and compute the range of the fairness gap of interpretable models.

Original: In short, given a classification task, the Rashomon Set [41] is defined as the set of almost-equally-accurate models (interpretable models in our case).
Revised: In short, given a classiﬁcation task, the Rashomon Set [41] is deﬁned as the set of almost-equally-accurate models (interpretable models in our case).

Original: More precisely, given a model class F , a loss function L D ( · ) over a dataset D of interest, a reference model f ∗ ( e.g. , optimal model) and a performance threshold τ ∈ [0 , 1] , theRashomon
Revised: More precisely, given a model class F , a loss function L D ( · ) over a dataset D of interest, a reference model f ⇤ ( e.g. , optimal model) and a performance threshold ⌧ 2 [0 , 1] , the

Original: [Equation]
Revised: Rashomon set R s ( F , f ⇤ , ⌧ ) = { f 2 F | L D ( f )  L D ( f ⇤ ) + ⌧ } .

Original: We extend the Fairness in the Rashomon Set (FaiRS) algorithm [19] to compute the range of the fairness gap of interpretable models that can be generated over the set of high-fidelity models satisfying a constraint on the KL divergence.
Revised: We extend the Fairness in the Rashomon Set (FaiRS) algorithm [19] to compute the range of the fairness gap of interpretable models that can be generated over the set of high-ﬁdelity models satisfying a constraint on the KL divergence.

Original: FaiRS exploits the so-called Rashomon Effect [14], which is an empirical phenomenon resulting in multiple models displaying the same performance overall ( e.g. , w.r.t their global accuracy) but have significant differences in terms of their individual predictions [19].
Revised: FaiRS exploits the so-called Rashomon Effect [14], which is an empirical phenomenon resulting in multiple models displaying the same performance overall ( e.g. , w.r.t their global accuracy) but have signiﬁcant differences in terms of their individual predictions [19].

Original: In our setting, the Rashomon effect implies that several interpretable models can achieve the same high fidelity w.r.t the black-box model while displaying different values of fairness gap.
Revised: In our setting, the Rashomon effect implies that several interpretable models can achieve the same high ﬁdelity w.r.t the black-box model while displaying different values of fairness gap.

Original: Thus, computing the range of the fairness gap of high-fidelity interpretable models that satisfy a constraint on C KL quantifies the robustness of FRAUD-Detect to the informed adversary.
Revised: Thus, computing the range of the fairness gap of high-ﬁdelity interpretable models that satisfy a constraint on C KL quantiﬁes the robustness of FRAUD-Detect to the informed adversary.

Original: To the best of our knowledge, the FaiRS algorithm of [19] is the only work proposing a practical solution to efficiently characterize the range of the fairness gap over the Rashomon set.
Revised: To the best of our knowledge, the FaiRS algorithm of [19] is the only work proposing a practical solution to efﬁciently characterize the range of the fairness gap over the Rashomon set.

Original: Alternative approaches would require measuring the range of the fairness gap over approximations of the
Revised: Alternative approaches would require measuring the range of the fairness gap over approximations of the

Original: Rashomon set obtained by either generating models using several hyperparameter values ( e.g. , seeds [33, 21]) or brute-forcing over a particular class of model ( e.g. , depth seven decision trees [41]).
Revised: Rashomon set obtained by either generating models using several hyperparameter values ( e.g. , seeds [33, 21]) or brute-forcing over a particular class of model ( e.g. , depth seven decision trees [41]).

Original: This is not practical due to the high computational cost and may lead to sub-optimal results.
Revised: This is not practical due to the high computational cost and may lead to sub-optimal results.

Original:  Empirical validation
Revised:  Empirical validation

Original: FRAUD-Detect aims to detect fairwashing by distinguishing between a fairwashed interpretable model and honest interpretable one during an audit.
Revised: FRAUD-Detect aims to detect fairwashing by distinguishing between a fairwashed interpretable model and honest interpretable one during an audit.

Original: Additionally, FRAUD-Detect seeks to be robust
Revised: Additionally, FRAUD-Detect seeks to be robust

Original: 1 1 − (cid:15) .
Revised: 1 1   .

Original: C K L
Revised: C K L

Original: AB & DT .
Revised: AB & DT .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: XGB & DT .
Revised: XGB & DT .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: DNN & DT .
Revised: DNN & DT .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: RF & DT .
Revised: RF & DT .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: AB & LR .
Revised: AB & LR .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: XGB & LR .
Revised: XGB & LR .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: DNN & LR .
Revised: DNN & LR .

Original: 1 1 − (cid:15)
Revised: 1 1

Original: RF & LR .
Revised: RF & LR .

Original: P a r i t
Revised: P a r i t

Original: y ga p
Revised: y ga p

Original: o f
Revised: o f

Original: I ( · ) even against an informed adversary, i.e. , demonstrate the ability to detect fairwashing even when a dishonest entity is aware of FRAUD-Detect and attempts evasion.
Revised: I ( · ) even against an informed adversary, i.e. , demonstrate the ability to detect fairwashing even when a dishonest entity is aware of FRAUD-Detect and attempts evasion.

Original: Therefore, we empirically validate the performance of FRAUD-Detect in: 1) capturing a relationship between C KL and demographic parity gap of the interpretable model I ( · ) :
Revised: Therefore, we empirically validate the performance of FRAUD-Detect in: 1) capturing a relationship between C KL and demographic parity gap of the interpretable model I ( · ) :

Original: C KL increases as the demographic parity gap of the interpretable model decreases ( i.e. as fairwashing becomes more severe); 2) restricting the fairwashing capabilities of the informed adversary: incorporation of C KL constraint into fairwashing problem prevents favorable demographic parity gaps for the interpretable model, precluding fairwashing.
Revised: C KL increases as the demographic parity gap of the interpretable model decreases ( i.e. as fairwashing becomes more severe); 2) restricting the fairwashing capabilities of the informed adversary: incorporation of C KL constraint into fairwashing problem prevents favorable demographic parity gaps for the interpretable model, precluding fairwashing.

Original: We assess the performance of FRAUD-Detect using a diverse set of black-box architectures, interpretable models and datasets.
Revised: We assess the performance of FRAUD-Detect using a diverse set of black-box architectures, interpretable models and datasets.

Original: More precisely, we consider four architectures of black-box models:
Revised: More precisely, we consider four architectures of black-box models:

Original: Deep Neural Networks (DNN), AdaBoost (AB) [24], Gradient Boosted Decision Trees (XGBs) [17] and Random Forests (RFs) [13].
Revised: Deep Neural Networks (DNN), AdaBoost (AB) [24], Gradient Boosted Decision Trees (XGBs) [17] and Random Forests (RFs) [13].

Original: We evaluate the approach on three real-world datasets corresponding to critical decision systems: Adult Income [22], Bank Marketing [34] and COMPAS [5].
Revised: We evaluate the approach on three real-world datasets corresponding to critical decision systems: Adult Income [22], Bank Marketing [34] and COMPAS [5].

Original: We rely on Logistic Regression (LR) and Decision Trees (DTs) as our interpretable models due to their high performance and ease of explainability.
Revised: We rely on Logistic Regression (LR) and Decision Trees (DTs) as our interpretable models due to their high performance and ease of explainability.

Original: We refer to Appendix E for details on these datasets, black-box models, interpretable models and implementation of FRAUD-Detect .
Revised: We refer to Appendix E for details on these datasets, black-box models, interpretable models and implementation of FRAUD-Detect .

Original: FRAUD-Detect successfully detects fairwashing.
Revised: FRAUD-Detect successfully detects fairwashing.

Original: Figure 2 illustrates both the demographic parity gap and C KL of interpretable models as a function of ϵ .
Revised: Figure 2 illustrates both the demographic parity gap and C KL of interpretable models as a function of ✏ .

Original: The bigger the value of ϵ , the stronger the fairwashing (see Definition 4).
Revised: The bigger the value of ✏ , the stronger the fairwashing (see Deﬁnition 4).

Original: Any graph within the figures may be read from left to right as the degree of fairwashing increases.
Revised: Any graph within the ﬁgures may be read from left to right as the degree of fairwashing increases.

Original: As fairwashing becomes more severe, the demographic parity gap decreases while C KL increases such that we observe a relationship between the demographic parity gap of interpretable models and the associated C KL values.
Revised: As fairwashing becomes more severe, the demographic parity gap decreases while C KL increases such that we observe a relationship between the demographic parity gap of interpretable models and the associated C KL values.

Original: For example, decreasing the demographic parity gap of Decision Trees trained on a COMPAS dataset labelled by a RF black-box model by half, increases C KL by a factor of 4.
Revised: For example, decreasing the demographic parity gap of Decision Trees trained on a COMPAS dataset labelled by a RF black-box model by half, increases C KL by a factor of 4.

Original: The amount of change in C KL and in the demographic parity gap of fairwashed interpretable models differ across the dataset and black-box models due to the different original fairness of the black-box models.
Revised: The amount of change in C KL and in the demographic parity gap of fairwashed interpretable models differ across the dataset and black-box models due to the different original fairness of the black-box models.

Original: As C KL is sensitive to changes in fairwashing,
Revised: As C KL is sensitive to changes in fairwashing,

Original: FRAUD-Detect can be used to detect fairwashing by not permitting C KL to pass a certain threshold value.
Revised: FRAUD-Detect can be used to detect fairwashing by not permitting C KL to pass a certain threshold value.

Original: The C KL thresholds corresponding to 5% and 50% fairwashing are reported in Appendix G.
Revised: The C KL thresholds corresponding to 5% and 50% fairwashing are reported in Appendix G.

Original: Per-seed results are provided in Appendix H.
Revised: Per-seed results are provided in Appendix H.

Original: FRAUD-Detect is robust to the informed adversary.
Revised: FRAUD-Detect is robust to the informed adversary.

Original: Figure 3 shows the range of the demographic parity gap of high-fidelity fairwashed interpretable models subjected to a constraint on the C KL by solving the optimization problem of Equation (11).
Revised: Figure 3 shows the range of the demographic parity gap of high-ﬁdelity fairwashed interpretable models subjected to a constraint on the C KL by solving the optimization problem of Equation (11).

Original: We use four types of black-box models (AB, XGB, DNN and RF).
Revised: We use four types of black-box models (AB, XGB, DNN and RF).

Original: However, for simplicity, we performed the experiment using logistic regression as the interpretable model.
Revised: However, for simplicity, we performed the experiment using logistic regression as the interpretable model.

Original: The fairwashing detection thresholds were chosen from a wide range, i.e. ∆ ∈ { 0 . 01 , 0 .
Revised: The fairwashing detection thresholds were chosen from a wide range, i.e.  2 { 0 . 01 , 0 .

Original: 10 , 0 . 20 } (the constraint on C KL in Equation (11)).
Revised: 10 , 0 . 20 } (the constraint on C KL in Equation (11)).

Original: A direct implication for FRAUD-Detect is that using low fairwashing detection thresholds makes fairwashing difficult.
Revised: A direct implication for FRAUD-Detect is that using low fairwashing detection thresholds makes fairwashing difﬁcult.

Original: Consistently over all these results, imposing a constraint on the C KL significantly narrows the range of demographic parity gap of fairwashed interpretable models.
Revised: Consistently over all these results, imposing a constraint on the C KL signiﬁcantly narrows the range of demographic parity gap of fairwashed interpretable models.

Original: For instance, on COMPAS with ∆ = 0 . 2 , for an AB model that has demographic parity gap of 0 . 214 , fairwashing can produce a completely fair ( i.e. , demographic parity gap of 0 . 0 ) high-fidelity interpretable model while remaining undetected ( i.e. , C KL < 0 . 2 ).
Revised: For instance, on COMPAS with  = 0 . 2 , for an AB model that has demographic parity gap of 0 . 214 , fairwashing can produce a completely fair ( i.e. , demographic parity gap of 0 . 0 ) high-ﬁdelity interpretable model while remaining undetected ( i.e. , C KL < 0 . 2 ).

Original: However, when C KL decreases to 0 .
Revised: However, when C KL decreases to 0 .

Original: 1 , the fairest high-fidelity interpretable model obtained by the adversary exhibits a demographic parity gap of 0 . 107 .
Revised: 1 , the fairest high-ﬁdelity interpretable model obtained by the adversary exhibits a demographic parity gap of 0 . 107 .

Original: AB .65 .69 .

Revised: AB .65 .69 .

Original: XGB .7 .

Revised: XGB .7 .

Original: DNN .75 .78 .84 .86 .
Revised: DNN .75 .78 .84 .86 .

Original: RF 0 .0 .0 .0 .0 .1 .
Revised: RF 0 .0 .0 .0 .0 .1 .

Original: Fidelity 0 .0 .0 .0 .1 .
Revised: Fidelity 0 .0 .0 .0 .1 .

Original: P a r i t
Revised: P a r i t

Original: y ga p
Revised: y ga p

Original: o f I ( · ) ∆: .
Revised: o f I ( · )  : .

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: Incorporating fairness and explainability in decision systems can help foster trust [29] in them by characterizing failure modes ( e g. , unfairness) and providing assurances ( e.g. , fairness constraints)
Revised: Incorporating fairness and explainability in decision systems can help foster trust [29] in them by characterizing failure modes ( e g. , unfairness) and providing assurances ( e.g. , fairness constraints)

Original: Auditing provides an oversight component and helps avoid first-order failures such as unfairness.
Revised: Auditing provides an oversight component and helps avoid ﬁrst-order failures such as unfairness.

Original: In this sense, fairwashing can be seen as failure of auditing—a second-order failure that reduces trust in both fairness and explainability.
Revised: In this sense, fairwashing can be seen as failure of auditing—a second-order failure that reduces trust in both fairness and explainability.

Original: Second-order failures help characterize the limits of techniques designed to reduce the risks of first-order failures.
Revised: Second-order failures help characterize the limits of techniques designed to reduce the risks of ﬁrst-order failures.

Original: Our work is the first to delineate the theoretical limits of what is possible in auditing fairness and in containing the risk of fairwashing.
Revised: Our work is the ﬁrst to delineate the theoretical limits of what is possible in auditing fairness and in containing the risk of fairwashing.

Original: In practice, we demonstrated the possibility of performing fairwashing through joint optimization of fidelity and fairness constraints when providing explanations.
Revised: In practice, we demonstrated the possibility of performing fairwashing through joint optimization of ﬁdelity and fairness constraints when providing explanations.

Original: This could thwart auditing processes that are solely based on an analysis of the delivered interpretable model.
Revised: This could thwart auditing processes that are solely based on an analysis of the delivered interpretable model.

Original: We address this issue by proposing an additional auditing protocol that queries the interpretable model.
Revised: We address this issue by proposing an additional auditing protocol that queries the interpretable model.

Original: Finally, through both a theoretical framework and an experimental evaluation, we demonstrate that our fairwashing detector cannot be evaded by an attacker, informed of our detector, without significantly degrading the fidelity of model explanations.
Revised: Finally, through both a theoretical framework and an experimental evaluation, we demonstrate that our fairwashing detector cannot be evaded by an attacker, informed of our detector, without signiﬁcantly degrading the ﬁdelity of model explanations.

Original: Future directions include the extension of the detector to other fairness measures ( e g. , measures incorporating ground-truth information) as well as extending the analysis to multi-class setups.
Revised: Future directions include the extension of the detector to other fairness measures ( e.g. , measures incorporating ground-truth information) as well as extending the analysis to multi-class setups.

Original: However, these would both require additional assumptions on the data manifold (to account for class correlations) to establish similar results as in Section 
Revised: However, these would both require additional assumptions on the data manifold (to account for class correlations) to establish similar results as in Section 3.

Original: Acknowledgments
Revised: Acknowledgments

Original: This work was supported by CIFAR (through a Canada CIFAR AI Chair and a Catalyst grant), by
Revised: This work was supported by CIFAR (through a Canada CIFAR AI Chair and a Catalyst grant), by

Original: NSERC (under the Discovery Program and COHESA strategic research network), by the Ontario
Revised: NSERC (under the Discovery Program, and COHESA strategic research network), by the Ontario

Original: Early Researcher Award, and by gifts from Intel and Meta.
Revised: Early Researcher Award and by gifts from Intel and Meta.

Original: We also thank the Vector Institute’s sponsors.
Revised: We also thank the Vector Institute’s sponsors.

Original: Ali Shahin Shamsabadi was also partially supported by The Alan Turing Institute.
Revised: Ali Shahin Shamsabadi was also partially supported by The Alan Turing Institute.

Original: Sierra
Revised: Sierra

Original: Wyllie was also partially supported by Engineering Science at University of Toronto through ESROP grant.
Revised: Wyllie was also partially supported by Engineering Science at University of Toronto through ESROP grant.

Original: Ulrich Aïvodji was supported by the NSERC Discovery Grants program (2022-04006).
Revised: Ulrich Aïvodji was supported by the NSERC Discovery Grants program (2022-04006).

Original: Sébastien Gambs is supported by the Canada Research Chair program, a Discovery Grant from
Revised: Sébastien Gambs is supported by the Canada Research Chair program, a Discovery Grant from

Original: NSERC as well as the NSERC-RDC DEEL project.Statement
Revised: NSERC as well as the NSERC-RDC DEEL project.

Original: FRAUD-Detect may be used to provide evaluations of the fairness of model explanations, resulting in increased trust in explanations, public expectation of model fairness, and model owner accountability.

Original: Since FRAUD-Detect is non-cooperative and resists evasion even against an informed adversary, it empowers auditors, protects the intellectual property and ownership of the model owners, and may resist gamification into a new arms-race system between fairwashing and auditor detection methods.

Original: By empowering auditors, FRAUD-Detect also enforces democratic ideals of equality by providing an avenue for users to receive fairness-vetted model explanations for high-stakes decisions, further increasing public trust in AI and model auditing.

Original: The potential risks of this work revolve around the fairness definition and datasets used in this paper.

Original: Demographic parity, while befitting our setting (as discussed in Section 2.1) is not always an appropriate fairness definition, depending on the model’s domain and application.

Original: We recognise that continual focus on a concept reinforces it and it’s ideology’s normalcy, and that our focus on demographic parity also reinforces the political and philosophical ideologies underlying it and deprives alternate definitions of due attention.

Original: Likewise, as noted in Appendix E, our evaluation of

Original: FRAUD-Detect on the current popular fairness benchmark datasets normalizes these applications of

Original: AI.

Original: We draw specific attention to the use of the COMPAS recidivism prediction dataset [5], and note that we do not necessarily condone the use of AI models for this application.

Original: To mitigate these societal risks, we advocate for similar auditing measures for second-order failures with respect to other fairness definitions (see Section 7).

Original: We also acknowledge the importance of investigating the impact of FRAUD-Detect in real-world scenarios, especially where the group of users suing for model explanations represents minority or sensitive demographics.

Original: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 7.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] In the Societal Impact Statement (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption 1.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix D 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See supplemental material.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See supplemental material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention the license of the assets?

Original: [Yes] See supplemental material.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See supplemental material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [Yes] See supplemental material.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] See supplemental material.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]

Original: A Explanation methods

Original: Methods to facilitate the understanding of machine learning models are usually classified into two main categories: transparent-box design and black-box model explanation [27].

Original: Transparent-box design concerns methods that produce inherently interpretable models such as logistic regressors and decision trees [12, 32].

Original: Black-box model explanation corresponds to approaches designed to explain how black-box models produce their outcomes.

Original: For instance, global explanations techniques ( e.g. ,

Original: TREPAN [20] and DECTEXT [11]) focus on explaining the whole logic of black-box models by training an inherently interpretable surrogate model.

Original: On the other hand, local explanations aim at explaining black-box models decisions on single data points ( e.g. , LIME [38], and GRADCAM [40]).

Original: For simplicity, in this paper, we focus on fairwashing performed with global explanation models.

Original: B Related Work

Original: We highlight some of the differences in settings, assumptions and solutions between our work and [44, 25]:

Original: Setting We rely on explainer models while they use a published data subset for fairness auditing.

Original: In particular, they reveal some subset of the training data and their predictions.

Original: We follow the setting considered in the literature introducing fairwashing [2, 3].

Original: Slack et al.

Original: explanations require input perturbations while ours require the model owner to provide an interpretable model.

Original: Assumptions Slack et al.

Original: assume that fairness auditing is performed via model-agnostic local explanations (e.g., LIME and SHAP) [44].

Original: Both [44] and [25] assume an ideal detector with knowledge of the underlying training distribution of the model; and query access to the black-box model.

Original: However, we only assume access to black-box model predictions on the suing set.

Original: We stress that this dataset is available before any formal audit takes place (in the form of asking for model explanations).

Original: This does not constitute query access to the black-box model; in fact it is only dataset access, which here is the predictions on the suing set.

Original: Solutions [44, 25] attempt to detect fairwashing using a Kolmogorov-Smirnov (KS) test over the underlying and company-provided subset distribution to determine if the data subset was honestly provided.

Original: More precisely, they show that detection is difficult when fairwashing is conducted by minimizing the Wasserstein distance between the distributions while subject to fairness; this optimization also minimizes the upper-bound of the advantage (i.e., distinguishability) of the KS test.

Original: This differs from our setting in which an informed malicious company must optimize the fairness subject to both fidelity (loss) and the detection threshold.

Original: Explanation methods like LIME and SHAP perturb inputs to gauge feature relevance, inadvertently querying with synthetic data that may be detected with out-of-distribution detection.

Original: Queries determined to originate from explainers are fed to a fair model whereas in-distribution points are given to the biased model.

Original: In contrast, our detection method is non-cooperative (we require no additional information from the black-box model as the auditor already has the predictions on the suing set) and therefore does not rely on input perturbations.

Original: C Fidelity

Original: Definition 6 (Fidelity [20]) .

Original: The fidelity of an interpretable model I ( · ) with respect to a black-box model B ( · ) is defined as the relative accuracy of I ( · ) with respect to B ( · ) :

Original: [Equation]

Original: where D is a data distribution, ˆ Y I is the prediction of the interpretable model and ˆ Y B is the prediction of black-box model.

Original: ˆ Y I and ˆ Y B are assumed to be both binary-valued random variables.

Original: D Additional Proofs

Original: Below we iterate the full proof for Theorem 1, the main theorem in the paper.

Original: Proof.

Original: Define T + a , the true-positive rate of I ( · ) with respect to B ( · ) on X ∼ D a , in which D a denotes the distribution over data with attribute a ∈ A .

Original: Define F + a , the false-positive rate of I ( · ) with respect to B ( · ) on X ∼ D a :

Original: [Equation]

Original: Note that by definition, T + a , F + a ∈

Original: We denote ˜ δ, δ ′ > 0 as the difference in value between

Original: T +0 and T +1 , and F +0 and F +1 , respectively.

Original: [-Without loss of generality, let T +
-]
Revised: [-Without loss of generality, let T +
-]

Original: =

Original: T +0 + ˜

Original: δ

Original: and

Original: F + 1 = F + 0 + δ ′ where ˜ δ, δ ′ ∈ [0 , 1] .

Original: To characterize γ

Original: = Γ B − Γ I , we first express Γ I and Γ B in terms of ˜ δ , δ ′ , T +0 , T +1 , F +0 , and F +1 , and

Original: Y B | A .

Original: Recall from Definition 3 that we define Γ I :=

Original: Pr (cid:104)

Original: [-ˆ Y I = 
-]
Revised: [-ˆ Y I = 
-]

Original: | A = 0 (cid:105) − Pr (cid:104)

Original: [-ˆ Y I = 
-]
Revised: [-ˆ Y I = 
-]

Original: | A = 1 (cid:105) .

Original: We use Bayes’ formula to re-write Pr (cid:104)

Original: [-ˆ Y I = 
-]
Revised: [-ˆ Y I = 
-]

Original: | A = a (cid:105) , a ∈ A = { 0 , 1 } in terms of the aforemen- tioned variables.

Original: For a ∈ { 0 , 1 } :

Original: [Equation]

Original: Given our derivation above, we continue by substituting terms into Γ I :

Original: [Equation]

Original: Now that we have derived Γ I in terms of our desired terms and Γ B

Original: [Equation]

Original: we eliminate Γ I from the equation for γ :

Original: [Equation]

Original: Corollary 1.

Original: Let T +1 , F +1 as defined in Section 3.

Original: Then:

Original: [Equation]

Original: 1 > 0 .

Original: Proof.

Original: [-Note that F +1 , T +1 ∈ [0 , 1] , therefore 1 + F +
-]
Revised: [-Note that F +1 , T +1 ∈ [0 , 1] , therefore 1 + F +
-]

Original: − T +1 ≥ 0 .

Original: We will show 1 + F +1 − T

Original: [-+
-]
Revised: [-+
-]

Original: =

Original: if and only if T + = 1 and F + = 0 , i.e. if I ( · ) has perfect fidelity w.r.t.

Original: B ( · ) .

Original: [-We re-arrange 1 + F +1 − T +
-]
Revised: [-We re-arrange 1 + F +1 − T +
-]

Original: = 0 as:

Original: [Equation]

Original: Since F +1 , T +1 ∈ [0 , 1] and the size of interval [0 , 1] is 1 − 0 =

Original: 1 = T +1 − F +1 , T +1 and F +1 must be endpoints 1 and 0 , respectively.

Original: E Details on Experimental Setup of FRAUD-Detect

Original: In this section, we provide additional details on our experimental setup.

Original: Datasets.

Original: COMPAS [5] is a recidivism dataset released by ProPublica that includes data on 6 ,offenders in prison from Broward County (Florida), with the classification task consisting in predicting whether an inmate will re-offend in the two years after their release.

Original: 3 We consider race

Original: [Table]

Original: ( African-American , Caucasian ) as the sensitive attribute in COMPAS

Original: Adult Income [22] includes the

Original: The Adult Income binary classification task is to predict whether an individual’s income will exceed $50,000 a year.

Original: We use gender ( Female , Male ) as the sensitive attribute in Adult Income.

Original: Finally, Bank Marketing [34] constitutes information collected from a direct marketing campaign of a Portuguese banking institution between 2008 to 2013.

Original: This dataset contains 41,188 profiles of individuals, each described by 20 attributes with the classification task being the prediction of subscription to a term deposit.

Original: We consider age ( between 30-60 or not between 30-60 ) as the sensitive attribute in Bank Marketing.

Original: Training of black-box and interpretable models.

Original: For each experiment, the dataset is split randomly into three partitions: the training set (67%), the suing set (16.5%) and the testing set (16.5%), following the same procedure as in [3].

Original: Each experiment is repeated with 10 separate random seeds.

Original: The black-box models are learned from the training data points with their associated true labels.

Original: However, the interpretable models are trained from the suing set, which has been labelled by the black-box model.

Original: More precisely, we use the exponentiated gradient algorithm [1] subject to the fairwashing constraint from the FairLearn Library [9] to learn the fairwashed interpretable model.

Original: The exponentiated gradient algorithm when applied for fairness identifies the saddle point solution jointly maximizing the fairness and minimizing the loss.

Original: To learn the fairwashed interpretable model of the informed adversary, we extend the Fairness In The Rashomon Set (FaiRS) [19] framework by adding an additional constraint on C KL .

Original: FaiRS reduces the problem of minimizing the unfairness under two constraints on the fidelity and C KL to a saddle point problem, which is solved with the exponentiated gradient technique [1].

Original: Table 1 summarizes the performance of the considered black-box models on each suing set.

Original: Implementation.

Original: FRAUD-Detect solely requires the confusion matrices from the interpretable model for both the majority and minority group.

Original: These matrices are flattened, then compared via KL as outlined in Equation 10.

Original: Experiments were conducted on Intel Xeon Silver 4110 CPUs (2.10 GHz, 8GB RAM) and the KL divergence detector implemented in Python 3.8 using scikit-learn [36].

Original: Given that the KL divergence constraint is not a linear constraint on the confusion matrices, the extended version of FaiRS may fail to obtain the global minimum of the demographic parity gap on the set of high-fidelity interpretable models.

Original: In our experiments, our approach converges for two datasets, namely COMPAS and Marketing, but does not achieve the global minimum of the demographic parity gap for Adult Income.

Original: We are planning to address this in future work by focusing on detecting fairwashing using linear constraints.

Original: F Additional results

Original: FRAUD-Detect successfully detects fairwashing.

Original: Figures 4 and 5 illustrate both the demographic parity gap and C KL of interpretable models as a function of ϵ for two datasets (Adult income and

Original: Bank marketing), respectively.

Original: FRAUD-Detect is robust to the informed adversary.

Original: Figures 6 and 7 show the range of the demographic parity gap of high-fidelity fairwashed interpretable models subjected to a constraint on the C KL by solving the optimization problem of Equation (11).

Original: We use four types of black-box models (AdaBoost, XgBoost, DNN and Random Forest).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: C K L

Original: AdaBoost and DT

Original: XgBoost and DT

Original: DNN and DT

Original: RF and DT

Original: [Figure]

Original: C

Original: [Figure]

Original: RF

Original: [Figure]

Original: LR

Original: [Figure]

Original: [Figure]

Original: P a r i t y ga

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: P a

Original: [Figure]

Original: [Figure]

Original: G Fairwashing Detection Thresholds

Original: Table 2 reports several fairwashing detection thresholds observed in Figures 2, 4 and 5.

Original: H Per seed fine grain C KL versus fairness

Original: Figures 8, 9, and 10 show the per-seed results aggregated to form figures 2, 4, and 5, respectively.

Original: In these figures (especially in 8 and 10), the order of increasing black-box fairnesses across the 10 seeds is often replicated in the order of decreasing interpretable model C KL values across the same seeds; thus a seed with a low black-box fairness has higher C KL values than a seed with high black-box fairness.

Original: [Table]

Original: Some seeds from the logistic regression interpretable model (e.g. Figure 9 AdaBoost and logistic regression green and grey lines) show discontinuities at high epsilons where fairwashing is significant.

Original: There, the C KL values jump to infinity due to a division by zero in the KL log term when there are 0s in the minority subpopulation confusion matrix.

Original: We also observe that for several seeds across all datasets and models, the fairness of the honest interpretable model is imperfectly aligned and usually higher than the black-box fairness.

Original: This is unintentional fairwashing, which may arise from the relatively small capacity of the interpretable model compared to the black-box model.

Original: I Bounds for KL Divergence of the Confusion Matrices

Original: First, denote γ 1 = Γ B (1 + F +1 − T +1 )

Original: =

Original: Γ B ( F − 1 + F +1 ) and similarly γ 0 =

Original: Γ B ( F − 0 + F +0 ) .

Original: Depending on the context of the problem either subgroup could be the more sensitive subgroup, so in Equation (6) we implicitly assumed that subgroup is z = 1 : γ = γ 1 .

Original: Equation (10) defines the detector based on the flattened confusion matrix, but since for a binary classification problem, the rank of the confusion matrix is 2, it is enough to consider KL (cid:18)(cid:20) F +

Original: F −(cid:21) , (cid:20) F +

Original: F −(cid:21)(cid:19) .

Original: 4 We will derive upper and lower bounds on this divergence measure in terms of γ 0 , γ 1 , F + i , F − i ; i ∈ { 0 , 1 } .

Original: First, note that γ 1 is the sum of type I F − 1 and type II F +1 detection errors; scaled by the unfairness of the black-box model Γ B .

Original: In a detection problem, typically, we bound the type I error and minimize the type II error.

Original: The particular choice of type I and type II error is context-dependent (are false positives a bigger concern than false negatives or not?).

Original: In the context of fairwashing, a reasonable choice is that

Original: [Equation]

Original: That is, the cost of falsely detecting a fairwashed model is larger than missing fairwashing—due to the legal and liability considerations, for instance.

Original: We can bound the irreducible fairwashing gap γ i :

Original: [Equation]

Original: Using this we bound the aforementioned KL divergence:

Original: [Equation]

Original: Similarly,

Original: KL

Original: [Equation]

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: κ :=

Original: KL

Original: [Equation]

Original: is bounded from below and above by:

Original: [Equation]

Original: Equivalently

Original: [Equation]

Original: we can re-write these bounds in terms of F + and T + to match Theorem 1 (using the fact that F − = 1 − T + ):

Original: [Equation]

Original: [Equation]

Original: Therefore any choice between κ min := F +0 log (cid:16) F +0 F − 1 · γ 0 γ

Original: [Equation]

Original: and κ max :

Original: = F − 0 log (cid:16) γ

Original: [Equation]

Original: 1 · F − 0 F +1 (cid:17) is valid.

Original: κ = κ min is the most conservative choice which increases the risk of false discovery (of fairwashing), while κ = κ max is a permissive choice that minimizes the risk of false discovery at the expense of increase in the false negatives (not detecting fairwashing).

Original: A balanced choice for threshold could be the intermediate point:

Original: [Equation]

Original: [Equation]

Original: where ∆( R ) =

Original: R 0 /R 1 and ∆ l ( R ) = log ∆( R ) .

Original: In the last equation, we have used the taylor expansion of log(1 − x ) at x = 0 .

Original: J Experimental results with two independent variables

Original: In this section, we repeat experiments shown in Section 6 with the following adjustment:

Original: [Equation]

Original: in which we use only two independent variables of true-positive rate T + and false-positive rate F + .

Original: Figures 11, 12 and 13 show similar results as Figures 2, 4 and 5.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: K Perfect Fairwashing is impossible

Original: Theorem 2.

Original: Consider binary-valued random variables ˆ Y B black-box model predictions, ˆ Y I proxy (interpretable) model predictions and A a sensitive attribute; then these conditions cannot be satisfied simultaneously:

Original: Interpretable Model Demographic Parity

Original: [Equation]

Original: Non-detectable

Original: [Equation]

Original: Proof.

Original: Let Equation (41) be equal to T + , and Equation (42) to F + .

Original: Define the base rates of the black-box model on subpopulations 0 and 1 as:

Original: [Equation]

Original: Using the rule of total probabilities, for a ∈ [0 , 1] ,

Original: [Equation]

Original: Fairness according to parity requires that

Original: [Equation]

Original: Substituting Equation (43) and Equation (45) and re-organizing we have:

Original: [Equation]

Original: [Equation]

Original: Case 1: If T + = F + , by eliminating both and recognizing that (1 − µ 0 ) =

Original: 1 − µ 0 , we see that the equality Equation (45) reduces to 0 = 0 which is trivially correct.

Original: Case 2: T + ̸ = F + Simplifying Equation (46) further and eliminating T + − F + ̸ =

Original: [Equation]

Original: But these are probabilities, for Equation (45) to hold we must have that

Original: [Equation]

Original: which means that the black-box predictions exclusively depend on the sensitive attribute.

Original: In other words, for demographic parity to hold for the interpretable model, the black-box models predictions all samples of one subpopulation A = 0 and reject the others A = 0 .

Original: Clearly, this is unfair to one subpopulation.

Original: To establish the impossibility result, we further show that such an assumption is equivalent to requiring completely inaccurate black-box models with maximum error rates.

Original: Consider the “true” base rates for each subpopulation:

Original: [Equation]

Original: If interpretable demographic parity Equation (47) holds then, these base rates reduce to:

Original: [Equation]

Original: In other words, the black-box model predictions is completely independent of the true label and exclusively dependent on the sensitive attribute A .

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: i t

Original: [Figure]

Original: ga

Original: [Figure]

Original: o f

Original: I ( · )

Original: [Figure]

Original: 1 Fairness .

Original: C K L

Original: AdaBoost and LR .

Original: 1 Fairness .

Original: DNN and LR .

Original: 1 Fairness .

Original: RF and LR .

Original: 1 Fairness .

Original: XgBoost and LR .

Original: 1 Fairness .

Original: C K L

Original: AdaBoost and DT .

Original: 1 Fairness .

Original: DNN and DT .

Original: 1 Fairness .

Original: RF and DT .

Original: 1 Fairness .

Original: XgBoost and DT

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: DNN and LR .

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: consider of the

Original: [Figure]

Original: [Figure]

Original: [Figure]