Paper: Beyond Demographic Parity: Redefining Equal Treatment (ICLR-2024)

Reviewer comment (Theory):
Because the notion of equal treatment is strictly stronger than the demographic fairness notion, it can be much more challenging to attain ET in practice than DP. Moreover, the trade-off between fairness and accuracy may make ET less suitable for real applications. While the paper has compared the two notions, it is still not convincing why equal treatment is a superior notion. It is helpful if authors can provide more justification with a real example.

Author response:
A.3 The main philosophical critique to the liberalism notion of equal of treatment as blindness, is that it is rarely achieved. Our paper confirms this classical intuition both, theoretically and experimentally. 

In Figure 3, we provide cases where Demographic Parity is achieved but Equal Treatment is not, these are cases where demographic parity may flag wrongly a fair ML model. This figure is further extended in table 5 where we compare against distinct methods to measure demographic parity.

Reviewer comment (Presentation):
The paper is not using the ICLR template.

Author response:
We use the following in the latex preamble: \usepackage{iclr2024_conference}

Reviewer comment (Theory):
It seems that the sources of unequal treatment can only be explained for linear models (as illustrated in Example 4.4 and Figure 3). How can the method be generalized to non-linear cases?

Author response:
A.2. The approach is not restricted to linear models only. For instance, in Appendix E.3 non-linear models are experimented with. If the ET Inspector is linear, then the explanations are readily available, but, in general, the explanation of a non-linear ET inspector is proposed in the process from Figure 1.

Reviewer comment (Evaluation):
Since ET can be much more difficult to achieve, can you provide a real example to illustrate why equal treatment is a more appropriate notion than demographic parity?

Author response:
A.3 The main philosophical critique to the liberalism notion of equal of treatment as blindness, is that it is rarely achieved. Our paper confirms this classical intuition both, theoretically and experimentally. 

In Figure 3, we provide cases where Demographic Parity is achieved but Equal Treatment is not, these are cases where demographic parity may flag wrongly a fair ML model. This figure is further extended in table 5 where we compare against distinct methods to measure demographic parity.

Reviewer comment (Theory):
I disagree with some statements that the paper used to motivate the research question. For example, the abstract states “Related work in machine learning has translated the concept of equal treatment into terms of equal outcome and measured it as demographic parity (also called statistical parity)”. To my understanding, it is well-recognized in the fair ML literature that equal treatment and equal outcome are different concepts. While I agree that equal outcome is often measured with statistical measures, I think it is inaccurate to frame equal outcome as a convenient proxy of equal treatment. Instead, one simplified interpretation of equal treatment is “fairness through unawareness” or “colorblindness”. Rather than relying on the distinction between ‘equal outcome vs. equal treatment’, which can refer to much more high-level philosophical differences than what is captured in this paper, I would find it clearer to simply focus on equal treatment (new definition) vs. demographic parity.

Author response:
A.5.1 We agree with the reviewer. We will compare equal treatment vs demographic parity instead of equal outcomes.

Reviewer comment (Theory):
In Section 4, the theoretical analysis relies on assuming exact calculations of Shapley value are available. How realistic is this assumption in practice? When we do not have access to exact Shapley values, how will the theoretical results be affected?

Author response:
I.2. The mathematical derivation of Shapley values in real cases, (non-iid data and non linear model), is not feasible. There are related works that focus on to improve the computation of Shapley values in those scenarios:

[1] Sebastian Bordt, Ulrike von Luxburg: From Shapley Values to Generalized Additive Models and back. pdf. AISTATS, 2023

[2] Interventional shap values and interaction values for piecewise linear regression trees, AAAI 2023

Reviewer comment (Evaluation):
What are other explanation functions that can be used in the framework? In the appendix, another example is given, but I wonder is there a large set of options or is designing an effective explanation function an open question itself? If there are multiple candidate explanation functions, what makes one function better than another?

Author response:
A.3. In this work, we have focused on feature attribution explanations that are well-developed for tabular data. We have selected LIME and Shapley because they comply with the efficiency property (equation 6), and this is required so the theoretical analysis hold. It is possible that alternative XAI explanation techniques, such as logical reasoning, argumentation, or counterfactual explanations, could be useful and offer their unique advantages. This remains a further work to explore. (This issue is discussed in the “Limitations” paragraph of the Conclusions.)

Reviewer comment (Theory):
Concern about the practical implications of the new equal treatment definition when demographic parity is violated only by a small amount, which is the common case in practice since no useful algorithm has demographic parity exactly zero. Most decision-making algorithms have a small violation tolerance. Question on how equal treatment can be used for bounding demographic parity, or whether any relationship exists between them. This scenario is important in contexts such as credit lending; in the U.S., the Equal Credit Opportunity Act enforces no discrimination on outcomes. From a legal standpoint, differences in explanations may not matter as long as outcomes are sufficiently similar (low demographic parity).

Author response:
A.1  Thank you for this comment. Lemma 4.2 shows that Equal Treatment implies Demographic Parity in the case of perfect independence. Apart from the trivial case where the turns out to coincide (Lemma 4.1), we have not explored in the paper bounds on DP given bounds on ET. This is something that we intend to do in future work, as it is a non-trivial property relating multivariate independence (ET) to univariate independence (DP). A promising approach will be to exploit the linearity property of Shapley values.

Reviewer comment (Novelty):
Using AUC as a test statistic for classifier-two-sample test (C2ST) is not novel, as shown in [1], though it is a well-behaved statistic. Introducing a C2ST into the framework results in: (i) a data-driven algorithm to judge the level of equal treatment, which in practice is not necessarily low-sample-size friendly (since it does not use permutations), is affected by the complexity of the classifier, and produces varying results depending on classifier choice (as explored in the Appendix); (ii) an additional notion of uncertainty in the fairness definition, since the null hypothesis is rejected with a probability threshold rather than yielding a deterministic number as in demographic parity. As a result, the guarantee is now probabilistic (“up to a level 1-α” equal treatment). Such a probabilistic fairness definition could be inadmissible in legal contexts like credit lending, raising concerns about the practical feasibility of the proposed definition.

Author response:
Dear reviewer,

We appreciate your point bringing to our attention the novelty of using AUC as a C2ST test and the relevant paper from 2023. We sincerely apologize for any oversight on our part and we will promptly update the manuscript to incorporate this information.

Regarding your comments, we would like to clarify that in Appendix E.1, we have presented statistical testing with the necessary parameters, and in Appendix E.5, we have included additional experiments using bootstrapping to address the raised concerns. We hope these sections sufficiently address the points you raised.

Reviewer comment (Writing):
The term "natural data" sounds unusual; in the machine learning community, "real data" is more commonly used.

Author response:
A.4 The opposite to real data, is unreal data. While the contrary of natural data, is synthetic. We made this choice as we believe it's an improvement wrt terminology precision.

Reviewer comment (Novelty):
[Major] The paper omits a critical discussion on related work. There appears to be a study, specifically on individual fairness [1], that resonates with the motivations of this paper. Individual fairness emphasizes that individuals with similar backgrounds (e.g., salary, job status) should receive similar treatment. However, this paper does not draw any comparisons with its own concept of fairness to that of individual fairness.

Author response:
> I.2[Major] The paper omits a critical discussion on related work. There appears to be a study, specifically on individual fairness [1], that resonates with the motivations of this paper. 

A.2 Individual fairness can not often be applied and is therefore not a popular metric [2]– unlike group fairness metrics. For example, when a company hires for one position, it obviously has to discriminate even if two individuals are identical wrt non-protected characteristics. Talking about unfainress is not  plausible in this situation. Only if the company shows a systematic, statistically significant pattern of discriminating a group of people, one can talk about unfairness. Some Individual fairness metrics may have the challenge of achieving a precise mathematical definition of “similarity”. See [1] for a summary of critics on fairness metrics.

[1] Salvatore Ruggieri, José M. Álvarez, Andrea Pugnana, Laura State, Franco Turini:
Can We Trust Fair-AI? AAAI 2023

[2] Fleisher, W. What's fair about individual fairness?. AIES 2021

Reviewer comment (Presentation):
[Medium] The motivation presented within the paper is somewhat unclear, and it is concerning that significant discussions related to the work are relegated to the appendix. This decision diminishes the visibility and importance of such discussions.

Author response:
> I.3[Medium] The motivation presented within the paper is somewhat unclear, and it is concerning that significant discussions related to the work are relegated to the appendix. This decision diminishes the visibility and importance of such discussions.

A.3. We will adapt the introduction based on the feedback of the reviewers, but, due to space constraints, we decided to privilege the main contributions in the main text, and the detailed discussions in the appendix.

Reviewer comment (Theory):
[Medium] I cannot agree with the authors that equal opportunity could lead to reverse discrimination and overcorrection. As far as I know, equal opportunity is proposed to address these limitations you mentioned which suffered by demographic parity. Can you cite the corresponding works that draw this conclusion?

Author response:
> I.4 [Medium] I cannot agree with the authors that equal opportunity could lead to reverse discrimination and overcorrection.

A.3 See first answer for clarification (A.1). 

From a technical perspective, Equal Opportunity requires labeled data which in practice, which after deployment is hard to obtain. Equal Opportuntiy forces to meet some quotas (True Positive Rate), this introduces a second order discrimination. Equal Treatment has no quotas, it is fine to “hire” the best, while the decision is taken independently from the protected attribute.

From a philosophical perspective much is discussed on the literature, the commentary at https://link.springer.com/content/pdf/10.1057/palgrave.cpt.9300060.pdf about the book [1] provides some hints.



[1] Matt Cavanagh, Against Equality of Opportunity. Clarendon Press, 2002.

Reviewer comment (Reproducibility):
When testing the operationalized tool `explanationspace`, the reviewer found that installing `explanationspace` from https://pypi.org/project/explanationspace/#description worked, whereas the provided installation step in the documentation (https://explanationspace.readthedocs.io/en/latest/installation.html) did not work.

Author response:
A.1 First of all, we would like to thank the reviewer for trying out the code. We really appreciate it. 
We have now made a software release `explanationspace==0.0.2, addressing the reviewer’s concerns. 

We would like to distinguish between the open-source software, which we will disseminate and maintain, and the repository  of the experimental results of this paper. 

We have now fixed the mentioned issues so the installation and tutorial of https://explanationspace.readthedocs.io/en/latest/index.html
works accordingly.

Reviewer comment (Presentation):
The paper’s structuring makes it hard to appreciate the contributions, as crucial and informative content (better experiments, presentation, explanation of results, and descriptions of explanation functions) is relegated to the appendix instead of the main body.

Author response:
A.2 Due to space constraints, we decided to privilege the main contributions in the main text, and the detalled discussions in the appendix - bade based on what we believe it is more adequate to ICLR. We are open to any restructuring suggestions.

Reviewer comment (Novelty):
In the introduction and section 2, several introduced ideas are not well connected or explained. There are so many ideas that it is easy to miss the main point. The paper is oriented towards using explanation methods (SHAP and LIME) to investigate disparities in feature importance across protected groups, but there is insufficient related work and problem background provided. There are also significant similarities with works such as "Model Explanation Disparities as a Fairness Diagnostic" (https://arxiv.org/pdf/2303.01704.pdf) and "Explanability for fair machine learning" (https://arxiv.org/pdf/2010.07389.pdf) that are not discussed.

Author response:
A.3. Both papers are distinct from ours. They do not focus on the intersection of equality notions and ML, nor on the techniques used (Classifier Two Sample Test), proposing methods that don’t have the theoretical background solidity of our approach.  

We will be happy to acknowledge the recent work reported in the two arxiv papers. To the best of our knowledge, the first paper is still under review, while the second, we are unaware of its publication status. However, we hope that their absence in the submission is not held against us.

Reviewer comment (Theory):
Some proofs and examples in the methodology seem limited and do not explore corner cases. For example, statistical independence of Z from the explanation of features is a necessary but not sufficient condition for statistical independence of the model from Z. In example 4.3, since feature X₃ is not statistically independent of Z and the function is linear, the proof is straightforward by zeroing out that feature. However, in most real cases, the model may not be linear, and feature relationships could be complex with hard-to-uncover causal graphs. It may be better to position the tool as a diagnostic rather than a fix.

Author response:
A.4 This basic example serves us to compare with the benefits of measuring bias on the input data.
We agree that in most cases the model might not be linear. It's in this situation where the Shapley values become of use, we have studied non-lineal models in Appendix E.2 and E.3
Often in tabular data problems, ML models are fed with many more features than the model actually use. This method helps to deal with this problem.

Reviewer comment (Experiments):
Given that one might not have access to test data, the reviewer questions whether it would be better to apply the ET inspector as a diagnostic on the train/validation data instead.

Author response:
A.5 We agree with the reviewer on this point. ET inspector can be used to detect bias (in train/val data), and to monitor bias in the absence of labeled data. We will point out this in the paper.

Reviewer comment (Evaluation):
For novelty, the authors use AUC rather than accuracy in their C2ST, instead of accuracy as done previously. This tradeoff means that while scale invariance may be beneficial, it can be detrimental when inspecting fairness cases where one may care more about false positives than false negatives.

Author response:
A.8. Even if we use AUC, other metrics are also possible as far as a test statistics is available (e.g., accuracy in Lopez-Paz and Oquab 2017)

Reviewer comment (Theory):
Reliance on Z as a binary variable is restrictive, especially given the existence of many intersectionalities.

Author response:
A.9 The inspector $g$ can be a multi-class classifier. In the experiment of Figure 3, there are several ethnicities. In this case we have used a 1vsAll approach, further methods can be extended even to continuous protected attributes using regression instead of classification