Paper: A Logical Framework for Verification of AI Fairness (ICLR-2024)

Reviewer comment (Evaluation):
The paper does not argue why this measure of fairness is valuable. For instance, it is unclear to me what happens in the case when a dataset we start from is biased on its own. There should be a way to alleviate this by sacrificing model accuracy but the authors do not discuss this nor they say how data bias affects their fairness measure.

Author response:
1.How is your fairness metric affected by dataset (not model) bias? 
     
- We have showcased the correlation between accuracy and fairness resulting from our methodology in Appendix F on page 17 (Figure 8).  We have added an explanation as follows: “Figure 8 illustrates the relationship between accuracy and fairness calculated across 5 distinct datasets. Each bar in the figure corresponds to the accuracy of a specific dataset, while the connecting lines depict the fairness values. The result suggests a correlation where fairness tends to align with accuracy.” (Appendix F, Para 1)Our findings suggest a close relationship between accuracy and fairness when employing our approach to detect bias within an AI model. While dataset bias often necessitates a tradeoff between accuracy and fairness, our current study predominantly addresses model bias rather than dataset bias. However, we acknowledge the importance of investigating dataset bias as a potential future avenue, as highlighted in the second paragraph of our conclusion “ Adding to this, the study can be extended to investigate accuracy- fairness tradeoff in case of dataset bias”.

Reviewer comment (Novelty):
The paper fails to relate with the state of the art in fairness analysis, including previous works on the use of logic. For example, these papers and references therein:  
[A] Alexey Ignatiev, Martin C. Cooper, Mohamed Siala, Emmanuel Hebrard, João Marques-Silva: Towards Formal Fairness in Machine Learning. CP 2020: 846-867  
[B] Ulrich Aïvodji, Julien Ferry, Sébastien Gambs, Marie-José Huguet, Mohamed Siala: FairCORELS, an Open-Source Library for Learning Fair Rule Lists. CIKM 2021: 4665-4669  
[C] Julien Ferry, Ulrich Aïvodji, Sébastien Gambs, Marie-José Huguet, Mohamed Siala: Improving fairness generalization through a sample-robust optimization method. Mach. Learn. 112(6): 2131-2192 (2023)

Author response:
2. How does your work relate with state of the art in logic-based fairness analysis? 

- It may be noted that  current logic-based fairness analysis evaluates the fairness of models through sensitive attributes 
(Counterfactual fairness, FTU). In our work, we emphasize statistical measures for this purpose. Please find the below-mentioned 
references:
  1. Alexey Ignatiev, Martin C. Cooper, Mohamed Siala, Emmanuel Hebrard, João Marques-Silva: Towards Formal Fairness in 
   Machine Learning. CP 2020: 846-867
  2. Kusner, Matt J., Joshua Loftus, Chris Russell, and Ricardo Silva. "Counterfactual fairness." Advances in neural information 
    processing systems 30 (2017).

Reviewer comment (Evaluation):
In my view, the presented experimental results are rather weak - the speedup on N milliseconds compared to M milliseconds does not look important. The authors argue that they reduce the number of function calls but I fail to see why this is significant to them if the performance of the tool is only slightly better than that of the competitor. There is no discussion / comparison of the proposed metric and the corresponding approach in terms of the quality of the produced fairness assessment.

Author response:
3. What function calls are meant here? Why are they important? 
   
- We have added clarification of this matter in (section 4 page 8 para 2)."Here the number of recursion or function calls can be crucial in assessing a model's performance for a few reasons. Firstly, it indicates the computational load and efficiency of the model. A high number of recursive calls can suggest increased computational complexity, potentially leading to longer processing times or resource- intensive operations." Additionally, we are providing references that emphasize the need for function calls and explain the importance of employing 
optimization methods.  These references are mentioned in the draft also.

   1. Ousterhout JK. Optimizing Program Performance

   2. Grzeszczyk, M.K., 2018. Optimization of Machine Learning Process Using Parallel Computing. Advances in Science and 
       Technology. Research Journal, 12(4)

   3. Nima Asadi, Jimmy Lin, and Arjen P De Vries. Runtime optimizations for tree-based machine
       learning models. IEEE Transactions on Knowledge and Data Engineering, 26(9):2281–2292, 2013.

Reviewer comment (Novelty):
How does your work relate with state of the art in logic-based fairness analysis?

Author response:
2. How does your work relate with state of the art in logic-based fairness analysis? 

- It may be noted that  current logic-based fairness analysis evaluates the fairness of models through sensitive attributes 
(Counterfactual fairness, FTU). In our work, we emphasize statistical measures for this purpose. Please find the below-mentioned 
references:
  1. Alexey Ignatiev, Martin C. Cooper, Mohamed Siala, Emmanuel Hebrard, João Marques-Silva: Towards Formal Fairness in 
   Machine Learning. CP 2020: 846-867
  2. Kusner, Matt J., Joshua Loftus, Chris Russell, and Ricardo Silva. "Counterfactual fairness." Advances in neural information 
    processing systems 30 (2017).

Reviewer comment (Writing):
What function calls are meant here? Why are they important?

Author response:
3. What function calls are meant here? Why are they important? 
   
- We have added clarification of this matter in (section 4 page 8 para 2)."Here the number of recursion or function calls can be crucial in assessing a model's performance for a few reasons. Firstly, it indicates the computational load and efficiency of the model. A high number of recursive calls can suggest increased computational complexity, potentially leading to longer processing times or resource- intensive operations." Additionally, we are providing references that emphasize the need for function calls and explain the importance of employing 
optimization methods.  These references are mentioned in the draft also.

   1. Ousterhout JK. Optimizing Program Performance

   2. Grzeszczyk, M.K., 2018. Optimization of Machine Learning Process Using Parallel Computing. Advances in Science and 
       Technology. Research Journal, 12(4)

   3. Nima Asadi, Jimmy Lin, and Arjen P De Vries. Runtime optimizations for tree-based machine
       learning models. IEEE Transactions on Knowledge and Data Engineering, 26(9):2281–2292, 2013.

Reviewer comment (Writing):
Section 2.1 introducing the diagrams is not clear. Please expand, including formal definitions and (especially) informal examples. Just adding the note that this is equivalent to monadic FOL with equality would be really helpful. I expect that this is not going to be familiar to most (including myself); I had to consult external references, and this should really be self-contained.

Author response:
- We have added "Spider diagrams are higher-order representations of Venn diagrams and are equivalent to monadic FOL with 
       equality" in the draft (subsection 2.1, para 1).

Reviewer comment (Novelty):
Can the authors simply use first-order logic instead? This is going to be familiar to a lot more readers. I do not understand what about the approach relies on spider diagrams specifically. E.g., is it claimed that they are more intuitive? Then there should be an example showing how they add to that. I saw that Appendix E just uses Venn diagrams, there is no need to add spiders or anything else.

Author response:
- We have included a demonstration that shows the working of our method using a toy example in Appendix B  page 15 (Figure 7).

Reviewer comment (Writing):
Page 3: "where each instance is a tuple ..." In the tuple, "yhat in 0, 1" should be "yhat in {0, 1}".

Author response:
- Thank you for bringing this to our attention. We have changed in the draft.

Reviewer comment (Theory):
Theorems are almost trivial, and I don't see any "verification" to be honest. Overall I have strong doubts about the correctness of the approach, let alone significance of the results.

Author response:
2. Theorems are almost trivial, and I don't see any "verification" to be honest 
  
- Please note that understanding the theorems becomes more straightforward when there's a clear understanding of spider diagrams, their application in visualizing bias, and confirming fairness. We perceive it as a strength rather than a weakness.

Reviewer comment (Writing):
In the abstract, it is not clear what is meant by actual outcome and expected outcome. Specifically, the phrases used are "actual outcome of the algorithm" and "expected outcome of the model". Is the actual outcome determined by an algorithm, or does it refer to some ground truth about the instance (say, determined by a target function)? It is actually not clear what is meant by model here. Is it a set of hypotheses, or is it a single hypothesis function? What is meant by expected outcome of the model? Does the model describe a probability distribution over possible labels? Or is it that depending on the available dataset (generated from some input distribution), a different function is learned? What is meant by algorithm and model here? How are they different? By model, do you mean the nearest-neighbor / similarity based method described in Section 2, Page 3?

Author response:
We have changed the phrase to "This framework compares the sets representing the actual outcome of the model and the expected outcome to identify bias in the model" (Abstract). "The expected outcome (ground truth) of the model is obtained by considering the similarity score between the individuals (how much alike the elements are, calculated based on the distance between the values of corresponding features). The actual outcome is the outcome given by the model's prediction" (Section 1 para 3)”. Model in this paper refers to a classification model and the definition is given in section 2 para 1.

Reviewer comment (Writing):
I suggest changing the notation of the set of expected outcomes, as \mathbb{E} is typically used to represent the expected value of a random variable.

Author response:
We have changed the notation to "\mathbb{G}".

Reviewer comment (Writing):
Page 2, para 2: "compares the set of expected outcome E to the set of actual outcome A". Do you mean to use the singular or plural here? e.g. set of expected outcomes. In a binary classification task, what are E and A?

Author response:
We have changed to  " To verify fairness in the model, the framework compares the set of the expected outcomes ("\mathbb{G}) to the set of the actual outcomes ("\mathbb{A})" (section 1 para 3). "In this paper, we introduce the notations \mathbb {G} and \mathbb{A} to denote the sets of expected and actual outcomes respectively, produced by an AI model". (section 2 para 1)

Reviewer comment (Writing):
Page 2, Section 2: "two groups of output"? Do you mean protected attribute values? Typically the output refers to the predicted label. What am I missing?

Author response:
The term “two groups of output” was initially used to represent two demographic groups in the model. In the revised draft, we have changed to "In this study, we use an AI model defined by Das & Rad (2020) with two demographic groups (subpopulation)—non-protected (α) and protected (ω)—based on the sensitive attribute(s)" for better clarity (Section 2 para1).

Reviewer comment (Writing):
Page 3, para 1: "protected groups are advantaged ..." Is this an assumption, or a requirement? It is not clear what is meant here.

Author response:
In an AI model, based on case studies such as COMPAS and Amazon’s recruitment algorithms, there exist two or more demographic subpopulations that can influence a model’s outcomes. The group for which the model's outcome aligns favorably is termed the "protected group," while the group where the outcome is unfavorable or doesn't align with the ground truth is termed the "non-protected group." In this scenario, we can describe the protected groups as benefiting from the model's predictions. This is an assumption that aligns with past experiences. (Section 2 para 1) (Case studies are referenced in the introduction of this work (Section 1 para 1)).

Reviewer comment (Writing):
Page 3, para 2: "generator": Is denoted by p, but then does not appear in Eq. (1). Is Q_1 the same as p? The sensitive attribute is denoted s, but then is not mentioned or discussed later in the paper.

Author response:
The symbol 's' denotes the sensitive attribute.  To avoid confusion, the sentence is rephrased as follows:" Let $a_1, a_2,\ldots,a_m$ be the attributes, that include both sensitive attributes (i.e. race, ethnicity, sex) and non-sensitive attributes in the model" (section 2 para 2). Q_1 is the same as p. We have changed in the draft.

Reviewer comment (Theory):
There is also a claim here: "If two entities Q_1 and Q_i are similar, the expected outcome of both ... should be classified into the same class label depending on the class of Q_1" This reads like a very strong assumption about the problem setting. Consider e.g. the setting where there is a single integer attribute, and all instances with odd value for the attribute have ground truth label 0 and all even instances have ground truth label 1. How do we handle such a problem?

Author response:
If a specific condition, like the one described earlier, helps the classification process for individuals, the proposed approach can be applied to ground truth value and categorize individuals. In this context, for a single integer attribute, we can compute the Euclidean distance between two points. Depending on this distance value, if it's odd, the individual is classified into class 0; otherwise, they're classified into class 1.

Reviewer comment (Writing):
In Eq. (1), what does 'n' refer to? Earlier, 'n' was used as a variable to index the instances. Here, its use seems different.

Author response:
We have changed 'n' to 'm'. Here 'm' denotes the total number of attributes in the model.  (Section 2 para 2 page 3).

Reviewer comment (Reproducibility):
What is the threshold for deciding whether the similarity between Q_1 and Q_i is sufficient to assign Q_i the same label as Q_1?

Author response:
The average value of Euclidean distance between the two individuals. (Discussed in page 3 para 3)

Reviewer comment (Writing):
Page 3, last 2 lines: "each closed curve is used to represent sets": Do you mean multiple sets or a single set? If multiple, how to interpret this statement?

Author response:
We have changed to " Spider diagrams are higher-order representations of Venn diagrams and are equivalent to monadic FOL with equality. Here each closed curve is used to represent a set and is labeled and enclosed in a rectangle." in the draft (Section 2.1 para 1).

Reviewer comment (Writing):
I did not find the discussion of Section 2.1 to be helpful. Referring back to the original papers by Howse et al. helped clarify some things, and I can see how spider diagrams are useful for logical reasoning, but I am unable to completely understand its use in evaluating an AI model. An example that builds from a toy AI problem with a small dataset and a simple biased model would be greatly appreciated.

Author response:
We have included a demonstration that shows the working of our method using a toy example in Appendix B  page 15 (Figure 7).

Reviewer comment (Writing):
Definition 1. Do you mean to say for each expected label e_i, there exists an actual label a_i, such that e_i = a_i? Could you illustrate how this works using the example of a binary classification problem? Can an instance i have multiple expected and actual labels? Is it possible for an instance to have an expected label but no actual label or vice-versa?

Author response:
An instance "i" can have only one expected outcome (i.e., ground truth) and one actual outcome (i.e., predicted outcome). This is demonstrated with a toy example using a small example dataset (Appendix B  page 15 (Figure 7)).

Reviewer comment (Writing):
With a few assumptions, I can possibly see how in Section 3, the proposed algorithm can be used to compute the degree of bias. However, it would help to clarify the presentation and provide a running example to remove any ambiguities.

Author response:
We have included a demonstration that shows the working of our method using a toy example in Appendix B  page 15 (Figure 7). It demonstrates the method used in calculating the degree of bias.