Paper: Evaluating model bias requires characterizing model mistakes (ICLR-2024)

Reviewer comment (Novelty):
The novelty of the proposed metric is a concern. The use of contingency tables, relevant statistics, and skewness for data analysis is not a new thing. It appears that SKEWSIZE merely combines these elements to measure fairness/bias. Could the authors elucidate further on what makes their proposed metric unique?

Author response:
*1. “The novelty of the proposed metric is a concern. The use of contingency tables, relevant statistics, and skewness for data analysis is not a new thing. It appears that SKEWSIZE merely combines these elements to measure fairness/bias. Could the authors elucidate further on what makes their proposed metric unique?”*

Our main contribution lies in proposing a principled and not yet seen in the literature approach to leverage established statistical methods to characterize biases in neural networks across multiple settings, including cases where models are used in open-ended ways such as in Section 3.4 of our work. We agree with the reviewer that the separate statistical elements are not novel.  As we describe in the Related Works section, prior work has also investigated statistical methods to investigate fairness, but these techniques had limitations such as providing  a single p-value [1]. In contrast, we show that by referring to the effect size, and using the skewness to aggregate results across classes, we obtain both a detailed and a global view of the bias. In addition, we believe that referring to well-established statistical methods is a strength of this work, as it is principled and provides guidance on which method should be used to measure effect size and how. 

We have added these elements in the abstract and discussion:
Highlighting in the abstract that our main contributions are related to propose a principled approach to measure bias that leverages well-established statistical analysis techniques: “Inspired by the well-established hypothesis testing framework, we introduce \metricname, a principled and flexible metric that captures bias from mistakes in a model's predictions.”
Further highlighting in the discussion (Section 5) that our metric adds to the literature of statistical methods to measure fairness by considering the effect size between interactions rather than p-values. By doing so we can provide both a per-class and global view of a model’s bias profile: “we draw on tools from contingency table hypoth-esis testing and propose to measure bias on a per-class basis by estimating the effect size of the interaction between model prediction and the bias variable, **rather than outputting a single p-value as previous work**”. 

[1] Tramer et al., “Fairtest:  Discovering  unwarranted  associations  in  data-driven applications”, 2017.

Reviewer comment (Theory):
The discussion in Section 2.2 is constrained to cases with only two items in Z. Is it feasible to extend this discussion to scenarios with multiple items?

Author response:
*2-“The discussion in Section 2.2 is constrained to cases with only two items in Z. Is it feasible to extend this discussion to scenarios with multiple items?”*

Thank you for this comment. We decided to use binary attributes in Definition 2.2 just for the sake of clarity in the presentation. While the definition of distributional bias employs the binary case, the $\mathcal{H}$ operator is not limited to binary attributes as we can instantiate it by using the Cramer’s V effect size, which works for contingency tables of any size.

Following the reviewer’s comment, we have added the following text to Section 2.2 in order to clarify this point:
 “Notice that the $\mathcal{H}$ operator is not limited to binary attributes and can be instantiated by approaches to simultaneously compare multiple families of distributions.”

Reviewer comment (Evaluation):
Regarding the experiment section, it's unclear how computing the correlation between SKEWSIZE and other metrics, as done in Sections 3.2 and 3.3, demonstrates "that the proposed metric identifies issues with models not exposed by these other metrics".

Author response:
*3-“Regarding the experiment section, it's unclear how computing the correlation between SKEWSIZE and other metrics, as done in Sections 3.2 and 3.3, demonstrates "that the proposed metric identifies issues with models not exposed by these other metrics".”*

We compute the correlation between different metrics to highlight how these are related to each other. For instance, Section 3.1 displays that EO will capture bias, but not necessarily on the class it originates from. Therefore, the correlation per class between these two metrics shows that they both can highlight biases, but they are not equivalent. When compared to other metrics such as GAP or worst group accuracy, the correlation is not significant, demonstrating that these metrics measure  fundamentally different aspects of bias. In particular, GAP or worst group accuracy are not sufficient to evaluate bias arising from the mispredictions of a model as demonstrated in Figure 1 in the paper. The fact that in Sections 3.2 and 3.3 this leads to different models having lower SkewSize than Accuracy (as demonstrated by the correlation), again highlights that SkewSize is exposing biases that are not captured by the accuracy measurements. Therefore, we claim that Effect size and Skewsize “provides complementary information to other metrics” (as mentioned in the results and in the discussion).

We have added the following sentence to Section 3.2 in order to make this point clearer and address the reviewer’s concern: “[the correlation results...] indicate the metrics are related but not equivalent as they capture distinct aspects of the bias.”

We would greatly appreciate it if you let us know in case there are any other questions so we can further improve the manuscript based on your feedback.

Reviewer comment (Novelty):
The paper's SkewSize metric is based on statistical concepts such as skewness and chi-square tests, which may not represent a fundamentally novel approach. While it introduces a new metric, the underlying statistical principles are not groundbreaking.

Author response:
*“While it introduces a new metric, the underlying statistical principles are not groundbreaking. The paper may lack excitement or the potential to generate significant interest, as it builds upon established statistical techniques rather than introducing innovative or revolutionary methodologies.”*

Our main contribution lies in proposing a principled approach not yet seen in the literature : we leverage established statistical methods to characterize biases in neural networks across multiple settings, including cases where models are used in open-ended ways (such as in Section 3.4 of our work). We agree with the reviewer that the separate statistical elements are not novel.  As we describe in the Related Works section, prior work has also investigated statistical methods to investigate fairness, but these techniques had limitations such as providing  a single p-value [1]. In contrast, we show that by referring to the effect size, and using the skewness to aggregate results across classes, we obtain both a detailed and a global view of the bias. In addition, we believe that referring to well-established statistical methods is a strength of this work, as it is principled and provides guidance on which method should be used to measure effect size and how. 

We have added these elements in the abstract and discussion:

- Highlighting in the abstract that our main contributions are related to propose a principled approach to measure bias that leverages well-established statistical analysis techniques: “Inspired by the well-established hypothesis testing framework, we introduce SkewSize, a principled and flexible metric that captures bias from mistakes in a model's predictions.”
- Further highlighting in the discussion (Section 5) that our metric adds to the literature of statistical methods to measure fairness by considering the effect size between interactions rather than p-values. By doing so we can provide both a per-class and global view of a model’s bias profile: “we draw on tools from contingency table hypothesis testing and propose to measure bias on a per-class basis by estimating the effect size of the interaction between model prediction and the bias variable, **rather than outputting a single p-value as previous work**”. 

[1] Tramer et al., “Fairtest:  Discovering  unwarranted  associations  in  data-driven applications”, 2017.

Reviewer comment (Novelty):
The paper may lack excitement or the potential to generate significant interest, as it builds upon established statistical techniques rather than introducing innovative or revolutionary methodologies.

Author response:
*“While it introduces a new metric, the underlying statistical principles are not groundbreaking. The paper may lack excitement or the potential to generate significant interest, as it builds upon established statistical techniques rather than introducing innovative or revolutionary methodologies.”*

Our main contribution lies in proposing a principled approach not yet seen in the literature : we leverage established statistical methods to characterize biases in neural networks across multiple settings, including cases where models are used in open-ended ways (such as in Section 3.4 of our work). We agree with the reviewer that the separate statistical elements are not novel.  As we describe in the Related Works section, prior work has also investigated statistical methods to investigate fairness, but these techniques had limitations such as providing  a single p-value [1]. In contrast, we show that by referring to the effect size, and using the skewness to aggregate results across classes, we obtain both a detailed and a global view of the bias. In addition, we believe that referring to well-established statistical methods is a strength of this work, as it is principled and provides guidance on which method should be used to measure effect size and how. 

We have added these elements in the abstract and discussion:

- Highlighting in the abstract that our main contributions are related to propose a principled approach to measure bias that leverages well-established statistical analysis techniques: “Inspired by the well-established hypothesis testing framework, we introduce SkewSize, a principled and flexible metric that captures bias from mistakes in a model's predictions.”
- Further highlighting in the discussion (Section 5) that our metric adds to the literature of statistical methods to measure fairness by considering the effect size between interactions rather than p-values. By doing so we can provide both a per-class and global view of a model’s bias profile: “we draw on tools from contingency table hypothesis testing and propose to measure bias on a per-class basis by estimating the effect size of the interaction between model prediction and the bias variable, **rather than outputting a single p-value as previous work**”. 

[1] Tramer et al., “Fairtest:  Discovering  unwarranted  associations  in  data-driven applications”, 2017.

Reviewer comment (Evaluation):
While the paper identifies biases in model predictions using the SkewSize metric, it doesn't definitively confirm whether these biases are genuine or artifacts. The sensitivity of the metric could potentially lead to the detection of biases that are not practically significant.

Author response:
*“While the paper identifies biases in model predictions using the SkewSize metric, it doesn't definitively confirm whether these biases are genuine or artifacts. The sensitivity of the metric could potentially lead to the detection of biases that are not practically significant.”*

We believe there is some misunderstanding with respect to how the reviewer is interpreting SkewSize. The metric **is designed** to be sensitive to different degrees of bias and this should be reflected in the strength of the effect size for each evaluated class. In more detail, we propose a metric that accounts for the effect size of the interaction between the bias and prediction variables, i.e. how much bias there is in the predictions across classes, and we **do not set a threshold** so as to deem classes/models as positive/negative biased. 

To further elucidate this point, we leverage the fact that our experimental setting with the dSprites dataset allows us to obtain models presenting varying levels of bias to show that our approach is capable of capturing such differences. In more detail, we consider a variation of the dSprites setting from Section 3.1 of our manuscript where we train models with subsets of different sizes containing part of the removed instances (green ellipsis/class 1 for this case) so as to introduce different bias levels. As reported in the table below (Table 5 in the revised version of the manuscript), we find that the effect size presents a monotonic relationship with bias strength, therefore a small change in the bias of the output leads to a small change in the metric and a large change in bias leads to a large change in the metric. **Confirming that the effect size is not too sensitive or not sensitive enough to variations in the bias strength.**

|                              | Accuracy metrics |       |       |  Effect size |         |         |
|:----------------------------:|:----------------:|:-----:|:-----:|:-----------:|:-------:|:-------:|
|          Bias level          |     Accuracy     |   WG  |  Gap  |   Class 0   | Class 1 | Class 2 |
| Unbiased (full training set) |       0.998      | 0.996 | 0.002 |    0.006    |   0.02  |  0.024  |
|      Mildly biased (5k)      |       0.977      | 0.936 | 0.041 |    0.002    |  0.253  |  0.017  |
|     Highly biased (2.5k)     |       0.933      | 0.806 | 0.127 |    0.013    |  0.492  |  0.032  |
|      Totally biased (0)      |       0.891      | 0.653 | 0.238 |    0.012    |  0.703  |  0.011  |

Finally, we remark that,  in our opinion, the definition of practically relevant bias is context-dependent and that a metric should account for all existing biases in a dataset/model so that a comprehensive profile of a model's performance can be taken into consideration at the evaluation. In case, for example, biases associated with negligible effect size are not practically relevant for a particular application, the corresponding classes can be excluded from the aggregation step when computing SkewSize.

To reflect in the manuscript our response to the pertinent point raised by the reviewer we have added the table along with the respective discussion to the Appendix C.

Reviewer comment (Evaluation):
There's a concern that the SkewSize metric might be overly sensitive to minor variations or noise in data, which could result in an overemphasis on biases that may not be robust or practically impactful.

Author response:
*“While the paper identifies biases in model predictions using the SkewSize metric, it doesn't definitively confirm whether these biases are genuine or artifacts. The sensitivity of the metric could potentially lead to the detection of biases that are not practically significant.”*

We believe there is some misunderstanding with respect to how the reviewer is interpreting SkewSize. The metric **is designed** to be sensitive to different degrees of bias and this should be reflected in the strength of the effect size for each evaluated class. In more detail, we propose a metric that accounts for the effect size of the interaction between the bias and prediction variables, i.e. how much bias there is in the predictions across classes, and we **do not set a threshold** so as to deem classes/models as positive/negative biased. 

To further elucidate this point, we leverage the fact that our experimental setting with the dSprites dataset allows us to obtain models presenting varying levels of bias to show that our approach is capable of capturing such differences. In more detail, we consider a variation of the dSprites setting from Section 3.1 of our manuscript where we train models with subsets of different sizes containing part of the removed instances (green ellipsis/class 1 for this case) so as to introduce different bias levels. As reported in the table below (Table 5 in the revised version of the manuscript), we find that the effect size presents a monotonic relationship with bias strength, therefore a small change in the bias of the output leads to a small change in the metric and a large change in bias leads to a large change in the metric. **Confirming that the effect size is not too sensitive or not sensitive enough to variations in the bias strength.**

|                              | Accuracy metrics |       |       |  Effect size |         |         |
|:----------------------------:|:----------------:|:-----:|:-----:|:-----------:|:-------:|:-------:|
|          Bias level          |     Accuracy     |   WG  |  Gap  |   Class 0   | Class 1 | Class 2 |
| Unbiased (full training set) |       0.998      | 0.996 | 0.002 |    0.006    |   0.02  |  0.024  |
|      Mildly biased (5k)      |       0.977      | 0.936 | 0.041 |    0.002    |  0.253  |  0.017  |
|     Highly biased (2.5k)     |       0.933      | 0.806 | 0.127 |    0.013    |  0.492  |  0.032  |
|      Totally biased (0)      |       0.891      | 0.653 | 0.238 |    0.012    |  0.703  |  0.011  |

Finally, we remark that,  in our opinion, the definition of practically relevant bias is context-dependent and that a metric should account for all existing biases in a dataset/model so that a comprehensive profile of a model's performance can be taken into consideration at the evaluation. In case, for example, biases associated with negligible effect size are not practically relevant for a particular application, the corresponding classes can be excluded from the aggregation step when computing SkewSize.

To reflect in the manuscript our response to the pertinent point raised by the reviewer we have added the table along with the respective discussion to the Appendix C.

Reviewer comment (Evaluation):
The paper doesn't thoroughly address the potential for false positives when using the SkewSize metric. It's crucial to differentiate between genuine biases and statistical anomalies.

Author response:
**In prediction tasks involving thousands of objects, like those in ImageNet, the effect size may appear significant by chance, leading to skewed SkewSize calculations.[...] The paper does not thoroughly address the risk of false positives associated with the SkewSize metric. It is imperative to distinguish between genuine biases and statistical anomalies for a more comprehensive evaluation.**

A key aspect of our method to deal with this issue is the use of the minimum expected value (MEV). We display in our prior response the impact of this hyper-parameter on our analyses. To provide a more thorough evaluation of false positives, we performed additional simulations by generating random predictions for 2 groups. Each group is drawn from the same distribution, and hence a small effect size is expected. We generate 2,000 samples per group, and vary the number of classes from 2 to 800 with a step of 20. For each sampling and number of classes, we compute the effect size, for MEV = 0 and for MEV = 5, repeating the operation 20 times.

We then display the mean and std across trials of the effect size across the number of classes. We observe that the MEV allows to limit the effect size and reaches a plateau across the number of classes. On the other hand, p-values display more stability across number of classes, with non-significant results. While p-values did not reflect the strength of the bias in our prior experiments when bias was indeed present (not shown), we will suggest to consider the effect size for significant tests only. This should prevent false positives and reduce the sensitivity of our metric. We will add this analysis to our methods (section 2.2), and the Appendix.

[Link to Figure: effect size vs number of classes](https://drive.google.com/file/d/15pZeBhrwq9SmkVrh4tEoMUdmkp4PjFAj/view?usp=sharing)

[Link to Figure: p-value vs number of classes](https://drive.google.com/file/d/1-pXxgpagCJhbhr85hmh0c0itxoRlc-vQ/view?usp=sharing)


As a note, while we agree that sensitivity might be an issue, this metric allows us to investigate potential sources of biases. Therefore, we believe it is safer to be on the side of more false positives than false negatives, and (partial) confusion matrices can be explored to ensure the bias is not concerning as defined by different stakeholders.

Reviewer comment (Theory):
In the code implementation of SkewSize, why the degree of freedom is calculated by the minimum of the prediction class and subgroup class minus 1? Shouldn't it be $(prediction class - 1)*(subgroup class - 1)$?

Author response:
*“In the code implementation of SkewSize, why the degree of freedom is calculated by the minimum of the prediction class and subgroup class minus 1? Shouldn't it be $(predictionclass-1)*(subgroupclass-1)$?”*

We compute the empirical Cramer’s V statistics as per the definitions in [3,4,5].

[3] Cramer, Harold. “Mathematical methods of statistics”. Princeton Press, NJ, 1946 \
[4] Bergsma, Wicher. "A bias-correction for Cramér’s V and Tschuprow’s T." Journal of the Korean Statistical Society, 2013. \
[5] [Wikipedia page for Cramer's V](https://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V#:~:text=Cram%C3%A9r%27s%20V%20varies%20from%200,of%20their%20maximum%20possible%20variationRe).

Reviewer comment (Evaluation):
When there exists a large group of object classes, the effect size may be large by chance, hence the calculated SkewSize will tend to be skew. I wonder if the authors have conducted experiments to understand the robustness of the proposed metric. SkewSize seems to be very sensitive in that it can identify potential bias in the model predictions. However, it is not clear that in the case where no bias exists in the predictions, SkewSize also demonstrates some degree of skewness.

Author response:
*“I wonder if the authors have conducted experiments to understand the robustness of the proposed metric. SkewSize seems to be very sensitive in that it can identify potential bias in the model predictions. However, it is not clear that in the case where no bias exists in the predictions, SkewSize also demonstrates some degree of skewness.”*

We hope that, based on the expanded Section 3.1 and the responses above, the reviewer will agree that SkewSize does not demonstrate high sensitivity and is in line with expectations when results are unbiased. In fact, in the initial version of the our manuscript we have investigated this aspect in Section 3.1 as the experimental setup allowed us to control for bias in the sense that we trained the model with a perfectly balanced dataset and then compute the effect size values for all classes (c.f. results in Table 1 in the manuscript). **Results show that when there is no bias, the effect size is small for all classes.** 

In addition to the experiment discussed above, the result reported in part 2 of the rebuttal further corroborates our point by showing that the effect is able to capture different levels of bias induced in the model. 

We would greatly appreciate it if you let us know in case there are any other questions so we can further improve the manuscript based on your feedback.

Reviewer comment (Evaluation):
The comparison with respect to other benchmark methods is somewhat unfair. The authors should study effectiveness of statistical measures (in general) in identifying bias. For example, one could perform a statistical test (e.g. MWU test) with null hypothesis that the model scores for a subgroup (that represents a bias attribute) comes from the same distribution of as model scores without any subgrouping. An acceptance of the null hypothesis (using p-value) would indicate an absence of bias and rejection of the null hypothesis in favour of alternate hypothesis would indicate the presence of bias with respect to the subgrouping attribute.

Author response:
*1- “The comparison with respect to other benchmark methods is somewhat unfair. For example, one could perform a statistical test (e.g. MWU test) with null hypothesis that the model scores for a subgroup (that represents a bias attribute) comes from the same distribution of as model scores without any subgrouping.”*

We thank the reviewer for this suggestion. While there might be other settings (e.g. domain adaptation) where drawing inspiration from the MWU test is suitable, in the setting we consider (e.g. distributional bias), we believe it could give spurious results due to its sensitivity to model specific properties that are reflected in scores (e.g. model confidence). Also, using the scores is a less general method as it requires access to the raw model output; it is not usable when one only has access to the predictions of a model. 

In order to reflect the reviewer’s comment in our work, we have added to the discussion the following statement: 

“Aspects to be investigated in future work include how our work could be extended to continuous attributes by using a different statistic (e.g. Tramer et al., 2017; Brown et al., 2023).  While the selected aggregation approach and metrics are suitable for the cases we consider in this work, in this case, other testing statistics or aggregations might be more appropriate”.

Reviewer comment (Experiments):
Essentially, it is probably important to present benchmark comparisons on how the standard statistical measures and methods perform in identifying bias as baselines. Then only it presents us opportunity to appreciate the proposed specific statistical measure in bias identification.

Author response:
*2. “Essentially, it is probably important to present benchmark comparisons on how the standard statistical measures and methods perform in identifying bias as baselines. Then only it presents us opportunity to appreciate the proposed specific statistical measure in bias identification.”*

We agree that other statistical methods could be considered in our evaluation in addition to the 3 accuracy based metrics and 2 fairness metrics we already considered in the paper. Following the reviewer’s suggestion, we have added to the manuscript (Appendix B) results with the Phi coefficient as a measure of effect size when computing SkewSize in the dSprites experiments. The results show that in this case the Phi Coefficient yields similar trends as the Cramer’s V. Notice, however, that it is not advisable to use the Phi Coefficient on contingency tables larger than 2x2, that is the reason why we decided to use the more general Cramer’s V when computing SkewSize throughout our work.

|          | Cramer’s V - Class 0 | Cramer’s V - Class 1 | Cramer’s V - Class 2 | Phi - Class 0 | Phi - Class 1 | Phi - Class 2 |
|:--------:|:--------------------:|:--------------------:|:--------------------:|:-------------:|:-------------:|:-------------:|
| Unbiased |         0.012        |         0.011        |         0.019        |     0.012     |     0.011     |     0.027     |
|  Class 0 |         0.670        |         0.015        |         0.016        |     0.948     |     0.015     |      0220     |
|  Class 1 |         0.014        |         0.683        |         0.108        |     0.014     |     0.966     |     0.152     |
|  Class 2 |         0.047        |         0.006        |         0.696        |     0.067     |     0.006     |     0.985     |


We would greatly appreciate it if you let us know in case there are any other questions so we can further improve the manuscript based on your feedback.