Paper: Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias (ICLR-2024)

Reviewer comment (Experiments):
In this study, the authors employ a synthetic process using a series of classifiers and generation models to emulate real-world data distribution shifts. How do the authors substantiate that this accurately reflects actual distribution shift behaviors in the real world?

Author response:
```
“How do the authors substantiate that this accurately reflects actual distribution shift behaviors in the real world?”
```

All of our settings revolve around a sequence of models trained with datasets that get updated over time (whether by adding synthetic data, synthetic labels, or some mixture of synthetic and ‘real’ data). Concerns of use of synthetic data, whether intentionally or otherwise have already appeared in research and media. Our work seeks to better understand MIDS and opportunities for algorithmic reparation. We do not attempt to reflect distribution shift in general, just those induced by models. We also do not study what the impact of MIDS alongside other types of distribution shift might entail, though we do explore interactions between MIDS.

Reviewer comment (Novelty):
Could the authors delve into a discussion regarding how current methodologies address the challenges of model-induced distribution shifts?

Author response:
```
“Could the authors delve into a discussion regarding how current methodologies address the challenges of model-induced distribution shifts?” 

“Could the authors also present comparisons against established baseline methods?”
```
Points 2 and 3 ask for a discussion and experimental comparison against related methods. We show one comparison in Appendix E.2, where we provide an ablation study for the amount of synthetic data present at each generation. We also anticipate that training with fairness at each generation may be insufficient to stop unfairness accumulating due to MIDS (see our response to reviewer QqzD, point 4.b). Otherwise, please refer us to works that can address MIDS.

Reviewer comment (Experiments):
Furthermore, the numerical experiments focus solely on a comparison between scenarios with and without algorithmic repair. Could the authors also present comparisons against established baseline methods?

Author response:
```
“Could the authors delve into a discussion regarding how current methodologies address the challenges of model-induced distribution shifts?” 

“Could the authors also present comparisons against established baseline methods?”
```
Points 2 and 3 ask for a discussion and experimental comparison against related methods. We show one comparison in Appendix E.2, where we provide an ablation study for the amount of synthetic data present at each generation. We also anticipate that training with fairness at each generation may be insufficient to stop unfairness accumulating due to MIDS (see our response to reviewer QqzD, point 4.b). Otherwise, please refer us to works that can address MIDS.

Reviewer comment (Novelty):
This paper attempts to encompass several issues such as model collapse, performative prediction, unfairness feedback loops, and algorithmic reparation. However, the benefit and motivation for the unifying MIDS is not clear to me. It is encouraged that the authors clearly state what addition challenge could be solved, or what existing challenges could be better solved by the MIDS framework.

Author response:
`However, the benefit and motivation for the unifying MIDS is not clear to me. It is encouraged that the authors clearly state what additional challenges could be solved, or what existing challenges could be better solved by the MIDS framework.` As discussed in our response to reviewer vnkU02, our unification of so-far disparate model-induced effects under MIDS allows us to study them in isolation or when they interact, as in SeqGenSeqClass. It also allows us to work towards fair interventions using algorithmic reparation, and motivate work addressing these problems such as provenance for data, models, and outputs.

Reviewer comment (Experiments):
There are existing algorithms that could solve unfairness feedback loops, class imbalance, etc. However, the MIDS framework is not compared with those existing benchmarks.

Author response:
`There are existing algorithms that could solve unfairness feedback loops, class imbalance, etc. However, the MIDS framework is not compared with those existing benchmarks.` 
	* So far, we have found little related work addressing multiple MIDS. Please refer us to specific related works we may compare against. 
	* Since submitting our work, we measured the relative performances of each classifier with respect to its predecessor (instead of the original distribution). For equalized odds, where the ground truth label is assigned by the classifier’s predecessor, we found that the results presented in the main body occur even when there is only a small level of unfairness between each generation. In the relative equalized odds behind the ColoredMNIST results in Figure 4, for example, fairness violations of 0.05 compound to unfairness of 0.2 when compared to the original distribution. This demonstrates that even if we trained each generation’s classifier to be fairly compared to its predecessor, there may still be enough unfairness accumulating over time to cause high unfairness. These results have been added as Appendix G.

Reviewer comment (Writing):
Algorithmic reparation (AR) should be the most important concept/baseline in this paper; however, it is not technically introduced in the paper. It is encouraged that the authors add more discussion, use case, and operational meaning of AR.

Author response:
`It is encouraged that the authors add more discussion, use case, and operational meaning of AR.`
Algorithmic Reparation is not necessarily a technical framework for achieving fairness, but encompasses how to build or adjust algorithms for reparative purposes, using the algorithm to intervene in and correct injustice. Therefore, the technical implementation and objectives of AR are difficult to gauge as they will be heavily context-dependent. Please see So et al. (2022) for an example of how recourse can be used as algorithmic reparation in US housing discrimination.

Reviewer comment (Theory):
The methodologies proposed in Section 3 for MIDS lack theoretical analysis, performance guarantee, etc.

Author response:
`The methodologies proposed in Section 3 for MIDS lack theoretical analysis, performance guarantee, etc.`
If interested in theoretical backgrounds behind MIDS, please see Shumailov et al. (2023) for a formulation of the SeqGenNoSeqClass setting in terms of a markov chain and Taori and Hashimoto (2023) for a formulation of SeqClass in terms of calibration.

Reviewer comment (Evaluation):
In Figure 4, as the number of generations increase, the accuracy drops from 92% to 82%, and the fairness metrics like DP and EO are still large. Why a reduction of accuracy and unfairness will occur at the same time?

Author response:
In Figure 4, as the number of generations increases, the accuracy drops from 92% to 82%, and the fairness metrics like DP and EO are still large. Why does a reduction of accuracy and unfairness occur at the same time?

Figure 4 depicts sequential classifiers undergoing performative prediction. In the non-reparative results, there are no fairness interventions and we do not expect the classifiers to achieve lower unfairness. However, each generation’s mistakes continue to compound, lowering the accuracy. The algorithmic reparation is successful in eventually yielding batches with even\ideal class and group representation (Figure 4, bottom row), benefiting the fairness metrics. However, as these even batches no longer match the original distribution’s imbalances, there must be some incorrect classification when compared to the original distribution, which is why the accuracy still decreases. Please also see our response to reviewer qqNH’s first question for more information.

Reviewer comment (Writing):
Introduction seems a bit verbose and overly lyrical in moments, making it harder to read and follow. "recent demographic information of the Black population" - it is written that the maps are from 1939 and 1955. I am not sure that this is very recent.

Author response:
* Thank you for the catch on the `recent demographic information of the Black population,` this is now fixed.

Reviewer comment (Novelty):
I am not sure that MIDS require their own "taxonomy", given that there are only label and input drifts (Table 1). Impact of feedback loop in fairness has been acknowledged as a problem for a while (Mehrabi et al. 2019).

Author response:
* `I am not sure that MIDS require their own "taxonomy", given that there are only label and input drifts (Table 1). Impact of feedback loop in fairness has been acknowledged as a problem for a while (Mehrabi et al. 2019)`  
While fairness feedback loops are well-known in literature, they have not yet been connected to other phenomena such as model collapse or disparity amplification. We seek to unify these effects as their causes are distinct from traditional forms of distribution shift or domain adaptation. Unifying and recognizing these MIDS together opens avenues to study their co-occurrence/ co-operation, as well as study fairness interventions such as algorithmic reparation. The problems caused by MIDS motivates work in data and model attribution, but also in provenance for both generated data and data annotation (the latter being often overlooked).

Reviewer comment (Novelty):
The related work is not clearly discussed in the paper, making it a bit hard to get the overall context and contributions with respect to prior work.

Author response:
* `The related work is not clearly discussed in the paper, making it a bit hard to get the overall context and contributions with respect to prior work.`  
Additional literature on fairness in ML and its critiques is discussed in Appendix A, and descriptions of known MIDS are provided in Appendix B. Related work so far focuses on describing a MIDS, exploring its effects, and perhaps establishing some theoretical bounds on it. An additional related work previously missing from our paper is discussed below.

Reviewer comment (Novelty):
A recent related work by Taori and Hashimoto (2023) is missing. How does this paper relate to them?

Author response:
* `A recent related work by Taori and Hashimoto (2023) is missing. How does this paper relate to them?`  
* * We were unaware of this work until shortly before reviews were released, and have since added a discussion and citation in Appendix B.3. Taori and Hashimoto (2023) find a theoretical upper-bound for amplification of biases due to model feedback, applicable to our SeqGenNoSeqClass and SeqClass settings. They measure bias amplification as the difference between the original dataset/distribution bias and the bias of the current generation, and propose two interventions: overfitting models and/or allowing for a mixture of synthetic and human-generated data. Both of these methods reduce the bias amplification by encouraging each model to better/more closely represent the original distribution over time.  
* * Our work departs from theirs in three main regards. Firstly, we primarily study their worst-case scenario of entirely model-generated training sets (though we ablate the amount of synthetic data in Appendix E.2). Secondly, we advocate for using models to intervene in unfair systems through Algorithmic Reparation. In our experiments, we implement AR via a sampling scheme that attempts to achieve a “fair ideal” which may be very different from the rates of the original distribution (consider the base-rate disparities of CelebA, for example). Thirdly, we expand their investigation by observing the effects of co-occurring MIDS in the SeqGenSeqClass setting, finding an interesting dynamic where the classifiers undergoing performative prediction keep up with the changes in their input distribution arising from model collapse.

Reviewer comment (Theory):
The paper does not propose a mathematical model for the proposed MIDS scheme. Although various effects of retraining are shown through experiments, there is no theoretical investigation on the root causes of these effects, or the efficacy of the presented Algorithmic Reparation (AR) scheme.

Author response:
* `The paper does not propose a mathematical model for the proposed MIDS scheme.` If interested in theoretical backgrounds behind MIDS, please see Shumailov et al. (2023) for a formulation of the SeqGenNoSeqClass setting in terms of a Markov chain and Taori and Hashimoto (2023) for a formulation of SeqClass in terms of calibration.

Reviewer comment (Evaluation):
What is the reason for the decrease in the equalized odds difference with reparation in Figure 4 not being monotonic?

Author response:
```
What is the reason for the decrease in the equalized odds difference with reparation in Figure 4 not being monotonic?
```
Equalized odds is the maximum of the difference between the true positive rate (TPR) difference or false positive rate (FPR) difference between groups, and is therefore dependent on the ground truth. In Figure 4, we show our metrics as measured with respect to the original distribution (see Table 2), where there is correlation between the sensitive attribute value and class. The first classifiers in early generations start off unfair wrt the original distribution. As AR starts to balance the training set, the classifiers achieve a minimal unfairness wrt the original distribution (generation 13). However, as AR continues to balance the dataset and reach its ideal (Figure 4 bottom right), in the view of the original distribution the minoritized group is being “unfairly” benefited over the majoritized group, which is why the unfairness increases. When plotted, AR causes the majoritized TPR to decrease until it is lower (by about 0.035 also the final EOdds at generation 40) than the minoritized TPR, showing the “unfair” benefit AR gives to the minoritized group when seen from the view of fairness in the original distribution. There is also the presence of label shift due to the sequential classifiers, but these effects taken together is why we mention that we might not want to achieve fairness in Figures 4-6.

Reviewer comment (Evaluation):
The paper claims that "Generator-side AR improves fairness and minoritized representation", but plots on fairness metrics in Figure 5 do not corroborate this claim. Please explain.

Author response:
```
The paper claims that "Generator-side AR improves fairness and minoritized representation", but plots on fairness metrics in Figure 5 do not corroborate this claim.
```
We find that using the generator-side AR technique does improve fairness and minoritized representation. In Figure 5, this is shown by the purple dotted line where the demographic parity and equalized odds values are lower (fairer) than (a) classifier-side reparation and (b) results without reparation. The generator-side reparation maintains the original minoritized group population at 50% of the total, as in the ColoredMNIST dataset. Because classifier-side AR and baseline results diminish this population, they result in less representation.

Reviewer comment (Writing):
I do not understand the claim, "Performative prediction on model collapse leads to higher utility". Please explain what you mean by this.

Author response:
```
I do not understand the claim, "Performative prediction on model collapse leads to higher utility". Please explain what you mean by this.
```
This claim means that the combined setting SeqGenSeqClass allows model collapse (MIDS on the generative models) and performative prediction (MIDS on the classifiers) to co-exist. Compared to the SeqGenNoSeqClass (i.e., just model collapse), this setting has higher utility (by this we mean accuracy). This result highlights cooperation between the model collapse and performative prediction MIDS as the SeqGenSeqClass classifiers adapt to changes due to model collapse. Please see Appendix E.3 for further discussion. 

In retrospect, the wording of this claim is confusing so we amended it to “Performative prediction combined with model collapse yields higher accuracy.”