Original: O N T HE I MPACT OF M ACHINE L EARNING R ANDOMNESS ON G ROUP F AIRNESS
Revised: O N T HE I MPACT OF M ACHINE L EARNING R ANDOMNESS ON G ROUP F AIRNESS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Statistical measures for group fairness in machine learning reflect the gap in performance of algorithms across different groups.
Revised: Statistical measures for group fairness in machine learning reflect the gap in performance of algorithms across different groups.

Original: These measures, however, exhibit a high variance, between different training instances, that makes them unreliable for empirical evaluation of fairness.
Revised: These measures, however, exhibit a high variance, between different training instances, that makes them unreliable for empirical evaluation of fairness.

Original: What is the cause of this variance, and how can we reduce it?
Revised: What is the cause of this variance, and how can we reduce it?

Original: We investigate the impact of different sources of randomness in machine learning on group fairness.
Revised: We investigate the impact of different sources of randomness in machine learning on group fairness.

Original: We show that the variance in group fairness measures is mainly due to the high volatility of the learning process on under-represented groups, which itself is largely caused by the stochasticity of data order during training.
Revised: We show that the variance in group fairness measures is mainly due to the high volatility of the learning process on under-represented groups, which itself is largely caused by the stochasticity of data order during training.

Original: Based on these findings, we show how to manipulate group level accuracy (i.e. model fairness), with high efficiency and negligible impact on the overall predictive power of the model, by changing the data order.
Revised: Based on these findings, we show how to manipulate group level accuracy (i.e. model fairness), with high efficiency and negligible impact on the overall predictive power of the model, by changing the data order.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models are shown to manifest and escalate historical prejudices and biases present in their training data (Crawford, 2013; Barocas & Selbst, 2016;
Revised: Machine learning models are shown to manifest and escalate historical prejudices and biases present in their training data (Crawford, 2013; Barocas & Selbst, 2016;

Original: Zhao et al., 2017; Abbasi et al., 2019).
Revised: Zhao et al., 2017; Abbasi et al., 2019).

Original: Understanding these biases and the following ethical considerations has led to the rise of fair machine learning research (Chouldechova & Roth, 2018; Caton & Haas, 2020; Mehrabi et al., 2021).
Revised: Understanding these biases and the following ethical considerations has led to the rise of fair machine learning research (Chouldechova & Roth, 2018; Caton & Haas, 2020; Mehrabi et al., 2021).

Original: Recent work in fair deep learning have observed a trend of high variance in fairness measures across multiple training runs (Qian et al., 2021;
Revised: Recent work in fair deep learning have observed a trend of high variance in fairness measures across multiple training runs (Qian et al., 2021;

Original: Amir et al., 2021; Sellam et al., 2021; Soares et al., 2022), usually attributed to non-determinism in training.
Revised: Amir et al., 2021; Sellam et al., 2021; Soares et al., 2022), usually attributed to non-determinism in training.

Original: These results have challenged the legitimacy of existing claims in the literature (Soares et al., 2022), and have even disputed the effectiveness of various bias mitigation techniques (Amir et al., 2021; Sellam et al., 2021).
Revised: These results have challenged the legitimacy of existing claims in the literature (Soares et al., 2022), and have even disputed the effectiveness of various bias mitigation techniques (Amir et al., 2021; Sellam et al., 2021).

Original: Thus, a reliable extraction of fairness trends in a model requires accounting for the high variance to avoid lottery winners (see Figure 1).
Revised: Thus, a reliable extraction of fairness trends in a model requires accounting for the high variance to avoid lottery winners (see Figure 1).

Original: The naive solution of executing a large number of identical training runs to capture the overall variance creates a huge computational demand, and discourages the examination of biases in several rapidly growing forefronts of machine learning research by increasing the resource requirements.
Revised: The naive solution of executing a large number of identical training runs to capture the overall variance creates a huge computational demand, and discourages the examination of biases in several rapidly growing forefronts of machine learning research by increasing the resource requirements.

Original: Therefore, understanding the actual cause of the high variance in the fairness measures is critical.
Revised: Therefore, understanding the actual cause of the high variance in the fairness measures is critical.

Original: To the best of our knowledge, we are the first to study fairness variance beyond trivially executing multiple identical training runs.
Revised: To the best of our knowledge, we are the first to study fairness variance beyond trivially executing multiple identical training runs.

Original: More specifically, we show the following
Revised: More specifically, we show the following

Original: : • We show that the trends of fairness variance observed in literature are dominated by random data reshuffling during training, which causes high fairness variance between epochs even within a single training run, while the non-determinism in weight initialization has minimal influence.
Revised: : • We show that the trends of fairness variance observed in literature are dominated by random data reshuffling during training, which causes high fairness variance between epochs even within a single training run, while the non-determinism in weight initialization has minimal influence.

Original: • We extract an empirical relationship between group representation and instability in group level performance, highlighting higher vulnerability of minority to changing model behavior.
Revised: • We extract an empirical relationship between group representation and instability in group level performance, highlighting higher vulnerability of minority to changing model behavior.

Original: Our results attribute the high fairness variance to lower prediction stability for under-represented subgroups.
Revised: Our results attribute the high fairness variance to lower prediction stability for under-represented subgroups.

Original: • We demonstrate an immediate dominance of the data order on model fairness.
Revised: • We demonstrate an immediate dominance of the data order on model fairness.

Original: A model’s fairness is predictable, based on only the most recent training points, irrespective of preceding model behavior.
Revised: A model’s fairness is predictable, based on only the most recent training points, irrespective of preceding model behavior.

Original: • Based on this information, we propose to use the fairness variance across epochs as a proxy to study the changing model fairness across multiple training runs, thus reducing the computational requirements by a significant margin.
Revised: • Based on this information, we propose to use the fairness variance across epochs as a proxy to study the changing model fairness across multiple training runs, thus reducing the computational requirements by a significant margin.

Original: Finally, we manipulate group level performances (i
Revised: Finally, we manipulate group level performances (i

Original: e., model fairness) by changing the data order, with a relatively minor impact on the overall accuracy.
Revised: e., model fairness) by changing the data order, with a relatively minor impact on the overall accuracy.

Original: This manipulation can improve fairness as well as reverse the effects of several bias mitigation algorithms within a single training epoch.
Revised: This manipulation can improve fairness as well as reverse the effects of several bias mitigation algorithms within a single training epoch.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Datasets and Models We will conduct our investigation on ACSIncome and ACSEmployment tasks of the Folktables dataset (Ding et al., 2021), and binary classification of the ’smiling’ label in
Revised: Datasets and Models We will conduct our investigation on ACSIncome and ACSEmployment tasks of the Folktables dataset (Ding et al., 2021), and binary classification of the ’smiling’ label in

Original: CelebA dataset (Liu et al., 2015), with gender (Female vs. Male) as the sensitive attribute for all datasets.
Revised: CelebA dataset (Liu et al., 2015), with gender (Female vs. Male) as the sensitive attribute for all datasets.

Original: For CelebA, input features are obtained by passing the input image through a pre-trained and frozen ResNet-50 backbone and extracting the output feature vector after the final average pooling layer.
Revised: For CelebA, input features are obtained by passing the input image through a pre-trained and frozen ResNet-50 backbone and extracting the output feature vector after the final average pooling layer.

Original: More details on the datasets are provided in Appendix B.
Revised: More details on the datasets are provided in Appendix A.

Original: We train a feed forward network on input features with a single hidden layer consisting ofneurons and ReLU activation, and train the model with cross-entropy (CE) loss objective for a total of 300 epochs at batch size 128 , in all our experiments unless specified otherwise.
Revised: We train a feed forward network on input features with a single hidden layer consisting ofneurons and ReLU activation, and train the model with cross-entropy (CE) loss objective for a total of 300 epochs at batch size 128 , in all our experiments unless specified otherwise.

Original: We include additional experiments by changing training hyperparameters, i.e., batch size, learning rate, and model architecture, in Appendix F.
Revised: We include additional experiments by changing training hyperparameters, i.e., batch size, learning rate, and model architecture, in Appendix E.4.

Original: We use train-val-test split of 0 .
Revised: We use train-val-test split of 0 .

Original: 7 : 0 . 1 : 0 . 2 , and maintain the same split throughout all our experiments, i.e. we do not consider potential non-determinism introduced due to changing train-val-test splits.
Revised: 7 : 0 . 1 : 0 . 2 , and maintain the same split throughout all our experiments, i.e. we do not consider potential non-determinism introduced due to changing train-val-test splits.

Original: All our evaluations are performed on the test split.
Revised: All our evaluations are performed on the test split.

Original: We will focus primarily on the ACSIncome dataset in the main text, while additional experiments onCelebA and
Revised: We will focus primarily on the ACSIncome dataset in the main text, while additional experiments on

Original:  ACSEmployment are included in the appendix.
Revised: CelebA and ACSEmployment are included in the appendix.

Original: Fairness Metrics and Variance Fairness in machine learning has been interpreted widely into a multitude of definitions.
Revised: Fairness Metrics and Variance Fairness in machine learning has been interpreted widely into a multitude of definitions.

Original: In this paper, we will focus on average odds (AO), empirically interpreted as the average disparity between true positive rates and false positive rates of various groups.
Revised: In this paper, we will focus on average odds (AO), empirically interpreted as the average disparity between true positive rates and false positive rates of various groups.

Original: We also include additional results for equalized opportunity (EOpp) and disparate impact (DI) in Appendix G.
Revised: We also include additional results for equalized opportunity (EOpp) and disparate impact (DI) in Appendix F.3.

Original: For model predictions R , true labels Y , and sensitive attributes A , average odds can be defined as,
Revised: For model predictions R , true labels Y , and sensitive attributes A , average odds can be defined as,

Original: [Equation]
Revised: [Equation]

Original: At the heart of our work is the study of fairness variance across model checkpoints.
Revised: At the heart of our work is the study of fairness variance across model checkpoints.

Original: Unless otherwise specified, variance across multiple training runs refers to the variance across final checkpoints at epoch 300 of each training run.
Revised: Unless otherwise specified, variance across multiple training runs refers to the variance across final checkpoints at epoch 300 of each training run.

Original: Similarly, variance across epochs refers to variance in a single training run across checkpoints at the end of every epoch for the last 200 epochs of training (i.e., from epoch 100 to epoch 300 ).
Revised: Similarly, variance across epochs refers to variance in a single training run across checkpoints at the end of every epoch for the last 200 epochs of training (i.e., from epoch 100 to epoch 300 ).

Original: We make this choice as the model has converged to stable accuracy scores before epoch 100 (refer to the training curve in Appendix C for more details).
Revised: We make this choice as the model has converged to stable accuracy scores before epoch 100 (refer to the training curve in Appendix A for more details).

Original: Non-Determinism in Model Training
Revised: Non-Determinism in Model Training

Original: In this work, we focus on randomness due to stochasticity in the training algorithm, and we set manual seeds at various intermediate locations in our code to control the randomness.
Revised: In this work, we focus on randomness due to stochasticity in the training algorithm, and we set manual seeds at various intermediate locations in our code to control the randomness.

Original: We refer to the seed set right before building the neural model as the weight initialization seed, which influences the randomness in sampling the weight values.
Revised: We refer to the seed set right before building the neural model as the weight initialization seed, which influences the randomness in sampling the weight values.

Original: Similarly, we refer to the seed set right before the first training data shuffling as random reshuffling seed, which influences the data order that will be used as reference for the rest of the training.
Revised: Similarly, we refer to the seed set right before the first training data shuffling as random reshuffling seed, which influences the data order that will be used as reference for the rest of the training.

Original: During training, we simply set the epoch number as seed right before reshuffling the reference data order at every epoch.
Revised: During training, we simply set the epoch number as seed right before reshuffling the reference data order at every epoch.

Original: As we can change the reference data order by changing the random reshuffling seed, this setup allows us to control non-determinism in data order throughout training with a single random seed.
Revised: As we can change the reference data order by changing the random reshuffling seed, this setup allows us to control non-determinism in data order throughout training with a single random seed.

Original: Fixing both weight initialization and random reshuffling seed allows us to deterministically replicate model training, while changing both seeds simultaneously is similar to the discussion of non-determinism present in literature.
Revised: Fixing both weight initialization and random reshuffling seed allows us to deterministically replicate model training, while changing both seeds simultaneously is similar to the discussion of non-determinism present in literature.

Original: Pseudo code detailing the definition of various random seeds is provided in
Revised: Pseudo code detailing the definition of various random seeds is provided in

Original: Appendix A. We also perform additional experiments on the impact of randomness introduced by dropout regularization in Appendix D.
Revised: Appendix A. We also perform additional experiments on the impact of randomness introduced by dropout regularization in Appendix A.

Original:  T HE S OURCES OF R ANDOMNESS
Revised:  T HE S OURCES OF R ANDOMNESS

Original: We start by moving past the trivial observation that different training runs lead to different outcomes, and investigate the high fairness variance by studying the intricacies of the training dynamics.
Revised: We start by moving past the trivial observation that different training runs lead to different outcomes, and investigate the high fairness variance by studying the intricacies of the training dynamics.

Original: 3. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING
Revised: 3. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING

Original: Neural model training under widely adapted mini-batch stochastic gradient descent (SGD) has two main sources of non-determinism, i.e. weight initialization and random reshuffling.
Revised: Neural model training under widely adapted mini-batch stochastic gradient descent (SGD) has two main sources of non-determinism, i.e. weight initialization and random reshuffling.

Original: To decouple their impact on fairness variance, we perform the following experiment.
Revised: To decouple their impact on fairness variance, we perform the following experiment.

Original: We execute three sets ofunique training runs each while , (i) allowing for both sources of randomness, (ii) changing only the weight initialization while keeping the random reshuffling fixed, and (iii) changing only the random reshuffling while keeping the weight initialization fixed.
Revised: We execute three sets ofunique training runs each while , (i) allowing for both sources of randomness, (ii) changing only the weight initialization while keeping the random reshuffling fixed, and (iii) changing only the random reshuffling while keeping the weight initialization fixed.

Original: For ACSIncome, we collect average odds (AO) and F1 score of all training runs in Figure 2 (see Figure 13 for the other datasets).
Revised: For ACSIncome, we collect average odds (AO) and F1 score of all training runs in Figure 2 (see Figure 13 for the other datasets).

Original: 8 % Average Odds
Revised: 8 % Average Odds

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: (a) Normal training 6 % 7 % 8 %
Revised: (a) Normal training 6 % 7 % 8 %

Original: [Figure]
Revised: [Figure]

Original: However, we observe high range of fairness variance by changing only (c) the random reshuffling, even for a fixed weight initialization.
Revised: However, we observe high range of fairness variance by changing only (c) the random reshuffling, even for a fixed weight initialization.

Original: These results suggest random reshuffling of the data order as the dominant source of variance in fairness, with little influence from weight initialization.
Revised: These results suggest random reshuffling of the data order as the dominant source of variance in fairness, with little influence from weight initialization.

Original: The large range of fairness scores reported by allowing for both sources of randomness represents the variance observed in existing literature.
Revised: The large range of fairness scores reported by allowing for both sources of randomness represents the variance observed in existing literature.

Original: Interestingly, when these sources are examined separately, the variance due to random reshuffling is equivalently large but the variance due to weight initialization drops significantly.
Revised: Interestingly, when these sources are examined separately, the variance due to random reshuffling is equivalently large but the variance due to weight initialization drops significantly.

Original: It is clear that fairness variance originates from data order shuffling, while randomness in weight initialization has minimal impact.
Revised: It is clear that fairness variance originates from data order shuffling, while randomness in weight initialization has minimal impact.

Original: To further probe the difference between these two sources of non-determinism, we plot the median, inter-quartile range and overall range of average odds (AO) across the complete set of 50 training runs for the last 200 epochs in the two isolated settings from the previous experiment, i.e., changing only the weight initialization and only the random reshuffling, respectively.
Revised: To further probe the difference between these two sources of non-determinism, we plot the median, inter-quartile range and overall range of average odds (AO) across the complete set of 50 training runs for the last 200 epochs in the two isolated settings from the previous experiment, i.e., changing only the weight initialization and only the random reshuffling, respectively.

Original: The resulting plots are collected in Figure 3.
Revised: The resulting plots are collected in Figure 3.

Original: We find high correlation in fairness scores across training runs with changing weight initialization (average pairwise pearson coefficient ≈ 0 . 91 ), despite high variance between epochs as observed in
Revised: We find high correlation in fairness scores across training runs with changing weight initialization (average pairwise pearson coefficient ≈ 0 . 91 ), despite high variance between epochs as observed in

Original: Figure 3(a).
Revised: Figure 3(a).

Original: This behavior supports our observations of low variance in fairness scores at the final
Revised: This behavior supports our observations of low variance in fairness scores at the final

Original: [Figure]
Revised: [Figure]

Original: epoch in Figure 2.
Revised: epoch in Figure 2.

Original: Furthermore, there is a lack of any reasonable correlation between fairness scores of training runs with changing random reshuffling (average pairwise pearson coefficient ≈ 0 . 13 ), with noticeably high variance across both changing epochs and training runs in Figure 3(b).
Revised: Furthermore, there is a lack of any reasonable correlation between fairness scores of training runs with changing random reshuffling (average pairwise pearson coefficient ≈ 0 . 13 ), with noticeably high variance across both changing epochs and training runs in Figure 3(b).

Original: Interestingly, the high fairness variance across epochs inside a single training run in Figure 3(a) closely matches the variance that we observe across multiple training runs in Figure 3(b).
Revised: Interestingly, the high fairness variance across epochs inside a single training run in Figure 3(a) closely matches the variance that we observe across multiple training runs in Figure 3(b).

Original: 3. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION
Revised: 3. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION

Original: In previous experiments, we observed high variance in model fairness, and thus in turn it’s predictive behavior, even between consecutive epochs.
Revised: In previous experiments, we observed high variance in model fairness, and thus in turn it’s predictive behavior, even between consecutive epochs.

Original: The changing predictive behavior of neural models is not surprising, and has been studied extensively in literature (Toneva et al., 2018;
Revised: The changing predictive behavior of neural models is not surprising, and has been studied extensively in literature (Toneva et al., 2018;

Original: Kirkpatrick et al., 2017; Jagielski et al., 2022; Tirumala et al., 2022).
Revised: Kirkpatrick et al., 2017; Jagielski et al., 2022; Tirumala et al., 2022).

Original: As we are concerned with the fairness of the final decisions made by the model, we focus on a change in the model’s discrete output class when discussing changing predictions.
Revised: As we are concerned with the fairness of the final decisions made by the model, we focus on a change in the model’s discrete output class when discussing changing predictions.

Original: More specifically, a model is said to have undergone a change in prediction for some input x during epoch
Revised: More specifically, a model is said to have undergone a change in prediction for some input x during epoch

Original: i , if ˆ y i ( x ) ̸
Revised: i , if ˆ y i ( x ) ̸

Original: = ˆ y
Revised: = ˆ y

Original: i − 1 ( x ) , where ˆ y i ( x ) is the output class when passing the input x through the model checkpoint at the end of epoch i .
Revised: i − 1 ( x ) , where ˆ y i ( x ) is the output class when passing the input x through the model checkpoint at the end of epoch i .

Original: [Figure]
Revised: [Figure]

Original: While these changing predictions maintain overall stable average performance, they can still have disparate impact on individual groups, the exact characteristics of which are less known.
Revised: While these changing predictions maintain overall stable average performance, they can still have disparate impact on individual groups, the exact characteristics of which are less known.

Original: We study this instability by investigating individual data points which change their predictions.
Revised: We study this instability by investigating individual data points which change their predictions.

Original: We plot the percentage of data points from each group that changed their prediction at least once between epochs 100 and k , where we gradually increase the value of k .
Revised: We plot the percentage of data points from each group that changed their prediction at least once between epochs 100 and k , where we gradually increase the value of k .

Original: The results are collected in Figure 4.
Revised: The results are collected in Figure 4.

Original: Clearly, the trends in percentage of unique data points with changing predictions mimics the distribution of each group in the original dataset, i.e. the groups which are represented the least are the most vulnerable to changing model behavior.
Revised: Clearly, the trends in percentage of unique data points with changing predictions mimics the distribution of each group in the original dataset, i.e. the groups which are represented the least are the most vulnerable to changing model behavior.

Original: In ACSIncome dataset, positive labels from group Female is severely under represented, and consequently can be seen to have almost twice the percentage of unique examples with changing predictions than any other group in the dataset.
Revised: In ACSIncome dataset, positive labels from group Female is severely under represented, and consequently can be seen to have almost twice the percentage of unique examples with changing predictions than any other group in the dataset.

Original: The higher instability of minority is an indication of higher uncertainty in the underlying model predictions.
Revised: The higher instability of minority is an indication of higher uncertainty in the underlying model predictions.

Original: Thus, any fairness metric defined on the output of such a model will also reflect this instability, and manifests as variance in existing literature (Amir et al., 2021; Sellam et al., 2021; Soares et al., 2022).
Revised: Thus, any fairness metric defined on the output of such a model will also reflect this instability, and manifests as variance in existing literature (Amir et al., 2021; Sellam et al., 2021; Soares et al., 2022).

Original:  D ATA O RDER AND F AIRNESS
Revised:  D ATA O RDER AND F AIRNESS

Original: The previous set of experiments suggest the existence of an intricate relationship between the data order and resulting model fairness.
Revised: The previous set of experiments suggest the existence of an intricate relationship between the data order and resulting model fairness.

Original: In this section, we dig deeper into this relationship and exploit it to approximate fairness variance without separately training multiple models.
Revised: In this section, we dig deeper into this relationship and exploit it to approximate fairness variance without separately training multiple models.

Original: 4. I MMEDIATE I MPACT OF D ATA O RDER
Revised: 4. I MMEDIATE I MPACT OF D ATA O RDER

Original: A single epoch of SGD comprises of learning on a sequence of mini-batches from the training data, changing the model parameters one gradient update at a time.
Revised: A single epoch of SGD comprises of learning on a sequence of mini-batches from the training data, changing the model parameters one gradient update at a time.

Original: The data order during training governs the order of gradient updates and thus its impact on resulting model behavior is quite obvious (Bengio et al., 2009).
Revised: The data order during training governs the order of gradient updates and thus its impact on resulting model behavior is quite obvious (Bengio et al., 2009).

Original: However, our investigation into fairness variance suggests that this impact might be more immediate than expected, capable of stably predicting fairness scores irrespective of the model’s preceding behavior.
Revised: However, our investigation into fairness variance suggests that this impact might be more immediate than expected, capable of stably predicting fairness scores irrespective of the model’s preceding behavior.

Original: To test this, we design an experiment as follows.
Revised: To test this, we design an experiment as follows.

Original: We sample 1000 unique checkpoints randomly from last 200 epochs of 50 different training runs while allowing both forms of training non-determinism simultaneously.
Revised: We sample 1000 unique checkpoints randomly from last 200 epochs of 50 different training runs while allowing both forms of training non-determinism simultaneously.

Original: These checkpoints represent a variety of models with different weight initialization, training data order and even number of training epochs.
Revised: These checkpoints represent a variety of models with different weight initialization, training data order and even number of training epochs.

Original: We then train each of these checkpoints for exactly one epoch on a common randomly chosen data order.
Revised: We then train each of these checkpoints for exactly one epoch on a common randomly chosen data order.

Original: We collect fairness variance across checkpoints before and after this single epoch of training in Figure 5.
Revised: We collect fairness variance across checkpoints before and after this single epoch of training in Figure 5.

Original: [Figure]
Revised: [Figure]

Original: Before training on the same data order for a single epoch, these checkpoints represent the complete range of fairness variance previously noted.
Revised: Before training on the same data order for a single epoch, these checkpoints represent the complete range of fairness variance previously noted.

Original: However, after only a single epoch of training on a common data order, these models have all moved towards the same fairness score with significantly lower variance.
Revised: However, after only a single epoch of training on a common data order, these models have all moved towards the same fairness score with significantly lower variance.

Original: This highlights the immediate impact of data order of model fairness, which is stable based on only the data order of the most recent epoch.
Revised: This highlights the immediate impact of data order of model fairness, which is stable based on only the data order of the most recent epoch.

Original: An existing line of research suggests the presence of no energy valleys in deep learning loss landscape between minimas of separately trained models (Draxler et al., 2018;
Revised: An existing line of research suggests the presence of no energy valleys in deep learning loss landscape between minimas of separately trained models (Draxler et al., 2018;

Original: Garipov et al., 2018), which could explain the immediate impact of data order, as various checkpoints might be connected in the loss landscape.
Revised: Garipov et al., 2018), which could explain the immediate impact of data order, as various checkpoints might be connected in the loss landscape.

Original: Another possible explanation of models converging to the same fairness scores relies on a recent work that shows that there is only one functionally unique minima in loss landscape of neural models (Ainsworth et al., 2022).
Revised: Another possible explanation of models converging to the same fairness scores relies on a recent work that shows that there is only one functionally unique minima in loss landscape of neural models (Ainsworth et al., 2022).

Original: Thus, it is possible that different models when fed the same data order for even one epoch move towards a functionally common model, with possible permutation symmetries in them.
Revised: Thus, it is possible that different models when fed the same data order for even one epoch move towards a functionally common model, with possible permutation symmetries in them.

Original: Epochs are pseudo constructs, used to record multiple passes through the data, and thus we expect the influence of data order on model fairness to have a granular manifestation as the few most recent gradient updates of the model.
Revised: Epochs are pseudo constructs, used to record multiple passes through the data, and thus we expect the influence of data order on model fairness to have a granular manifestation as the few most recent gradient updates of the model.

Original: Furthermore, while we have shown that the fairness variance is indeed reduced, we are yet to show that it is also predictable.
Revised: Furthermore, while we have shown that the fairness variance is indeed reduced, we are yet to show that it is also predictable.

Original: To achieve both simultaneously, we extend the experiment above as follows.
Revised: To achieve both simultaneously, we extend the experiment above as follows.

Original: We start by extracting the data order corresponding to the epochs achieving best and worst fairness scores on the validation set in a standard training run.
Revised: We start by extracting the data order corresponding to the epochs achieving best and worst fairness scores on the validation set in a standard training run.

Original: Next, we train these checkpoints on exactly b batches, taken from the suffix of data order with best and worst fairness scores as described above.
Revised: Next, we train these checkpoints on exactly b batches, taken from the suffix of data order with best and worst fairness scores as described above.

Original: This allows us to obtain models which have experienced the same most recent b gradient updates.
Revised: This allows us to obtain models which have experienced the same most recent b gradient updates.

Original: For example, at b = 0 , this setup involves no training and we simply report the variance across the original set of checkpoints, while at b = 1070 (i.e., total number of batches in an epoch for ACSIncome), this setup will repeat the previous experiment.
Revised: For example, at b = 0 , this setup involves no training and we simply report the variance across the original set of checkpoints, while at b = 1070 (i.e., total number of batches in an epoch for ACSIncome), this setup will repeat the previous experiment.

Original: We test for b = { 0 , 20 , 50 , 100 , 200 } and the results are collected in Figure 6.
Revised: We test for b = { 0 , 20 , 50 , 100 , 200 } and the results are collected in Figure 6.

Original: [Figure]
Revised: [Figure]

Original: As the number of fixed batches b increases, the model fairness gets more stable, and already at b = 200 , the model fairness is as stable as seen in previous set of experiments.
Revised: As the number of fixed batches b increases, the model fairness gets more stable, and already at b = 200 , the model fairness is as stable as seen in previous set of experiments.

Original: This shows that it is indeed the most recent gradient updates seen by the model which dominate its fairness scores, irrespective of model behavior prior to those updates.
Revised: This shows that it is indeed the most recent gradient updates seen by the model which dominate its fairness scores, irrespective of model behavior prior to those updates.

Original: Moreover, the trends of stability move towards expected fairness based on the choice of original data order taken from best or worst fairness epoch as described earlier, which helps us show the predictability of the impact of data order.
Revised: Moreover, the trends of stability move towards expected fairness based on the choice of original data order taken from best or worst fairness epoch as described earlier, which helps us show the predictability of the impact of data order.

Original: 4. C APTURING F AIRNESS V ARIANCE IN A S INGLE T RAINING R UN
Revised: 4. C APTURING F AIRNESS V ARIANCE IN A S INGLE T RAINING R UN

Original: We now return to our original problem of capturing fairness variance without wasting computing resources on a large number of training runs.
Revised: We now return to our original problem of capturing fairness variance without wasting computing resources on a large number of training runs.

Original: We saw stability and predictability in fairness scores based on the data order, and thus fairness variance across multiple training runs is simply the randomness in data order across their last epochs.
Revised: We saw stability and predictability in fairness scores based on the data order, and thus fairness variance across multiple training runs is simply the randomness in data order across their last epochs.

Original: As these data orders are obtained by random shuffling, their distribution across multiple training runs should result in the same empirical range of fairness as their distribution across epochs.
Revised: As these data orders are obtained by random shuffling, their distribution across multiple training runs should result in the same empirical range of fairness as their distribution across epochs.

Original: Thus, we propose evaluating intermediate checkpoints in a single training run as a proxy for reporting fairness variance across multiple training runs.
Revised: Thus, we propose evaluating intermediate checkpoints in a single training run as a proxy for reporting fairness variance across multiple training runs.

Original: We test this by first plotting the distribution of fairness scores for sampling across multiple training runs (three different stopping epochs) and sampling across epochs (three different training runs) in
Revised: We test this by first plotting the distribution of fairness scores for sampling across multiple training runs (three different stopping epochs) and sampling across epochs (three different training runs) in

Original: Figure 7(a).
Revised: Figure 7(a).

Original: Furthermore, we also test the quality of black swans, i.e. the best models captured, as a function of number of unique training runs and number of epochs evaluated per training run.
Revised: Furthermore, we also test the quality of black swans, i.e. the best models captured, as a function of number of unique training runs and number of epochs evaluated per training run.

Original: For a data point epochs =
Revised: For a data point epochs =

Original: e ; seeds = s , we perform s unique training runs, and evaluate the model for last e epochs in each training run, thus accumulating a total of e ∗ s checkpoints.
Revised: e ; seeds = s , we perform s unique training runs, and evaluate the model for last e epochs in each training run, thus accumulating a total of e ∗ s checkpoints.

Original: These checkpoints are then used to calculate some quality measure, and averaged over 50 repeats to compensate for randomness in choosing s training runs.
Revised: These checkpoints are then used to calculate some quality measure, and averaged over 50 repeats to compensate for randomness in choosing s training runs.

Original: The experiment is performed for all combinations of e ∈ [1 , 50] and s ∈ [1 , 50] .
Revised: The experiment is performed for all combinations of e ∈ [1 , 50] and s ∈ [1 , 50] .

Original: We use two different quality measures for a set of checkpoints as described above, (i) the best fairness achieved across all checkpoints, and (ii) the Hausdorff distance (Birsan & Tiba, 2005) of the pareto-front (including both fairness and F1 scores) from the best achievable pareto-front, i.e., for e = 50; s = 50 .
Revised: We use two different quality measures for a set of checkpoints as described above, (i) the best fairness achieved across all checkpoints, and (ii) the Hausdorff distance (Birsan & Tiba, 2005) of the pareto-front (including both fairness and F1 scores) from the best achievable pareto-front, i.e., for e = 50; s = 50 .

Original: The results are collected in Figure 7(b).
Revised: The results are collected in Figure 7(b).

Original: [Figure]
Revised: [Figure]

Original: As we can notice from the plot in Figure 7(a), the empirical distribution of fairness across multiple training runs closely matches the distribution across a single training run.
Revised: As we can notice from the plot in Figure 7(a), the empirical distribution of fairness across multiple training runs closely matches the distribution across a single training run.

Original: We even performed the
Revised: We even performed the

Original: Kolmogorov–Smirnov (KS) test (Massey Jr, 1951) to match sampling across multiple runs (at epoch 300) and sampling across epochs (for a single training run with seed 0) and found the maximum difference in empirical CDF to be only 0 .
Revised: Kolmogorov–Smirnov (KS) test (Massey Jr, 1951) to match sampling across multiple runs (at epoch 300) and sampling across epochs (for a single training run with seed 0) and found the maximum difference in empirical CDF to be only 0 .

Original: 07 , with a p-value of 0 .
Revised: 07 , with a p-value of 0 .

Original: 712 , i.e. the probability that both set of checkpoints were sampled from the same underlying distribution.
Revised: 712 , i.e. the probability that both set of checkpoints were sampled from the same underlying distribution.

Original: Moreover, the quality of black swans in Figure 7(b) for both quality measures suggests similar trends.
Revised: Moreover, the quality of black swans in Figure 7(b) for both quality measures suggests similar trends.

Original: For example, when comparing the two extremes, i.e. e = 1; s = 50 and e = 50; s = 1 , we see similar results, with latter requiring 50 times less computation.
Revised: For example, when comparing the two extremes, i.e. e = 1; s = 50 and e = 50; s = 1 , we see similar results, with latter requiring 50 times less computation.

Original: This shows that the commonly used method to capture fairness variance in literature (i.e., e = 1; s = 50 ) is highly inefficient use of computing resources, and one can achieve the same by simply observing model behavior across a single training run.
Revised: This shows that the commonly used method to capture fairness variance in literature (i.e., e = 1; s = 50 ) is highly inefficient use of computing resources, and one can achieve the same by simply observing model behavior across a single training run.

Original: With these experiments, we show direct benefits of evaluating multiple epochs in a single training run, saving huge amounts of resources and time in capturing the overall model variance.
Revised: With these experiments, we show direct benefits of evaluating multiple epochs in a single training run, saving huge amounts of resources and time in capturing the overall model variance.

Original:  M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER
Revised:  M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER

Original: In previous sections, we explored the instability in model fairness and took advantage of our insights to reduce computational requirements for studying this variance.
Revised: In previous sections, we explored the instability in model fairness and took advantage of our insights to reduce computational requirements for studying this variance.

Original: However, the predictability of fairness scores based on the most recent batches implies the possibility that we should be able to finely manipulate the fairness scores, or more precisely the group level accuracy, by simply controlling the data order.
Revised: However, the predictability of fairness scores based on the most recent batches implies the possibility that we should be able to finely manipulate the fairness scores, or more precisely the group level accuracy, by simply controlling the data order.

Original: We hypothesize that the data distribution seen by the model in the most recent batches can temporarily change the loss landscape and nudge the group level accuracy behavior towards the chosen distribution.
Revised: We hypothesize that the data distribution seen by the model in the most recent batches can temporarily change the loss landscape and nudge the group level accuracy behavior towards the chosen distribution.

Original: Coupled with the immediate impact of data order shown in previous sections, this could allow us to manipulate group level accuracy in only a single epoch of fine-tuning.
Revised: Coupled with the immediate impact of data order shown in previous sections, this could allow us to manipulate group level accuracy in only a single epoch of fine-tuning.

Original: To test this hypothesis, we perform the following experiment.
Revised: To test this hypothesis, we perform the following experiment.

Original: We start by defining the ratio between different groups, to indicate the distribution that will dominate the model behavior.
Revised: We start by defining the ratio between different groups, to indicate the distribution that will dominate the model behavior.

Original: For example, a ratio of 1 : 10 between two groups will distinctly favor the latter group, while a ratio of 10 : 1 will favor the first group.
Revised: For example, a ratio of 1 : 10 between two groups will distinctly favor the latter group, while a ratio of 10 : 1 will favor the first group.

Original: We then form batches in the exact ratio as provided until we run out of data points (which will happen for any ratio that is not the exact distribution of the original dataset).
Revised: We then form batches in the exact ratio as provided until we run out of data points (which will happen for any ratio that is not the exact distribution of the original dataset).

Original: The excess data points are then shuffled randomly and placed at the prefix of data order.
Revised: The excess data points are then shuffled randomly and placed at the prefix of data order.

Original: Similar to previous experiment settings, we pick the last epoch checkpoint from 50 unique training runs while allowing for both forms of non-determinism (i.e. weight initialization and random reshuffling), and then train them all for a single epoch on our custom data order.
Revised: Similar to previous experiment settings, we pick the last epoch checkpoint from 50 unique training runs while allowing for both forms of non-determinism (i.e. weight initialization and random reshuffling), and then train them all for a single epoch on our custom data order.

Original: For our first experiment, we fix all other ratios and only change the ratio between positive and negative labels in group Female.
Revised: For our first experiment, we fix all other ratios and only change the ratio between positive and negative labels in group Female.

Original: We record the group level accuracy for all subgroups separately, along with the overall model accuracy.
Revised: We record the group level accuracy for all subgroups separately, along with the overall model accuracy.

Original: We repeat the experiment by now changing only the ratio between positive and negative labels in group Male.
Revised: We repeat the experiment by now changing only the ratio between positive and negative labels in group Male.

Original: The results are collected in Figure 8.
Revised: The results are collected in Figure 8.

Original: [Figure]
Revised: [Figure]

Original: It is clear that by manipulating the data distribution seen by the model in the most recent gradient updates, we can control the group level accuracy of the model.
Revised: It is clear that by manipulating the data distribution seen by the model in the most recent gradient updates, we can control the group level accuracy of the model.

Original: While the overall accuracy drops noticeably for extreme ratios, it does not change much in the middle despite significant variance in group level performances.
Revised: While the overall accuracy drops noticeably for extreme ratios, it does not change much in the middle despite significant variance in group level performances.

Original: This further strengthens our claim on the dominance of data order on model fairness and group level accuracy.
Revised: This further strengthens our claim on the dominance of data order on model fairness and group level accuracy.

Original: Fairbatch (Roh et al., 2020), a state-of-the-art bias mitigation algorithm, follows a similar formulation as our custom data order.
Revised: Fairbatch (Roh et al., 2020), a state-of-the-art bias mitigation algorithm, follows a similar formulation as our custom data order.

Original: They create batches with fixed ratio between groups, and the ratio is optimized at every epoch to counter the existing bias in the model.
Revised: They create batches with fixed ratio between groups, and the ratio is optimized at every epoch to counter the existing bias in the model.

Original: Our results not only explain the success of their algorithm, but also state that instead of regularly adapting the distribution to compensate for the model bias, one can directly use the final desired distribution and the model will adapt to it immediately.
Revised: Our results not only explain the success of their algorithm, but also state that instead of regularly adapting the distribution to compensate for the model bias, one can directly use the final desired distribution and the model will adapt to it immediately.

Original: Finally, we extract two special cases from the above experiments, i.e. ratios of 1 : 1 and 1 : 3 between positive and negative labels of group Female (for ACSIncome dataset), and call them EqualOrder and AdvOrder respectively.
Revised: Finally, we extract two special cases from the above experiments, i.e. ratios of 1 : 1 and 1 : 3 between positive and negative labels of group Female (for ACSIncome dataset), and call them EqualOrder and AdvOrder respectively.

Original: While EqualOrder achieves competitive performances with state-ofthe-art bias mitigation methods, AdvOrder is capable of derailing the fairness scores of various mitigation algorithms.
Revised: While EqualOrder achieves competitive performances with state-ofthe-art bias mitigation methods, AdvOrder is capable of derailing the fairness scores of various mitigation algorithms.

Original: Note that we can force even worse fairness gaps than achieved by AdvOrder by pushing the ratio to its extreme, however that will also noticeably impact the model’s overall accuracy.
Revised: Note that we can force even worse fairness gaps than achieved by AdvOrder by pushing the ratio to its extreme, however that will also noticeably impact the model’s overall accuracy.

Original: We perform experiments with three unique setups, standard training and two commonly used bias mitigation algorithms.
Revised: We perform experiments with three unique setups, standard training and two commonly used bias mitigation algorithms.

Original: This includes, (i)
Revised: This includes, (i)

Original: Reweighing (Kamiran & Calders, 2012), a data pre-processing algorithm which weighs every label-group pair based on its representation in the overall dataset, and (ii) Equalized Odds Loss (Fukuchi et al., 2020), an in-processing loss function to nudge the model towards fair predictions.
Revised: Reweighing (Kamiran & Calders, 2012), a data pre-processing algorithm which weighs every label-group pair based on its representation in the overall dataset, and (ii) Equalized Odds Loss (Fukuchi et al., 2020), an in-processing loss function to nudge the model towards fair predictions.

Original: We incorporate both EqualOrder and AdvOrder in these setups to see the influence of data order.
Revised: We incorporate both EqualOrder and AdvOrder in these setups to see the influence of data order.

Original: The results are collected in Figure 9.
Revised: The results are collected in Figure 9.

Original: By simply training with EqualOrder for a single epoch, the baseline models achieve competitive fairness scores to state-of-the-art bias mitigation algorithms.
Revised: By simply training with EqualOrder for a single epoch, the baseline models achieve competitive fairness scores to state-of-the-art bias mitigation algorithms.

Original: Reweighing suffers from an unexpected high bias under EqualOrder , as the combination of ideally distributed data order along with increase in the weight of minority data points pushes the model towards significant unfair behavior against the majority.
Revised: Reweighing suffers from an unexpected high bias under EqualOrder , as the combination of ideally distributed data order along with increase in the weight of minority data points pushes the model towards significant unfair behavior against the majority.

Original: On the other hand, using AdvOrder in the baseline setting can further push the model bias, emphasizing the adversarial power of data ordering.
Revised: On the other hand, using AdvOrder in the baseline setting can further push the model bias, emphasizing the adversarial power of data ordering.

Original: Both Reweighing and Equalized Odds Loss also suffers similar fates, but with controlled damage in latter due to the loss function regularly adapting to the degrading behavior.
Revised: Both Reweighing and Equalized Odds Loss also suffers similar fates, but with controlled damage in latter due to the loss function regularly adapting to the degrading behavior.

Original: These results together cement the effectiveness of manipulating group level accuracy by controlling the data order for just a single epoch of training.
Revised: These results together cement the effectiveness of manipulating group level accuracy by controlling the data order for just a single epoch of training.

Original: [Figure]
Revised: [Figure]

Original: S c
Revised: S c

Original: o r e a s f a s f a f a
Revised: o r e a s f a s f a f a

Original: [Figure]
Revised: [Figure]

Original: a
Revised: a

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e
Revised: A v e r a g e

Original: O dd s a s f a s f a f a d a f a f
Revised: O dd s a s f a s f a f a d a f a f

Original: a s f
Revised: a s f

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Variance in Fair Machine Learning Recently, there has been a growing awareness of high variance and instability in fair deep learning, associated with non-determinism in model training or underlying implementations (Amir et al., 2021;
Revised: Variance in Fair Machine Learning Recently, there has been a growing awareness of high variance and instability in fair deep learning, associated with non-determinism in model training or underlying implementations (Amir et al., 2021;

Original: Sellam et al., 2021; Soares et al., 2022;
Revised: Sellam et al., 2021; Soares et al., 2022;

Original: Qian et al., 2021), and the challenges it creates in trusting existing results.
Revised: Qian et al., 2021), and the challenges it creates in trusting existing results.

Original: The high variance in fairness scores has even been used to propose a new leave-one-out unfairness metric (Black & Fredrikson, 2021).
Revised: The high variance in fairness scores has even been used to propose a new leave-one-out unfairness metric (Black & Fredrikson, 2021).

Original: Soares et al.
Revised: Soares et al.

Original: (2022) refute various trends in literature associated with fairness of large language models by simply executing multiple identical training runs, while Sellam et al.
Revised: (2022) refute various trends in literature associated with fairness of large language models by simply executing multiple identical training runs, while Sellam et al.

Original: (2021) focus on the impact of randomness in pre-training paradigm and find further increase in instability.
Revised: (2021) focus on the impact of randomness in pre-training paradigm and find further increase in instability.

Original: Amir et al.
Revised: Amir et al.

Original: (2021) even revisit state-of-the-art bias mitigation in clinical texts and show a lack of statistically significant improvement after accounting for non-determinism in training.
Revised: (2021) even revisit state-of-the-art bias mitigation in clinical texts and show a lack of statistically significant improvement after accounting for non-determinism in training.

Original: While existing literature focuses on exploring the impact of fairness variance, we instead focus on investigating the source of this non-determinism.
Revised: While existing literature focuses on exploring the impact of fairness variance, we instead focus on investigating the source of this non-determinism.

Original: Furthermore, we propose to move away from the practice of executing multiple training runs to capture this variance and instead provide a computationally efficient proxy.
Revised: Furthermore, we propose to move away from the practice of executing multiple training runs to capture this variance and instead provide a computationally efficient proxy.

Original: Data Order and Example Forgetting Controlling the data order during training has gained noticeable success in improving convergence speeds (Bengio et al., 2009; Shah et al., 2020;
Revised: Data Order and Example Forgetting Controlling the data order during training has gained noticeable success in improving convergence speeds (Bengio et al., 2009; Shah et al., 2020;

Original: Soviany et al., 2022; Mohtashami et al., 2022), while its adversarial capabilities have also been explored (Shumailov et al., 2021).
Revised: Soviany et al., 2022; Mohtashami et al., 2022), while its adversarial capabilities have also been explored (Shumailov et al., 2021).

Original: Even under randomly shuffled data order, neural models are known to undergo changes in predictive behavior during training (Toneva et al., 2018;
Revised: Even under randomly shuffled data order, neural models are known to undergo changes in predictive behavior during training (Toneva et al., 2018;

Original: Kirkpatrick et al., 2017;
Revised: Kirkpatrick et al., 2017;

Original: Jagielski et al., 2022; Tirumala et al., 2022).
Revised: Jagielski et al., 2022; Tirumala et al., 2022).

Original: We extend the discussion on non-determinism in model behavior to model fairness, and investigate its connections with data order.
Revised: We extend the discussion on non-determinism in model behavior to model fairness, and investigate its connections with data order.

Original: We focus specifically on the instability in predictions made by the learning model and not its internal state, as we are concerned about the disparity in group level decisions which highlights the encoded bias.
Revised: We focus specifically on the instability in predictions made by the learning model and not its internal state, as we are concerned about the disparity in group level decisions which highlights the encoded bias.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Fairness variance in deep learning has raised concerns regarding the reliability of existing results in literature.
Revised: Fairness variance in deep learning has raised concerns regarding the reliability of existing results in literature.

Original: In this paper, we took a closer look at this variance and proposed a proxy to training multiple models by studying fairness scores inside a single training run, which exploits a surprisingly dominant impact of data order on model behavior.
Revised: In this paper, we took a closer look at this variance and proposed a proxy to training multiple models by studying fairness scores inside a single training run, which exploits a surprisingly dominant impact of data order on model behavior.

Original: We even manipulated the data order to minutely control group level accuracy without losing overall accuracy in only a single epoch on training.
Revised: We even manipulated the data order to minutely control group level accuracy without losing overall accuracy in only a single epoch on training.

Original: In our work, we focused only on the discrete decisions made by the model, as we were investigating the impact of non-determinism in model training on its fairness.
Revised: In our work, we focused only on the discrete decisions made by the model, as we were investigating the impact of non-determinism in model training on its fairness.

Original: However, further extensions of this discussion to trends in the internal state of the learned model can reveal even granular characteristics, and has potential application in similar fields of research suffering from high variance.
Revised: However, further extensions of this discussion to trends in the internal state of the learned model can reveal even granular characteristics, and has potential application in similar fields of research suffering from high variance.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: Our work studies various sources of non-determinism in neural model training and its impact on model fairness.
Revised: Our work studies various sources of non-determinism in neural model training and its impact on model fairness.

Original: Thus, it is important to address the implementation choices we have made to allow easy reproducibility.
Revised: Thus, it is important to address the implementation choices we have made to allow easy reproducibility.

Original: We provide a minimal psuedo code in Appendix A to clearly define various random seeds used in our setup to control training non-determinism.
Revised: We provide a minimal psuedo code in Appendix A to clearly define various random seeds used in our setup to control training non-determinism.

Original: The complete training and evaluation pipeline is deterministic outside the choice of these random seeds, i e., one can perfectly replicate the training by simply fixing these seeds.
Revised: The complete training and evaluation pipeline is deterministic outside the choice of these random seeds, i e., one can perfectly replicate the training by simply fixing these seeds.

Original: We also provide our code to train models and perform several evaluations done in the paper as supplementary material to allow easy adaptation for future research.
Revised: We also provide our code to train models and perform several evaluations done in the paper as supplementary material to allow easy adaptation for future research.

Original: Appendix Table of Contents
Revised: Appendix Table of Contents

Original: B
Revised: B

Original: CTraining and Convergence
Revised: C and Convergence

Original: D Dropout Regularization
Revised: D Dropout Regularization

Original: E Additional Experiments
Revised: E Additional Experiments

Original: E.1 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .
Revised: E.1 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .

Original: E.2 Changing Predictions and Data Distribution . . . . . . . . . . . . . . . . . . . .
Revised: E.2 Changing Predictions and Data Distribution . . . . . . . . . . . . . . . . . . . .

Original: E.3 Capturing Variance in a Single Training Run . . . . . . . . . . . . . . . . . . .
Revised: E.3 Capturing Variance in a Single Training Run . . . . . . . . . . . . . . . . . . .

Original: E.4 Manipulating Group Level Accuracy with Data Order . . . . . . . . . . . . . .
Revised: E.4 Manipulating Group Level Accuracy with Data Order . . . . . . . . . . . . . .

Original: F Additional Experiments for Changing Hyperparameters
Revised: F Additional Experiments for Changing Hyperparameters

Original: F.1 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .
Revised: F.1 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .

Original: F.2 Changing Predictions and Data Distribution . . . . . . . . . . . . . . . . . . . .
Revised: F.2 Changing Predictions and Data Distribution . . . . . . . . . . . . . . . . . . . .

Original: F.3 Manipulating Group Level Accuracy with Data Order . . . . . . . . . . . . . .
Revised: F.3 Manipulating Group Level Accuracy with Data Order . . . . . . . . . . . . . .

Original: G Additional Experiments for Other Fairness Metrics
Revised: G Additional Experiments for Other Fairness Metrics

Original: G.1 High Variance in Fairness Scores . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 High Variance in Fairness Scores . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .
Revised: G.2 Weight Initialization and Random Reshuffling . . . . . . . . . . . . . . . . . .

Original: G.3 Capturing Variance in a Single Training Run . . . . . . . . . . . . . . . . . . .
Revised: G.3 Capturing Variance in a Single Training Run . . . . . . . . . . . . . . . . . . .

Original: H 10 Raw Training Runs from Figure
Revised: H 10 Raw Training Runs from Figure

Original: A N ON -D ETERMINISM IN M ODEL T RAINING
Revised: A N ON -D ETERMINISM IN M ODEL T RAINING

Original: We provide a minimal pseudo-code to explain our control over training non-determinism and define weight initialization and random reshuffling seeds.
Revised: We provide a minimal pseudo-code to explain our control over training non-determinism and define weight initialization and random reshuffling seeds.

Original: torch.manual_seed(weight_initialization_seed) model=
Revised: torch.manual_seed(weight_initialization_seed) model=

Original: MLPModel() # Initialize Neural Model ...
Revised: MLPModel() # Initialize Neural Model ...

Original: train_data, ..
Revised: train_data, ..

Original: =
Revised: =

Original: load_dataset() # load data with deterministic order train_data = shuffle(train_data, seed=random_reshuffling_seed) ...
Revised: load_dataset() # load data with deterministic order train_data = shuffle(train_data, seed=random_reshuffling_seed) ...

Original: trainloader = torch.utils.data.
Revised: trainloader = torch.utils.data.

Original: DataLoader(train_data, ..., shuffle= True ) ... for epoch in range(total_epochs): ... torch.manual_seed(epoch) for batch in trainloader: ...
Revised: DataLoader(train_data, ..., shuffle= True ) ... for epoch in range(total_epochs): ... torch.manual_seed(epoch) for batch in trainloader: ...

Original: B D ATASETS
Revised: B D ATASETS

Original: ACSIncome ACSIncome is one of the five pre-defined tasks in the Folktables dataset Ding et al.
Revised: ACSIncome ACSIncome is one of the five pre-defined tasks in the Folktables dataset Ding et al.

Original: (2021), which was recently collected to improve the older and commonly used UCI Adult Income dataset Dua & Graff (2017).
Revised: (2021), which was recently collected to improve the older and commonly used UCI Adult Income dataset Dua & Graff (2017).

Original: More specifically, we use the subset of Folktables dataset from the state of California, USA for 2018.
Revised: More specifically, we use the subset of Folktables dataset from the state of California, USA for 2018.

Original: The dataset contains a total of 195 , 665 data points with 10 features each, where each data points represents an individual.
Revised: The dataset contains a total of 195 , 665 data points with 10 features each, where each data points represents an individual.

Original: The task is a binary classification to predict whether the individual’s income is above $50 , 000 .
Revised: The task is a binary classification to predict whether the individual’s income is above $50 , 000 .

Original: For fairness measures, we use gender (Female vs
Revised: For fairness measures, we use gender (Female vs

Original: Male) as the sensitive attribute, which is also one of the 10 input features.
Revised: Male) as the sensitive attribute, which is also one of the 10 input features.

Original: List of features : Age , Class of worker , Educational attainment , Marital status , Occupation , Place of birth , Relationship , Usual hours worked per week in past 12 months , Sex , Recoded detailed race code
Revised: List of features : Age , Class of worker , Educational attainment , Marital status , Occupation , Place of birth , Relationship , Usual hours worked per week in past 12 months , Sex , Recoded detailed race code

Original: ACSEmployment ACSEmployment is another one of the five pre-defined tasks in the Folktables dataset Ding et al.
Revised: ACSEmployment ACSEmployment is another one of the five pre-defined tasks in the Folktables dataset Ding et al.

Original: We use the same subset from the state of California, USA for 2018 as above.
Revised: We use the same subset from the state of California, USA for 2018 as above.

Original: The dataset contains a total of 378 , 817 data points with 16 features each, and the task is a binary classification to predict whether the individual is employed or not.
Revised: The dataset contains a total of 378 , 817 data points with 16 features each, and the task is a binary classification to predict whether the individual is employed or not.

Original: For fairness measures, same as above, we use gender (Female vs Male) as the sensitive attribute.
Revised: For fairness measures, same as above, we use gender (Female vs Male) as the sensitive attribute.

Original: List of features : Age , Educational attainment , Marital status , Sex , Disability recode , Employment status of parents , Mobility status , Citizenship status , Military service , Ancestry recode , Nativity , Relationship , Hearing difficulty , Vision difficulty , Cognitive difficulty , Recoded detailed race code , Grandparents living with grandchildren
Revised: List of features : Age , Educational attainment , Marital status , Sex , Disability recode , Employment status of parents , Mobility status , Citizenship status , Military service , Ancestry recode , Nativity , Relationship , Hearing difficulty , Vision difficulty , Cognitive difficulty , Recoded detailed race code , Grandparents living with grandchildren

Original: CelebA CelebA dataset is a large scale celebrity face attributes dataset Liu et al.
Revised: CelebA CelebA dataset is a large scale celebrity face attributes dataset Liu et al.

Original: (2015), which contains a total of 202 , 599 images of celebrities with 40 different binary labels each.
Revised: (2015), which contains a total of 202 , 599 images of celebrities with 40 different binary labels each.

Original: We focus on the binary classification task of the ’smiling’ label in the dataset, while we use the ’gender’ label as the sensitive attribute for fairness evaluation.
Revised: We focus on the binary classification task of the ’smiling’ label in the dataset, while we use the ’gender’ label as the sensitive attribute for fairness evaluation.

Original: Moreover, we do not directly use the images of CelebA dataset, but instead pass them through a pre-trained and frozen ResNet-50 backbone He et al.
Revised: Moreover, we do not directly use the images of CelebA dataset, but instead pass them through a pre-trained and frozen ResNet-50 backbone He et al.

Original: (2016) to extract image representations, which are treated as inputs to our model.
Revised: (2016) to extract image representations, which are treated as inputs to our model.

Original: C T RAINING C URVE AND C ONVERGENCE
Revised: C T RAINING C URVE AND C ONVERGENCE

Original: To understand why we choose to study the model behavior between epochs 100 and 300 , we point the reader towards the overall training curve of the model plotted in Figure 10.
Revised: To understand why we choose to study the model behavior between epochs 100 and 300 , we point the reader towards the overall training curve of the model plotted in Figure 10.

Original: It is clear that the model has converged by epoch 100 , and maintains stable accuracy scores for the last 200 epochs.
Revised: It is clear that the model has converged by epoch 100 , and maintains stable accuracy scores for the last 200 epochs.

Original: [Figure]
Revised: [Figure]

Original: Even though the accuracy scores have converged, we still find high variance in fairness scores as discussed in the main text of the paper.
Revised: Even though the accuracy scores have converged, we still find high variance in fairness scores as discussed in the main text of the paper.

Original: One might suspect this implies that fairness scores could take longer to converge.
Revised: One might suspect this implies that fairness scores could take longer to converge.

Original: To check this, we allow a single training to run for a total of 3000 epochs (as opposed to the standard 300 epochs used in all other experiments in our paper) and collect the fairness scores in Figure 11.
Revised: To check this, we allow a single training to run for a total of 3000 epochs (as opposed to the standard 300 epochs used in all other experiments in our paper) and collect the fairness scores in Figure 11.

Original: It is clear that the model fairness does not stabilize by simply increasing the number of epochs, which further supports our hypothesis of never-ending local oscillations due to
Revised: It is clear that the model fairness does not stabilize by simply increasing the number of epochs, which further supports our hypothesis of never-ending local oscillations due to

Original: SGD noise that cause high fairness variance.
Revised: SGD noise that cause high fairness variance.

Original: [Figure]
Revised: [Figure]

Original: D D ROPOUT R EGULARIZATION
Revised: D D ROPOUT R EGULARIZATION

Original: Another notable source of randomness in neural model training is dropout regularization (Srivastava et al., 2014).
Revised: Another notable source of randomness in neural model training is dropout regularization (Srivastava et al., 2014).

Original: Dropout regularization randomly drops a certain percentage (known as dropout rate) of connections between consecutive layers in the model at every forward pass during training.
Revised: Dropout regularization randomly drops a certain percentage (known as dropout rate) of connections between consecutive layers in the model at every forward pass during training.

Original: We extend our discussion from Section 3 to study the impact of dropout on the trends of random reshuffling seen in Figure 3.
Revised: We extend our discussion from Section 3 to study the impact of dropout on the trends of random reshuffling seen in Figure 3.

Original: More specifically, we repeat the experiments while introducing various rates of dropout in the training setup.
Revised: More specifically, we repeat the experiments while introducing various rates of dropout in the training setup.

Original: The results are collected in Figure 12.
Revised: The results are collected in Figure 12.

Original: Randomness in dropout regularization under fixed weight initialization and fixed random reshuffling (Figure12(c), (h)) introduces minute variance across multiple runs.
Revised: Randomness in dropout regularization under fixed weight initialization and fixed random reshuffling (Figure12(e), (h)) introduces minute variance across multiple runs.

Original: While it adds noise to previously noted trends, the variance in fairness across consecutive epochs in a single training run, as well as the dominance of random reshuffling on fairness variance is still clearly noticeable (Figure 12(c), (d), (f), (g)).
Revised: While it adds noise to previously noted trends, the variance in fairness across consecutive epochs in a single training run, as well as the dominance of random reshuffling on fairness variance is still clearly noticeable (Figure 12(c), (d), (f), (g)).

Original: Interestingly, the overall range of fairness variance does go down with higher dropout rate, but gradually impacts the final achieved accuracy scores (Figure 12(j), (k)) as well as fairness.
Revised: Interestingly, the overall range of fairness variance does go down with higher dropout rate, but gradually impacts the final achieved accuracy scores (Figure 12(j), (k)) as well as fairness.

Original: No
Revised: No

Original: [Figure]
Revised: [Figure]

Original: Dropout Rate = 10%
Revised: Dropout Rate = 10%

Original: [Figure]
Revised: [Figure]

Original: Dropout Rate =
Revised: Dropout Rate =

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: These results further highlight the dominant impact of random reshuffling on fairness
Revised: These results further highlight the dominant impact of random reshuffling on fairness

Original: [Figure]
Revised: [Figure]

Original: E A DDITIONAL E XPERIMENTS FOR A LL D ATASETS E. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING
Revised: E A DDITIONAL E XPERIMENTS FOR A LL D ATASETS E. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING

Original: We provide additional results in Figure 13, 14 on CelebA and ACSEmployment dataset to show the dominance of random reshuffling on model fairness.
Revised: We provide additional results in Figure 13, 14 on CelebA and ACSEmployment dataset to show the dominance of random reshuffling on model fairness.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION
Revised: E. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION

Original: We show the relationship between data distribution and changing predictions for CelebA and ACSEmployment in Figure 15.
Revised: We show the relationship between data distribution and changing predictions for CelebA and ACSEmployment in Figure 15.

Original: The group with least representation maintains being the most vulnerable to changing predictions
Revised: The group with least representation maintains being the most vulnerable to changing predictions

Original: [Figure]
Revised: [Figure]

Original: We showed that the fairness scores are dominated by the most recent gradient updates as seen by the model.
Revised: We showed that the fairness scores are dominated by the most recent gradient updates as seen by the model.

Original: As a sanity check, we also provide ablation study for experiments in Figure
Revised: As a sanity check, we also provide ablation study for experiments in Figure

Original: [Figure]
Revised: [Figure]

Original: but choose the b batches randomly instead of from the prefix.
Revised: but choose the b batches randomly instead of from the prefix.

Original: The results are collected in Figure 16.
Revised: The results are collected in Figure 16.

Original: The results show that it is indeed the batches from data order prefix that govern the predictability of model fairness
Revised: The results show that it is indeed the batches from data order prefix that govern the predictability of model fairness

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: of best and worst fairness scores as seen in
Revised: of best and worst fairness scores as seen in

Original: E. C APTURING V ARIANCE IN A S INGLE T RAINING R UN
Revised: E. C APTURING V ARIANCE IN A S INGLE T RAINING R UN

Original: We show the empirical similarity of distribution across multiple training runs and multiple epochs in a single training run for additional datasets CelebA and ACSEmployment in Figure
Revised: We show the empirical similarity of distribution across multiple training runs and multiple epochs in a single training run for additional datasets CelebA and ACSEmployment in Figure

Original: [Figure]
Revised: [Figure]

Original: reveal similar trends as seen in Figure 7.
Revised: reveal similar trends as seen in Figure 7.

Original: Fairness scores (average odds) across multiple training runs and across epochs in a single training run have similar empirical distributions.
Revised: Fairness scores (average odds) across multiple training runs and across epochs in a single training run have similar empirical distributions.

Original: Thus, studying this distribution across epochs provides a highly efficient alternative to executing multiple training runs.
Revised: Thus, studying this distribution across epochs provides a highly efficient alternative to executing multiple training runs.

Original: E. M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER
Revised: E. M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER

Original: We provide additional results in Figure 18, 19, highlighting the predictability of model fairness based on the data order
Revised: We provide additional results in Figure 18, 19, highlighting the predictability of model fairness based on the data order

Original: [Figure]
Revised: [Figure]

Original: and its impact compared against various state-of-the-art baselines
Revised: and its impact compared against various state-of-the-art baselines

Original: [Figure]
Revised: [Figure]

Original: CelebA
Revised: CelebA

Original: [Figure]
Revised: [Figure]

Original: F A DDITIONAL E XPERIMENTS FOR C HANGING H YPERPARAMETERS
Revised: F A DDITIONAL E XPERIMENTS FOR C HANGING H YPERPARAMETERS

Original: We provide additional experiments for (i) batch size 16 and 1024 , deviating from the default batch size of 128 in the main text, (ii) learning rate 0 . 01 and 0 . 0001 , deviating from the default learning rate of 0 .
Revised: We provide additional experiments for (i) batch size 16 and 1024 , deviating from the default batch size of 128 in the main text, (ii) learning rate 0 . 01 and 0 . 0001 , deviating from the default learning rate of 0 .

Original: 001 in the main text, and (iii) model architecture with two hidden layers containingand 64 neurons respectively, deviating from the single hidden layer architecture used in the main text.
Revised: 001 in the main text, and (iii) model architecture with two hidden layers containingand 64 neurons respectively, deviating from the single hidden layer architecture used in the main text.

Original: F. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING
Revised: F. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING

Original: We provide additional results in Figure 21 for changing batch size, learning rate, and architecture on
Revised: We provide additional results in Figure 21 for changing batch size, learning rate, and architecture on

Original: ACSIncome dataset to show the dominance of random reshuffling on model fairness.
Revised: ACSIncome dataset to show the dominance of random reshuffling on model fairness.

Original: With a decrease in batch size, the range of fairness variance increases significantly, but the overall expected trends follow the same behavior as noted in the main text, i.e. a sharp change in fairness scores across epochs under fixed data reshuffling, and high variance even for a single epoch across multiple runs under changing data reshuffling (fixed weight initialization).
Revised: With a decrease in batch size, the range of fairness variance increases significantly, but the overall expected trends follow the same behavior as noted in the main text, i.e. a sharp change in fairness scores across epochs under fixed data reshuffling, and high variance even for a single epoch across multiple runs under changing data reshuffling (fixed weight initialization).

Original: Similar trends can be observed when we increase the learning rate, or provide the training algorithm with a bigger neural model.
Revised: Similar trends can be observed when we increase the learning rate, or provide the training algorithm with a bigger neural model.

Original: The increase in variance suggests high instability with smaller batch size, higher learning rate, and bigger neural models, all of which is expected.
Revised: The increase in variance suggests high instability with smaller batch size, higher learning rate, and bigger neural models, all of which is expected.

Original: On the other hand, one would expect more stable model behavior with a bigger batch size or a smaller learning rate.
Revised: On the other hand, one would expect more stable model behavior with a bigger batch size or a smaller learning rate.

Original: While this is indeed the case, the comparison here is not fair because these models have not yet converged, as evident by the clear downward trend of fairness scores in Figure 21.
Revised: While this is indeed the case, the comparison here is not fair because these models have not yet converged, as evident by the clear downward trend of fairness scores in Figure 21.

Original: Note that compared to these models, the original set of models converged before epoch 100 (see Appendix
Revised: Note that compared to these models, the original set of models converged before epoch 100 (see Appendix

Original: C).
Revised: A).

Original: To further highlight the same, we also show the training curve of all 5 settings compared against the default setup in Figure 20.
Revised: To further highlight the same, we also show the training curve of all 5 settings compared against the default setup in Figure 20.

Original: It is clear that certain hyperparameter settings are not conducive to efficient convergence, even though they might eventually provide more stable fairness scores.
Revised: It is clear that certain hyperparameter settings are not conducive to efficient convergence, even though they might eventually provide more stable fairness scores.

Original: Finally, to understand the difficulty of reaching convergence, we continue training a single instance with batch size 1024 (and learning rate 0 . 0001 ) for a total of 1000 epochs and compare it against the standard training setup (i.e., with batch size 128 and learning rate 0 . 001 ).
Revised: Finally, to understand the difficulty of reaching convergence, we continue training a single instance with batch size 1024 (and learning rate 0 . 0001 ) for a total of 1000 epochs and compare it against the standard training setup (i.e., with batch size 128 and learning rate 0 . 001 ).

Original: The results are collected in Figure 22, 23.
Revised: The results are collected in Figure 22, 23.

Original: It is clear that a larger batch size (or lower learning rate) significantly slows down the convergence speed.
Revised: It is clear that a larger batch size (or lower learning rate) significantly slows down the convergence speed.

Original: Moreover, it also does not achieve the same accuracy scores previously seen, possibly due to not being able to take complete advantage of the noise usually found in mini-batch gradient descent.
Revised: Moreover, it also does not achieve the same accuracy scores previously seen, possibly due to not being able to take complete advantage of the noise usually found in mini-batch gradient descent.

Original: Thus, using a high number of training epochs for a model with hyperparameters that enforce stability (i.e. large batch size or lower learning rate) is an inefficient solution to solving the problem of fairness variance.
Revised: Thus, using a high number of training epochs for a model with hyperparameters that enforce stability (i.e. large batch size or lower learning rate) is an inefficient solution to solving the problem of fairness variance.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: g e O
Revised: g e O

Original: [Figure]
Revised: [Figure]

Original: Model Architecture = {
Revised: Model Architecture = {

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION
Revised: F. C HANGING P REDICTIONS AND D ATA D ISTRIBUTION

Original: We note changing predictions for all 5 settings described above in Figure 24.
Revised: We note changing predictions for all 5 settings described above in Figure 24.

Original: Note that since we are focusing on changing training hyperparameters for ACSIncome, the data distribution remains the same as in Figure 4(a) (also copied here in Figure 24 for reference).
Revised: Note that since we are focusing on changing training hyperparameters for ACSIncome, the data distribution remains the same as in Figure 4(a) (also copied here in Figure 24 for reference).

Original: Same as before, model instability has increased with smaller batch size, higher learning rate, or bigger model architecture, but the trends of vulnerability for various subgroups remains the same.
Revised: Same as before, model instability has increased with smaller batch size, higher learning rate, or bigger model architecture, but the trends of vulnerability for various subgroups remains the same.

Original: We already know that the models have not converged for higher batch size or smaller learning rate
Revised: We already know that the models have not converged for higher batch size or smaller learning rate

Original: [Figure]
Revised: [Figure]

Original: but we can still see the same trends of higher vulnerability for minority, although they are not as pronounced as in already converged models
Revised: but we can still see the same trends of higher vulnerability for minority, although they are not as pronounced as in already converged models

Original: [Figure]
Revised: [Figure]

Original: ACSIncome Data Distribution
Revised: ACSIncome Data Distribution

Original: Model Architecture = { 2048, 64 }
Revised: Model Architecture = { 2048, 64 }

Original: [Figure]
Revised: [Figure]

Original: similar trends as seen in Figure 4.
Revised: similar trends as seen in Figure 4.

Original: These results highlight subgroups with least representation being the most vulnerable to changing predictions.
Revised: These results highlight subgroups with least representation being the most vulnerable to changing predictions.

Original: F. M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER
Revised: F. M ANIPULATING G ROUP L EVEL A CCURACY WITH D ATA O RDER

Original: We provide additional results in Figure 25, highlighting the predictability of model fairness based on the data order.
Revised: We provide additional results in Figure 25, highlighting the predictability of model fairness based on the data order.

Original: Note that the hyperparameter setting for an additional epoch of fine-tuning during group accuracy manipulation is the same as the setup used for training that particular model.
Revised: Note that the hyperparameter setting for an additional epoch of fine-tuning during group accuracy manipulation is the same as the setup used for training that particular model.

Original: For example, when manipulating models which were trained with batch size 16 , the single epoch of fine-tuning is also done with batch size 16 .
Revised: For example, when manipulating models which were trained with batch size 16 , the single epoch of fine-tuning is also done with batch size 16 .

Original: [Figure]
Revised: [Figure]

Original: Rate =
Revised: Rate =

Original: [Figure]
Revised: [Figure]

Original: Model Architecture = { 2048, 64 }
Revised: Model Architecture = { 2048, 64 }

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: We also provide results for models with high batch size and low learning rate separately in Figure 26.
Revised: We also provide results for models with high batch size and low learning rate separately in Figure 26.

Original: However, note that not only are these models not converged, but they are also fine-tuned on the same inefficient hyperparameter settings.
Revised: However, note that not only are these models not converged, but they are also fine-tuned on the same inefficient hyperparameter settings.

Original: Thus, the trends here are not comparable, but added for completeness.
Revised: Thus, the trends here are not comparable, but added for completeness.

Original: [Figure]
Revised: [Figure]

Original: G A DDITIONAL E XPERIMENTS FOR O THER F AIRNESS M ETRICS
Revised: G A DDITIONAL E XPERIMENTS FOR O THER F AIRNESS M ETRICS

Original: We repeat the experiment in the main text for two more fairness metrics, Equal Opportunity (EOpp) and Disparate Impact (DI) Hardt et al.
Revised: We repeat the experiment in the main text for two more fairness metrics, Equal Opportunity (EOpp) and Disparate Impact (DI) Hardt et al.

Original: For model predictions R , true labels Y
Revised: For model predictions R , true labels Y

Original: [Figure]
Revised: [Figure]

Original: and sensitive attributes A , EOpp can be defined as,
Revised: and sensitive attributes A , EOpp can be defined as,

Original: EqualOpportunity := | P
Revised: EqualOpportunity := | P

Original: R =
Revised: R =

Original: [Equation]
Revised: [Equation]

Original: | Y
Revised: | Y

Original: [Equation]
Revised: [Equation]

Original: 1 , A
Revised: 1 , A

Original: [Equation]
Revised: [Equation]

Original: 0) − P ( R =
Revised: 0) − P ( R =

Original: [Equation]
Revised: [Equation]

Original: | Y = 1 , A = 1) | .
Revised: | Y = 1 , A = 1) | .

Original: Similarly, DI can be defined as,
Revised: Similarly, DI can be defined as,

Original: DisparateImpact
Revised: DisparateImpact

Original: [Equation]
Revised: [Equation]

Original: G. H IGH V ARIANCE IN F AIRNESS S CORES
Revised: G. H IGH V ARIANCE IN F AIRNESS S CORES

Original: We start by repeating the results in Figure 1 to show high variance in different fairness measures.
Revised: We start by repeating the results in Figure 1 to show high variance in different fairness measures.

Original: [Figure]
Revised: [Figure]

Original: G. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING
Revised: G. W EIGHT I NITIALIZATION AND R ANDOM R ESHUFFLING

Original: We provide additional results for EOpp and DI for the experiment conducted in Figure 3 on ACSIncome dataset.
Revised: We provide additional results for EOpp and DI for the experiment conducted in Figure 3 on ACSIncome dataset.

Original: The results collected in Figure 28 show similar trends as seen in the main text.
Revised: The results collected in Figure 28 show similar trends as seen in the main text.

Original: [Figure]
Revised: [Figure]

Original: G. C APTURING V ARIANCE IN A S INGLE T RAINING R UN
Revised: G. C APTURING V ARIANCE IN A S INGLE T RAINING R UN

Original: We show the empirical similarity of distribution across multiple training runs and multiple epochs in a single training run for fairness measures EOpp and DI in Figure
Revised: We show the empirical similarity of distribution across multiple training runs and multiple epochs in a single training run for fairness measures EOpp and DI in Figure

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: H 10 R AW T RAINING R UNS FROM F IGURE
Revised: H 10 R AW T RAINING R UNS FROM F IGURE

Original: We plot 10 randomly chosen training runs each for fixed weight initialization and fixed random reshuffling in Figure 30 and Figure 31, respectively.
Revised: We plot 10 randomly chosen training runs each for fixed weight initialization and fixed random reshuffling in Figure 30 and Figure 31, respectively.

Original: As expected, each individual training run in both settings has high variance across epochs even after convergence.
Revised: As expected, each individual training run in both settings has high variance across epochs even after convergence.

Original: More importantly, the trends of fairness matches closely across multiple runs for fixed random reshuffling, even though they started from different weight initialization.
Revised: More importantly, the trends of fairness matches closely across multiple runs for fixed random reshuffling, even though they started from different weight initialization.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e O dd s
Revised: A v e r a g e O dd s

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] A v e r a g e O dd s

Original: [Figure] 
Revised: [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e O dd s
Revised: A v e r a g e O dd s

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] A v e r a g e O dd s
Revised: [Figure]

Original: [Figure][Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e O dd s
Revised: A v e r a g e O dd s

Original: [Figure]
Revised: [Figure]

Original: A v e r a g e O dd s
Revised: A v e r a g e O dd s