Replication: Fairness without demographics through Adversarially Reweighted Learning
Anonymous Author(s)
Reproducibility Summary
Scope of Reproducibility
We test the claim that Adversarially Reweighted Learning (ARL) improves Rawlsian Max-Min fairness for supervisedclassiﬁcation compared to previous methods and simple baselines in the case of missing demographic data.
Methodology
We completely re-implemented all models and training routines in PyTorch, using the paper and the published code as areference.
We compared our implementation to the one provided by the authors and then reproduced the hyperparametersearch as described in the paper using our implementation.
In addition, we applied the method to image data, in order totest how well it generalizes across modalities.
Due to the general simplicity of the used models, it was straightforward to implement the code base from scratch andrun the experiments.
Parts of the experiments were run on a computing cluster with 12 CPUs and parts on Google
Colab (GPU).
Overall, it took 4 weeks to produce the results, with four people working on it.
A complete grid searchtook 4 hours and producing the ﬁnal results with ﬁxed hyperparameters 5 hours, of which a single training run of onemodel on one dataset took about 2 minutes.
Results
We could not replicate the advantage of ARL over the investigated baselines.
This seems to be mainly due to a betterbaseline performance than reported in the paper.
Our baseline’s performance is on average 2.615 standard deviationshigher than the authors’.
Our ARL results do not deviate signiﬁcantly from the papers’ result, they are on average 0.841standard deviations higher.
What was easy
ARL itself was very easy to implement.
We were also able to run the code provided by the authors quite easily.
Runningthe experiments required little computational resources because of the small datasets and models.
What was difﬁcult
Pre-processing the data took time because the notebooks provided by the authors contained some errors that we neededto debug.
For the replication of the grid search for hyperparameter optimization of all models on all dataset, we had tolimit the training duration to a maximum of 5k steps in order to ﬁnish all experiments on time
Communication with original authors
We asked the authors about details regarding their training procedure.
The authors provided us with the missing details and adapted their GitHub repository as a response to our communication
Introduction
As machine learning (ML) systems are increasingly often applied to high-stakes situations, it is becoming moreimportant to ensure that they do not discriminate against groups of individuals based on features such as race and sex.
Lahoti et al.
present an approach called Adversarially Reweighted Learning (ARL) that aims to improve a model’sperformance measured according to the Rawlsian Max-Min fairness principle by reformulating the training procedureas an adversarial min-max game [1].
While previous work mainly addresses the case where protected features are partof the dataset, ARL does not require such information.
As Lahoti et al.
point out, such an approach is important becauseinformation about group membership is often sensitive and not available in many real-world scenarios, for example dueto privacy concerns.
1.1 Replication of Main Experiments
The main empirical claim of Lahoti et al.
is that ARL outperforms several other methods across several (tabular)datasets in terms of both performance and fairness.
Their most important comparison models are Distributionally robustoptimization (DRO) [2], inverse probability weighting (IPW) and a simple empiricial risk minimization baseline.
Thefocus of our reproducibility study is therefore whether we can replicate an advantage of ARL over these methods.
We implemented all of the methods ourselves in PyTorch [3].
For ARL, IPW and the baseline, we mostly relied on thedescription in the paper and referred to the authors’ TensorFlow [4] code where necessary.
For DRO, we adapted thepublicly available implementation to our framework.
Our own code is available in the supplementary material.
Following the paper, we performed a grid search over hyperparameters.
We report the performance of our ownimplementation on the same datasets as used by the authors, namely UCI Adult, LSAC and COMPAS.
Based on ourown results and the process of obtaining these, we evaluate the replicability of the paper.
We pay special attention to thefollowing criteria:
• Are our own results in line with the results reported by the authors?
• What challenges did we meet when replicating the authors’ method?
What
differences exist between the authors’ paper and their code?• Can we support the explicit and implicit assumptions made by the authors to substantiate their method?
1.2 Additional Contributions
Given the tabular nature of the UCI Adult, LSAC and COMPAS datasets, the authors apply their method to fullyconnected networks.
Adversarial reweighting is, however, not limited to fully-connected networks, but serves as ageneral framework to modify optimization such that worst-case performance over unobserved protected groups isimproved.
We therefore applied the method to image data, in order to test how well it generalizes across modalities.
Forthis, we trained a convolutional ARL on a custom image dataset based on EMNIST.
Methodology
2.1 Model Descriptions
ARL, DRO and IPW have in common that they modify the optimization objective in order to improve fairness.
Eventhough the ARL introduces an additional adversary network to dynamically learn the exact form of this modiﬁcation,the network architecture of the inference model is identical for all different methods.
(Note that this does not transfer tothe adversary network of ARL, which is not used for inference.)
For training on the UCI Adult, LSAC and COMPAS datasets, we used a simple fully-connected feed forward neuralnetwork.
As in the paper, the network consists of two hidden layers with 64 and 32 units respectively, and ReLUactivation functions applied after each hidden layer.
For the given binary classiﬁcation tasks, the output layer consistsof one output unit.
Adversarially Reweighted Learning (ARL)
The key component to ARL is its adversarial setting.
Concretely, a so-called learner is trained to minimize a taskspeciﬁc loss function.
Alternatingly, a so-called adversary is trained to reweight the loss terms that are produced by the learner for each individual training sample in order to maximize the overall loss.
Using θ and φ to respectively denote the parameters of the learner and the adversary, the overall training objective for n training samples is formulated as:
[Equation]
where (cid:96) : ˆ Y × Y → R denotes the loss function of the underlying task and h θ : X → ˆ Y denotes the function that is implemented by the learner.
The weights λ φ ( X, Y ) are produced by the adversary as follows:
[Equation]
where f φ ( x i , y i ) is the adversary’s output.
During training, the adversary implicitly learns to identify areas in theinput space that lead to high losses (cid:96) .
By assigning higher weights to samples from these areas, it drives the learnertowards improving in the identiﬁed subspaces.
The authors claim that this ultimately leads to improved performance forprotected groups whose sensible characteristic features are not included in the input data X , i.e. the groups that areidentiﬁed by the adversary correspond to a certain degree to the groups that are to be protected.
We implemented the adversary with one linear layer, followed by a sigmoid function and the normalization, scaling andshifting as stated above.
Instead of using the sum of single reweighted losses, we used the mean, following the authors’code.
In their code they used two consecutive linear layers, which is not perfectly in line with the paper.
Their code alsoreveals that the learner network was pretrained, which they do not mention in their paper.
Empiricial Risk Minimization (Baseline)
This model is identical to the learner in ARL.
The authors note that they increased the number of hidden units in thismodel to compensate for the added capacity in form of the adversary in ARL.
However, it was neither clear from thepaper how they implemented this, nor evident from their code that they actually did.
Going by parameter count, asingle additional unit would already result in more parameters than for ARL.
Since the simple fully-connected modeldid already match the performance of ARL (see Table 1), we did not investigate the effect of adding more units.
Thefully-connected model will also be referred to as ’baseline’ in the rest of the text.
Inverse Probability Weighting (IPW)
Instead of learning the weights to reweight the loss function dynamically, a simpler approach is to weight each samplewith the inverse probability of drawing this sample from the dataset.
This gives underrepresented groups in the dataseta larger weight.
In the IPW(S) approach, the probabilities are calculated based only on the membership to protectedgroups (e.g. "sex" or "race"), resulting in weights of the form 1 /p ( s ) .
Additionally to the protected group we cantake into account the label, resulting in a model called IPW(S+Y).
Here, the probabilities are calculated as the jointprobabilities of being in a certain protected group and belonging to a certain label (e.g. "being black" and "earningmore than ﬁfty thousand dollars a year").
This results in weights of the form 1 /p
( s, y ) .
IPW can only be used if group probabilities are known.
It therefore offers a comparison of the ARL’s claim of improvingfairness without demographics to a method that improves fairness with demographics.
Distributionally Robust Optimization (DRO)
Similar to ARL, DRO aims to improve Rawlsian Max-Min Fairness for tasks where protected groups are unknown.
Hashimoto et al.
propose the following optimization objective:
[Equation]
P denotes the data generating distribution, (cid:96) ( θ ; Z ) the loss of a query Z under the model with parameters θ , andη a hyperparameter to threshold the losses.
In the publicly available code by Hashimoto et al.
that we used in ourimplementation, a ReLU activation function is applied after subtracting η from losses of single data points, ensuring thatonly positive losses remain.
The method focuses on the worst-case group by only considering losses for optimizationthat exceed the threshold η , which implicitly decides about the size of the worst-case group.
Squaring the thresholdedlosses upweights the inﬂuence of higher losses on the learning objective further.
2.2 Datasets
We use the three datasets also used in the paper: UCI Adult [5], COMPAS [6] and LSAC [7], all of which are publiclyavailable.
Every dataset describes a binary prediction task.
In UCI Adult, the income of a person is to be predicted to beabove ﬁfty thousand dollars.
COMPAS is concerned with predicting the recidivism of indviduals convicted of crimes.
For LSAC, the task is to predict whether a given law student will pass the bar exam.
All datasets exhibit some form of imbalance and include information about sensitive attributes like sex and race whichmakes them suitable for evaluating model fairness.
UCI Adult consists of 48,842 records, of which 67% are male and85% white (10% black).
The dataset is also imbalanced in its target class, as only 24% have an income of more thanﬁfty thousand dollars.
LSAC consists of 26,551 samples, of which 56% are male and 83% white (6% black).
COMPASconsists of 7,185 records with 81% being male and 37% white (51% black).
In all datasets, there exists some correlationbetween sensitive attributes and target class [8, 7, 9].
Pipeline
We split each dataset randomly into 70% training data and 30% test data, except for UCI Adult, where a split is alreadyprovided.
For our main experiments, the sensitive columns race and sex were removed from the features, such that thesedemographics were not available to the models.
To access sensitive information during evaluation, each data elementwas categorized as being a member of one of the following groups: "not black and male", "not black and female","black and male", "black and female".
Of these, the group with the fewest members is referred to as minority.
Thistreatment of the subgroups was inferred from the publicly available code.
The paper however used the term "white"instead of "not black", which does not accurately reﬂect the work done in their code.
We will adopt their term usagefrom hereon for better comparability.
We represented categorical features using one-hot encoding and normalized numerical features to have zero mean andunit standard deviation.
All features of a datapoint were then appended into one feature vector.
A script to downloadand preprocess the datasets is available as part of our code base.
2.3 Hyperparameters
We determined the optimal batch size and learning rate for each method using a grid search and 5-fold cross validationover the same search space as described in the appendix to the paper.
All results were obtained by training from scratch with the obtained optimal hyperparameters using ten different randomseeds, then evaluating on the test set and averaging the results.
We pretrained the learner for ARL for 250 steps, whichis the default in the code provided by the authors.
We used AdaGrad as the optimizer, following the paper.
Initializationfor the weights and the optimizer were chosen to agree with TensorFlow defaults.
In the initial experiments, we raneach grid search for 5000 training steps, with early stopping if the overall AUC on the validation set had not improvedfor 10 epochs.
For the grid search space and the optimal hyperparameters found, see Appendix C. For the hyperparameters obtained bythe authors and a comparison to the hyperparameters found in our grid search, see Appendix D.
2.4 Computational Requirements
The grid searches took about 3-4h on twelve CPUs for the 5k step runs on tabular data and an additional 10h on a GPUfor the image grid searches.
The 100k step grid seach took much longer, about 1-2 days.
The ﬁnal results (training eachmodel on ten different random seeds) were obtained from a notebook that ran about 5h on Google Colab with a GPU.
Results
Table 1 shows the results of comparing ARL to DRO and the baseline model.
We observe that all methods obtain verysimilar results for each dataset and metric.
Many of the differences that do exist between methods are not signiﬁcant,and in any case which methods performs best varies strongly between metrics and datasets.
Table 2 shows the relative deviation of our results in Table 1 to those reported in the paper.
Positive numbers mean thatour performance was higher, negative that it was lower than that from the paper.
All numbers are measured in multiplesof the standard error of the difference, calculated as (cid:112) σ 2 ours + σ 2 theirs , where σ ours and σ theirs are the standard deviationsof our and their results over the 10 runs with different seeds.
Because Lahoti et al.
do not report standard deviations formacro-avg, minimum and minority AUC, we calculated the errors for those measure using error propagation from the values they report for all protected subgroups (tables 6 to 8 in their supplementary material).
Overall, all three methodstend to have a slightly better performance in our results than in the paper. But for ARL, this effect is smaller than forthe baseline and in particular than for DRO. This explains why in our results, ARL doesn’t have an advantage like itdoes in the paper: it doesn’t perform any worse, but the comparison methods perform better.
[Table]
[Table] We also ran the code provided by the authors with their optimal hyperparameters (results in Appendix A). There, we getresults much closer to those reported in the paper and in particular a (very slight) advantage for ARL. Additionally, wedirectly compared our Pytorch implementation with the original Tensorﬂow implementation (Appendix B) and foundthat our implementation tended to perform somewhat better.
We will discuss these various discrepancies further below. Since some training runs did not end via early stopping but rather due to the maximum number of training steps beingreached, we repeated the grid search and evaluation with ten seeds using a maximum number of 100k instead of 5ksteps.
We did not observe a signiﬁcant increase in performance for any of the used methods, so we can probably ruleout that training for too few training steps distorted our results. The exact results for 100k training steps can be found in Table 11. In the remainder of the main body of this report we use 5k training steps if not stated otherwise.
[Figure]
Figure 1 shows the micro-average AUC, minimum group AUC and the AUC of the minority group of ARL, baseline and
DRO on the validation split of the LSAC dataset.
As can be seen, ARL shows no consistent advantage in the minimum group AUC and minority AUC, and a neglectable advantage in the micro-average AUC.
Figures 1b and 1c show that,depending on the point in time where the models are saved, there exist some points of comparison where ARL seems tooutperform the other models.
Given these ﬂuctuations, it is possible that the differences between the authors’ and ourresults can be explained by different points of comparison.
We used early stopping and compared the best-performingmodels, whereas the authors used a ﬁxed number of training steps.
In Table 3, we compare the results between ARL and the naive reweighting approach IPW, either using only protectedgroups (S) or using protected groups and labels (S+Y) to compute the inverse probability weights.
Similarly to theresults in Table 1, ARL does not show an advantage over other methods on any of the datasets.
[Table]
3.1 Computational Identiﬁability
[Table]
To investigate whether a linear adversary is actually capable of identifying protected groups based on the observed features and labels, theauthors trained a linear model to predict race or sex based on the features and the labels.
They did not provide further details about theirtraining method.
Since we only want to test whether the adversaryis in principle capable of performing this task and precise accuraciesare thus not important, we limit ourselves to a single run.
Training, validation and testing are performed with the optimal hyperparameters found via the grid search in the previous section.
Theobtained test accuracies for predicting race or sex are shown in Table 4.
The results are broadly consistent with the results from the original paper.
The per-group accuracies show that in many cases, the linear model achieves its performance by concentrating on themajority class, or even just always predicting the majority class. This raises doubts on whether the adversary is indeedable to adequately identify and upweight minority groups.
3.2 Adversary Outputs
To inspect whether the adversary learns meaningfulweights, the authors plotted the weights returned by theadversary on the training set of UCI Adult.
We evaluatethe adversary on the test set of UCI Adult to see whetherit has meaningful weights on the test set as well.
Weperform kernel density estimation with a Gaussian kernelwith a bandwidth parameter of 0.3 on the test weights toproduce continuous distributions over λ .
Results obtainedusing scikit-learn [10] are shown in Figure 2.
The computed densities roughly match the authors’ results.
Thus, we can support their claim that ARL learnsto upweight not only misclassiﬁed examples, but also underrepresented classes (bottom-right quadrant of Figure 2.
Recall that the UCI Adult dataset exhibits a strong classimbalance and as we can see the adversary places largerweights on the less frequent class.
[Table] It is important to point out that ARL apparently does notplace higher weights on underrepresented demographics (see the orange and red curves in Figure 2).
This is in line
withthe results from Lahoti et al.
but does not agree with the original motivation for ARL.
Extension of the Presented Method
Not just tabular data, but also other modalities are in principle prone to issues regarding fairness.
This is why anevaluation of ARL on a different type of data marks a straightforward extension to the analyses presented by Lahotiet al.
In this section, we present and evaluate a simple extension of ARL to image data.
4.1 Dataset
We resorted to building a custom dataset for the evaluation in order to meet our limited computational resources andmaintain the ability to easily tailor the input data to our experiments.
The dataset is based on balanced EMNIST [11], acharacter classiﬁcation dataset similar to MNIST [12] containing a balanced set of 47 different greyscale characters.
Wecustomized the dataset, which we refer to as EMNIST_35, such that the task is a binary prediction with balanced labels,and artiﬁcially created one protected minority subgroup.
The minority is represented by 35% of the data for both labels.
Further details and different data conﬁgurations are presented in Appendix G.1.
4.2 Model Architecture
Both the learner of ARL and the baseline model contain a single convolutional layer with 64 ﬁlters and a 3 × 3 kernelwith stride 1.
Subsequently, a max-pooling layer with kernel size 2 × 2 with stride 1 is employed.
The adversary uses asimilar setup with 32 ﬁlters.
Both learner and baseline then use two fully connected layers with 64 and 32 neurons witha ReLU activation function in-between.
In the adversary, the sample label is appended to the ﬂattened output of theconvolutional section and the model output is produced as a linear combination of the resulting vector for each sample.
The training procedure and hyperparameters of all image models are further described in Appendix G.3.
4.3 Results and Analysis
The model performances regarding the different AUC metrics and the classiﬁcation accuracy are listed in Table 5.
Thebaseline clearly outperforms ARL on all evaluation criteria.
Further experiments showed that the adversary is unable toidentify protected groups in the adversarial setting and tends to weight few individual samples with very high weights.
Since the architecture for the learner and the baseline is the same, we further considered the inﬂuence of the adversaryby investigating if it is capable of identifying the protected groups in a supervised setting.
It turns out that the adversaryresorts to predicting the majority class in most cases.
We ran several experiments with different architectures for theadversary but were not able to produce a performance boost over the reference models similar to those reported by
Lahoti et al.
on tabular data.
Details regarding the results of our experiments are given in Appendix G.
Discussion
In this report we re-implemented the Adversarially Reweighted Learning (ARL) framework by Lahoti et al.
in PyTorch.
This framework promises a general approach to dealing with group fairness if demographic data are lacking or unknown.
However, as shown in Tables 1 and 3 and the analyses that followed, we could not replicate the advantage of ARLover the investigated baselines.
This seems to be mainly due to a better baseline performance than in Lahoti et al.
Asdisplayed in Table 2, our ARL results are not deviating signiﬁcantly (mostly < 3 σ ) from the papers’ result, but ourbaselines are considerably better than in the original paper.
One possible cause for this is that we used early stopping, whereas the authors of the original paper didn’t, as far astheir code and paper suggest. This could mean that the number of training steps (which was ﬁxed across methods) happened to work better for ARL than it did for the other methods.
In our setup, the training duration is effectivelychosen optimally for each method, which in our view makes for a fairer comparison. A potential reason why ARL doesn’t outperform a simple risk-minimizing baseline is that ARL apparently does notactually upweight minorities, as we showed in Figure 2.
As mentioned, this is entirely consistent with the ﬁgures in theoriginal paper but does not agree with the motivation for ARL.
Perhaps the linear adversary is too weak to be effective –our identiﬁability results in Table 4 suggest that it mainly learns to exploit the class imbalance. We also point out thatthe advantage of ARL reported in the original paper is not particularly large, neither in absolute terms nor compared tothe errors. This reinforces the idea that the difference between our and the original results is caused by relatively smallchanges in training procedure, such as the use of early stopping.
Still, ARL is theoretically appealing and our results don’t rule out that it might have beneﬁts in other circumstances.
Perhaps the datasets chosen by the authors are simply not hard enough to differentiate between the various methods:the baseline model is already quite fair towards subgroups, not leaving much space for ARL to show any differentialadvantage.
Experiments by Zhong suggest that reweighting and oversampling, both methods related to ARL, are noteffective at improving fairness in these datasets [13], which further supports our impression of the datasets not beingsuitable.
Our experiments with ARL on image data show that such an extension is not trivial, at the very least.
Includingan adversary lead to deteriorating performance compared to the baseline which could be explained by real-world imagesgenerally containing large amounts of noise.
An extensive investigation regarding this and other potential sources oferror are left as future work.
On the positive side, we found it relatively easy to run the code provided by the authors and to re-implement theirmethod.
There were some issues regarding the dataset preparation and slight differences between the description of thealgorithms in the paper and their actual implementation in the code, but overall, none of these posed serious obstacles toreplicating the paper.
In conclusion, we did not ﬁnd an advantage of ARL over baseline models, neither for the datasets used by the authosnor for image data.
Given that the results of the original paper favor ARL only very slightly and that the adversaryapparently does not weight minorities higher, we do not believe that ARL in its current form can improve fairness onthese datasets.
However, since the theoretical motivation appears sound, it may be promising to apply ARL to datasetswhere baselines suffer from more severe fairness issues, or to further investigate whether the capacity of the adversarycan be adjusted to improve performance. A Results using the original setup
We used the code and the optimal hyperparameters provided by the authors to reproduce their experiment.
This shouldyield results very close to those reported in the original paper, though slight deviations are to be expected since theoriginal code doesn’t use random seeds.
We again averaged our results over 10 runs with different random seeds andcalculated the standard deviation over those runs as the error. The error for the authors’ results was again calculatedbased on the values reported in Appendix 8.2 of the original paper. Because the paper doesn’t report any errors for the IPW results, we only compare the baseline and ARL results here.
Furthermore, we only use the AUC metrics, since thepaper doesn’t report any accuracies.
Table 6 shows the results we obtained using this setup.
In Table 7, we show the relative deviations of these results tothose obtained in the paper.
Deviations that are signiﬁcant at the 2 σ level are bolded.
While most of the results are compatible, there are a few with signiﬁcant differences (sometimes our results are worseand sometimes better than those reported in the paper).
We are uncertain what causes these differences – perhaps aslight difference in the data preparation, since the provided notebook didn’t run without errors and we had to implementparts of it ourselves.
However, as Table 6 shows, these deviations don’t change the overall conclusion: just as in the paper, ARL outperformsthe basline for the Adult and LSAC datasets, while tending to perform worse on the COMPAS dataset.
In most cases,the differences between baseline and ARL are relatively small, and sometimes not even signiﬁcant at the 1 σ level.
[Table]
[Table]
B Comparison of Implementations
We compared the Tensorﬂow implementation by the original authors to our Pytorch implementation using exactly thesame settings, to check whether they are equivalent.
Our results can be found in Table 8.
There are some signiﬁcantdifferences, even though we took care to use Tensorﬂow defaults for initialization scheme and optimizer settings.
Oneremaining difference is that the authors implemented the linear layer of the adversary as two consecutive linear layers.
This doesn’t change the expressiveness but does affect the training dynamics.
However, this is only relevant for ARL,and there are signiﬁcant differences even in the baseline model.
While we were unable to ﬁnd the cause for these deviations, our implementation tends to perform sightly better incases where there is any difference at all.
So it is unlikely that we made an implementational error that would affect thevalidity of our results.
[Table]
C Optimal Hyperparameters
We performed grid search to determine optimal hyperparameters.
The search space for (batch size, primary learningrate, adversary learning rate, η ) consisted of the following values:[32 , 64 , 128 , 256 , 512] ×
1 , 1 , 2 , 5] × 1 , 1 , 2 , 5] × [0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1] , where in the case of methods other than DRO, the η parameter was not searched over and in case of methods other than ARL, the adversary learning rate was not searched over.
Table 9 shows the optimal hyperparameters for each datasetand model combination, when optimized for overall AUC.
[Table]
D Comparison to authors’ hyperparameters
The authors report different optimal hyperparameters than those that we found.
We ﬁrst reran our experiments usingtheir hyperparameters.
The results are given in Table 10.
They are comparable and if anything slightly worse than thosewe obtained with our hyperparameters
[Table]
and ARL does not have any advantage there either
[Table]
To understand why the reported hyperparameters differ [Table] we also investigated how important the choice of hyperparameters is after all. Figure 3 shows a histogram of how often each AUC occured during the grid search.
For Adult and COMPAS, we cansee that most hyperparameter combinations yield essentially equally good results, all close to the top performance. For [Table] [Figure]
LSAC, the distribution is wider but there are still several hyperparameter combinations with very good results. So giventhe deviations that exist between the authors’ results and ours, it is not surprising that we also happen to get differenthyperparameters.
This doesn’t necessarily have any deep reason, but might just be due to random differences, as Fig.
3suggests.
E AUC learning curves
Figure 4 shows the AUC learning curves of ARL, IPW(S) and IPW(S+Y) on the validation set of LSAC. As can be seenfrom the learning curves, IPW(S+Y) consistently outperforms ARL on the LSAC dataset.
[Figure]
F Results for 100k maximum training steps
As can be seen from Tables 11 and 12, changing the maximum number of training steps from 5k to 100k did not changeperformance signiﬁcantly.
We did not have enough computational resources to train the DRO model on the Adultdataset
[Figure]
but given the rest of the results it seems unlikely it would have yielded different results.
[Table]
[Table]
G Extended Experiments on Image Classiﬁcation
In order to investigate the performance of ARL on our custom image dataset, we ran a series of different experiments.
However, drawing deﬁnite conclusions from the obtained results and proposing alterations is beyond the scope of thispaper and is thus left as future work.
G.1 Dataset Details
For keeping the classiﬁcation task binary, all original class labels larger than 23 were given the new label 1 and allother original class labels were given the new label 0.
For each new label, half of the original classes were assigned asbelonging to a protected group.
To introduce noise, this re-assignment happens with a probability of 0.9.
Samples withoriginal labels that are not part of this re-assignment have a probability of 0.1 of becoming a member of a protectedgroup.
To eschew the ratio of protected/non-protected samples, samples that become a member of a protected groupare discarded from the dataset with a certain probability.
In the dataset EMNIST_35, this probability is 0.5 and inthe dataset EMNIST_10, this probability is 0.9.
This results in about 10% of the samples belonging to the protectedgroup in EMNIST_10 and about 35% of the samples belonging to the protected group in EMNIST_35.
The newbinary labels are balanced with about 48% of the samples belonging to the new class 1 in both datasets.
The datasetreﬂects that non-noise members of protected groups can be structurally different from non-members with regardsto their features.
For EMNIST_35, the training set contains approximately 84000 samples and the test set containsapproximately 14000 samples while for EMNIST_10, the training set contains approximately 61000 samples and thetest set contains approximately 10000 samples.
G.2 Different Adversary Architectures
We tested a total of three different architectures for the ARL adversary.
All consist of a single convolutional layer with3 × 3 kernels and stride 1, a max-pooling layer with a 2 × 2 kernel and stride 1 and a subsequent single linear layerthat directly maps the ﬂattened output of the previous layer with appended target label to the scalar output.
This setupwas chosen in order to stay close to the original architecture for tabular data, as proposed by Lahoti et al.
The model ARL_strong uses 64 ﬁlters, the model ARL uses 32 ﬁlters and the model ARL_weak uses 2 ﬁlters in the convolutionallayer.
The learner and the baseline are kept as described in Section 4.2. [Table]
[Table] G.3 Training Procedure and Optimal Hyperparameters
The learning rate and batch size of all image models were determined by performing a grid search over the samehyperparameter space as deﬁned for the other datasets in Appendix C.
The grid search procedure is the same as in
Section 2.3 but with reserving 10% of the training set as validation set instead of performing cross validation.
For allimage models, the hyperparameters are listed in Table 14.
All image models were trained using the AdaGrad optimizerand all reported results are obtained from models that were trained for 5k steps or until the early stopping criterion, thatis the overall AUC on the validation set had not improved for 10 epochs, is reached.
G.4 The Effects of Different Datasets
Table 13 shows the extended results for ARL on the different datasets.
In our experiments on image data, the claim of
Lahoti et al.
that ARL improves the performance on the worst-off group could not be conﬁrmed to hold.
Rather, ARLleads to an even lower performance on the minority sub-group which stands in contrast to the original motivation of
ARL, that is to increase the performance on the worst-off group.
This is independent from the ratio of protected tonon-protected samples in the dataset as it happens in both EMNIST_35 and EMNIST_10.
It is notable that ARL sees alarge performance drop over ARL_weak on EMNIST_10 but not on EMNIST_35.
G.5 Computational Identiﬁability
Similar to Section 3.1, we tested whether the adversary is in principle capable of predicting the protected group ina supervised setting.
Table 15 shows the accuracies that the different settings for ARL achieve on this task.
Dueto computational constraints, the reported numbers are results from a single training run of 5000 steps using thehyperparameters that were obtained in the grid search of the complete learner/adversary setting.
ARL has majordifﬁculties identifying the minority sub-group in the strongly imbalanced dataset EMNIST_10.
The performance on theminority sub-group increases with increasing representation in the dataset as the results on EMNIST_35 show.
This isin line with our expectations of a classiﬁer whose optimization target is to improve the overall performance.
Further,increasing the complexity of the adversary heavily increases the performance on the minority sub-group while theoverall performance and performance on the majority sub-group is only slightly reduced. [Table]