On the Impact of Random Seeds on the Fairness of Clinical Classifiers (2021)

Abstract: Recent work has shown that fine-tuning large networks is surprisingly sensitive to changes in random seed(s). We explore the implications of this phenomenon for model fairness across demographic groups in clinical prediction tasks over electronic health records (EHR) in MIMIC-III —— the standard dataset in clinical NLP research. Apparent subgroup performance varies substantially for seeds that yield similar overall performance, although there is no evidence of a trade-off between overall and subgroup performance. However, we also find that the small sample sizes inherent to looking at intersections of minority groups and somewhat rare conditions limit our ability to accurately estimate disparities. Further, we find that jointly optimizing for high overall performance and low disparities does not yield statistically significant improvements. Our results suggest that fairness work using MIMIC-III should carefully account for variations in apparent differences that may arise from stochasticity and small sample sizes.

## introduction fine-tuning pre-trained transformers @xcite such as bert @xcite has become the dominant paradigm in nlp, owing to their performance across a range of downstream tasks. clinical nlp -in which we often aim to make predictions on the basis of notes in electronic health records (ehrs) -is no exception @xcite . however, fine-tuning large networks is a stochastic process. performance can vary considerably as a function of hyperparameter choice, and many parameter sets can yield the same validation accuracy (i.e., the model is not identifiable), and more generally the problem is underspecified (d' @xcite . recent work has demonstrated that the choice of random seeds alone can have dramatic impact on model performance in nlp and beyond, even when all other hyper-parameters are kept fixed @xcite @xcite . in this work, we explore the intersection of randomness and fairness in the context of clinical nlp. fairness is a particularly acute concern in clinical predictive tasks, given the potential of such models to influence treatment decisions. this has motivated work investigating biases in predictive models trained over ehr @xcite @xcite @xcite @xcite . we investigate the impact of random seeds on the fairness of fine-tuned classifiers with respect to demographic characteristics such as gender and ethnicity. there are many definitions of algorithmic fairness which formalize different desired properties @xcite . following prior work, here we adopt a simple measure: the mean differences in model performance across demographic subgroups @xcite . we find that, on the popular mimic-iii dataset @xcite , seeds with comparable validation performance can give rise to large variations in disparities across demographic subgroups (figure 1 ). ## data and methods we investigate the variability of overall model performance and fairness across random seeds for a set of clinical prediction tasks derived from the multiparameter intelligence monitoring in intensive care (mimic-iii) set of electronic health records @xcite . for each task, we train a classifier on top of the contextualized representations of a bert @xcite model pretrained over ehr data @xcite . following recent work exploring randomness and fine-tuning, we consider the seeds used to shuffle the training data and to initialize the model parameters independently @xcite . specifically, we generate @xmath0, 10000). for each seed pair, we measure the overall performance as well as the performance for each demographic subgroup in terms of the area under the roc curve (auc). ## results we compare the overall performance with the performance for each subgroup as a function of random seeds. figure 2 shows the overall performance (left) along with the gap between the best and worst observed subgroup aucs (right), across tasks. we observe a large variance in both the overall performance and the gap. the former observation corroborates previous findings @xcite . to quantify how random seeds affect individual subgroups, we measure the the absolute differences (∆s) between overall and subgroup performances. we then evaluate whether there are correlations between overall performance and subgroup ∆s. figures 3 and 4 present the results for the shock phenotype classification task -one of the tasks with largest disparities observed in prior work @xcite . similar trends were found for the remaining tasks, and we report all results in the appendix . figure 3 shows that the performance of all subgroups varies significantly across random seeds and that variances are higher for minority groups. larger variations in minority subgroups are to be expected, as any empirical estimate will have a variance that is inversely proportional to the sample size of a group. in figure 4 , we observe that there seem to be two distinct clusters of seeds: one corresponding to high performing models (right of plots), and another to suboptimal models. 3 while the best performing models tend to have a lower variance of subgroup performance, there is otherwise no clear relationship between overall and subgroup performance. indeed, we find that many models with similar overall performance correspond to widely different subgroup ∆s, particularly for the minority groups. to explore the implications of this phenomenon, we simulate a grid search over all the random seeds on the validation set. we select the best seed along with all other seeds with similar performance (i.e., within a difference @xmath1.01 absolute auc). figure 1 shows the test set subgroup performance ∆s, for the best validation seeds, in the shock phenotype classification task (see for the other tasks). figure 5 summarizes the overall performance (left) along with the subgroup performance gap (right) across tasks. we can see that selecting seeds on the basis of overall performance helps to reduce the subgroup performance gap (compare the right subplots in figures 2 and 5). however, the top performing models show disparities with respect to both gender and ethnicity, suggesting that these models maximize performance for some groups at the expense of others. moreover, we find multiple seeds with similar levels of validation performance that correspond to very different subgroup ∆s. since we have not encoded any model selection preferences into the pipeline this variance may reflect a form of underspecification. can we define ## conclusions we have investigated the impact of random seeds on the fairness of fine-tuned pre-trained models for clinical tasks. specifically, we measured gaps in performance across gender and racial subgroups as a function of the choice of random seeds for data shuffling and parameter initialization. in line with prior work, we found that classifiers trained on mimic-iii data are often biased with respect to demographic subgroups. the contribution of this work is the empirical confirmation that choice of random seed alone significantly affects the apparent bias: seeds that yield comparable performance in aggregate on the validation data correspond to very different performances on subgroups in test data. our analyses corroborate @xcite 's findings on the importance of carefully chosen random seeds, but also suggest that an equal amount of attention should be payed to the impact of these choices on model fairness. however, interpretation of these results is complicated by sample size effects. while mimic-iii is in itself a large dataset, it also exhibits significant imbalance, both in terms of subgroups of patients and the prevalence of medical conditions. these imbalances compound when considering subsets of patients in the context of specific prediction tasks, which often leads to small sample sizes for minority subgroups. while we observed higher apparent variances for demographic minorities, our results also suggest that these variances can in large part be explained by the smaller sample sizes. indeed, we found the variances in subgroup performance to be inversely proportional to the size of the subgroup.