AI models in health care are not colour blind and we should not be either
Authors: Jenna Wiens, Melissa Creary, Michael W Sjoding
Journal: The Lancet Digital Health (2022)

In the article by Judy Wawira Gichoya and colleagues,1Gichoya JW Banerjee I Bhimireddy AR et al.AI recognition of patient race in medical imaging: a modelling study.Lancet Digit Health. 2022; (published online May 11.)https://doi.org/10.1016/S2589-7500(22)00063-2Summary Full Text Full Text PDF PubMed Scopus (3) Google Scholar the authors found that artificial intelligence (AI) systems can be trained to determine a person's self-reported race based on a medical image, such as a chest or spinal x-ray image. The issue, which has been highlighted by others,2Jabbour S Fouhey D Kazerooni E Sjoding MW Wiens J Deep learning applied to chest x-rays: exploiting and preventing shortcuts.in: Doshi-Velez F Fackler J Jung K Kale D Ranganath R Wallace B Wiens J Proceedings of the 5th Machine Learning for Healthcare Conference. 2020: 750-782Google Scholar is that if a model can easily determine a patient's race, it can use this information to predict other clinical outcomes. This is a scary prospect given the persistent bias against racial and ethnic minority groups in health care, leading to unequal access to care and unequal health, which are all also encoded in medical datasets.3Geiger J H Racial and ethnic disparities in diagnosis and treatment: a review of the evidence and a consideration of causes.in: Institute of Medicine (US) Committee on Understanding and Eliminating Racial and Ethnic Disparities Smedley BD Stith AY Nelson AR Unequal Treatment: confronting racial and ethnic disparities in health care. National Academies Press, Washington DC2003Google Scholar The work by Gichoya and colleagues suggests that an AI system could easily learn to perpetuate such racial biases, which could lead to the underdiagnosis of disease or less frequent treatment recommendations in racial and ethnic minority groups. Gichoya and colleagues study showed that an AI model could identify Asian, Black, and White patients only on the basis of a chest x-ray, chest CT, mammography, or spinal x-ray with an unexpectedly high degree of accuracy. Even more concerning is that the authors were not able to pinpoint what aspects of the medical images the model was using to differentiate race. In their analysis of chest x-rays, Gichoya and colleagues were able to explain the predictive ability of the model only in small part by correlations between race and identifiable patient demographics on a chest x-ray (eg, age or sex). They also found that the model could still accurately identify race when shown only portions of the image, or after limiting its ability to pick up on particular features (eg, bone density), suggesting that AI systems are highly capable at differentiating race. The fact that an AI tool that was trained and tested on large observational datasets collected from health-care systems in the USA could identify an individual patient's self-reported race is, in retrospect, perhaps not that surprising. There is a long history of bias and discrimination of patients on the basis of race in the USA, which gives rise to significant disparities in health outcomes.4Nong P Raj M Creary M Kardia SLR Platt JE Patient-reported experiences of discrimination in the us health care System.JAMA Netw Open. 2020; 3e2029650 Crossref PubMed Scopus (26) Google Scholar, 5Mateo CM Williams DR Racism: a fundamental driver of racial disparities in health-care quality.Nat Rev Dis Primers. 2021; 7: 20Crossref PubMed Scopus (13) Google Scholar, 6Agency for Healthcare Research and Quality2018 National Healthcare Quality and Disparities Report.https://www.ahrq.gov/research/findings/nhqrdr/nhqdr18/index.htmlDate: April 6, 2022Date accessed: April 6, 2022Google Scholar, 7Bailey ZD Feldman JM Bassett MT How structural racism works—racist policies as a root cause of US racial health inequities.N Engl J Med. 2021; 384: 768-773Crossref PubMed Scopus (297) Google Scholar As Melissa Creary and colleagues8Creary M Thiel D Eisen A Social meaning and the unintended consequences of inclusion.Am J Bioeth. 2017; 17: 63-65Crossref PubMed Scopus (2) Google Scholar state, "Race is not an arbitrary phenotypic difference, but rather is a powerful and deadly social system, and the consequent processes of racism get embodied with biological consequences". It could be hypothesised that these differences in health outcomes (the biological embodiment of racism) are, in fact, what the AI tool is learning. Gichoya and colleagues tested this hypothesis to some extent by training a model that used data with more similar disease distributions across racial groups, specifically by creating a dataset where the proportion of chest x-rays without abnormalities was equivalent across groups. Yet, the number of other subtle health differences across groups was still likely to be high, making it difficult to control for all potential confounders. The fact that there was no one single feature of the medical images studied that captured race suggests that there are several factors relating to race that are embedded in our health-care system and society that individually might only correlate slightly with race but, as a whole, provide strong evidence that AI leverages. The authors conclude that AI is not colour blind. Colour blindness, the problematic ideology that race should not and does not matter, is counterproductive because it ignores the fact that racism exists and can be coded in technologies.8Creary M Thiel D Eisen A Social meaning and the unintended consequences of inclusion.Am J Bioeth. 2017; 17: 63-65Crossref PubMed Scopus (2) Google Scholar, 9Bonilla-Silva E Color-blind racism.in: Rothenberg PS Race, class, and gender in the United States. Worth Publishers, New York, USA2007: 131-138Google Scholar, 10Benjamin R Race after technology: abolitionist tools for the new Jim code.Soc Forces. 2020; 98: 1-3Crossref Google Scholar Recognising racism in health care is fundamental to tackling it. To guard against unwittingly perpetuating biases, we must first recognise the ability of AI systems to identify race and thereby learn racial biases present in medical datasets. From there, the scientific community is better able to actively evaluate AI systems for such bias. The benefit of collecting data on patient race is the opportunity to monitor differences in clinical treatment, but we must also use these data to attend to the unjust social milieu that produces the disparate outcomes that potentially serve as the source of information that AI uses. That is where the value of the work by Gichoya and colleagues lies. By acknowledging the potential of AI tools to recognise and use race in their predictions, the necessary steps can be made to make sure that no harm comes of it. A greater emphasis on external validation across diverse patient subgroups, in particular the subgroups in which a model will be applied, is crucial to this endeavour.11Kamran F Tang S Otles E et al.Early identification of patients admitted to hospital for covid-19 at risk of clinical deterioration: model development and multisite external validation study.BMJ. 2022; 376e068576 PubMed Google Scholar To this end, we must work to make sure that our test sets reflect the demographic diversity that we expect to see in clinical care. We must go above and beyond when collecting data from minoritised populations to make sure that we have similar confidence in the performance of the model; otherwise, ultimately, we will always be less confident in the model's performance in these subgroups and risk exacerbating existing disparities in clinical care. We declare no competing interests. AI recognition of patient race in medical imaging: a modelling studyThe results from our study emphasise that the ability of AI deep learning models to predict self-reported race is itself not the issue of importance. However, our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging. Full-Text PDF Open Access