There is no such thing as race in health-care algorithms
Authors: None The Lancet Digital Health
Journal: The Lancet Digital Health (2019)

“All policies, even the most trivial, are either racist or anti-racist, — they support equity, or they don't”Ibram X. Kendi On Oct 24, 2019, a study published in Science showed how bias is scaled up and compounded by algorithms. Software sold by Optum, a leading health services company, used to identify high-risk patients for complex health needs, has been unintentionally but systemically discriminating against black people. The researchers found that the algorithm assigned consistently lower risk scores to black patients despite being equally as sick as their white counterparts. Hospitals and insurers use this algorithm, and others like it to help manage care for over 200 million people in the USA each year. Obermeyer and colleagues found that the algorithm assigned risk scores to patients based on total health-care costs accrued in 1 year. Given that higher health-care costs are associated with greater needs, this assumption seemed reasonable. However, the data showed that the health care provided to black people cost an average of US$1 800 less per year than the care given to a white person with the same number of chronic health problems. This finding meant that the algorithm failed to identify less than half the number of black patients at risk of complicated medical needs as white people. Obermeyer and colleagues are quick to comment that the development of the racist algorithm might not have been driven by hateful ideology, but the result was that black patients were less likely to receive targeted interventions to improve their health. The algorithm excluded race in their data, therefore the developers considered their algorithm to be “race-blind”. However, they did not account for embedded prejudice and that projected cost as a risk factor discriminates against black people. This study was conducted on one health-care system in the USA, using only patients with commercial insurance or Medicare. However, imbalances embedded in society, such as poverty, geography, and education, all differentially affect black patients, which is noted in the major historic role Medicare plays in ensuring health-care coverage to black people. A recent survey highlighted that a third of very ill Medicare beneficiaries had trouble paying for prescriptions drugs and medical bills, and 40% of these patients said that they had exhausted all their savings, while about 30% report being contacted by a collection's agency. Deep inequalities in institutional infrastructures are major drivers of bias in data used to train algorithms and Obermeyer and colleagues suggested similar biases exist across several industries globally. To ensure that systemic racism is not amplified in our health-care systems by rapidly progressing artificial intelligence (AI), forward-thinking policies are desperately needed. The NHSx, a UK health department focused on digital transformation, published a report last month with an ambitious title “Artificial intelligence: how to get it right”, which highlighted policies to ensure AI is developed within frameworks that “not only cover the intentions and responsibilities of different people involved in developing, deploying or using AI, but also the impacts that AI has on individuals, groups, systems, or whole populations.” The report identifies the need for algorithms to be developed with a deeper understanding of the wider cultural context. However, in the 107-page NHSx report, there is little guidance on the use of proprietary algorithms. Obermeyer and colleagues' study is unique as the researchers were allowed to access the algorithm and modify the model, which lead to an 84% reduction of bias. This level of transparency, sharing, and accountability is currently not common practice. Regulators, such as NHSx, must mandate open and transparent evaluation of algorithms and data sharing for rigorous validation to ensure safe real-world clinical use in patients. Health-care software developers must test and validate algorithms to ensure that they do not reinforce racial biases and, like doctors, their guiding principle must be to do no harm. Emphasis must be placed on the importance of diversifying the AI workforce in health-care, to ensure that race as a proxy for bias, disadvantage, and poor treatment is better understood. A 2016 NEJM editorial stated that the term racism is rarely used in medical literature and called for a shift in clinical and research focus from race to racism. In the application of AI there is certainly no such thing as race in health-care, only racism, and this must be overcome to safeguard the future equity of our health-care system.