Investigating anatomical bias in clinical machine learning algorithms (2023)

Abstract: Clinical machine learning algorithms have shown promising results and could potentially be implemented in clinical practice to provide diagnosis support and improve patient treatment. Barriers for realisation of the algorithms’ full potential include bias which is systematic and unfair discrimination against certain individuals in favor of others. The objective of this work is to measure anatomical bias in clinical text algorithms. We define anatomical bias as unfair algorithmic outcomes against patients with medical conditions in specific anatomical locations. We measure the degree of anatomical bias across two machine learning models and two Danish clinical text classification tasks, and find that clinical text algorithms are highly prone to anatomical bias. We argue that datasets for creating clinical text algorithms should be curated carefully to isolate the effect of anatomical location in order to avoid bias against patient subgroups.

## introduction research in clinical machine learning algorithms have shown promising results for automating clinical tasks. the algorithms could potentially be implemented in clinical practice to provide diagnosis support, improve patient treatment and provide time-savings for medical doctors @xcite . however, despite appealing research results, there are currently limited examples of algorithms being successfully deployed into clinical practice @xcite . barriers for realisation of the algorithms' full potential include bias and generali-* equal contribution sation issues @xcite @xcite . algorithmic bias can be defined as systematic and unfair discrimination against certain individuals or groups of individuals in favor of others @xcite . previous studies have raised serious concerns of algorithms that contain age, gender and racial bias @xcite -even for algorithms that have been taken into use @xcite . although machine learning algorithms are trained to be able to generalise to previously unseen data, they tend to overfit to the data they have been trained on. as a consequence of this, bias can unintendedly arise if some subgroups of the target population are not represented in the data used to train the algorithm. moreover, if the training data itself include biases against some populations, e.g. data reflecting a negative attitude against people with disabilities @xcite , these biases might be encoded and reinforced. if biased algorithms are adopted, healthcare systems risk doing injustice to certain patient groups and harming patient safety @xcite . therefore, identifying and mitigating bias is important for successful implementation of novel clinical machine learning algorithms. this paper investigates anatomical bias in clinical machine learning algorithms developed to classify and extract specific medical conditions from the narrative text of electronic health records (ehr). we define anatomical bias as unfair algorithmic outcomes against a subgroup of patients with the same medical condition, where the algorithm performs differently depending on the anatomical location of the condition. if the performance of clinical algorithms varies depending on the anatomical location, it is reflected in some patient subgroups receiving worse treatment than others. we hypothesised that careful dataset curation is needed to measure and mitigate anatomical bias because the text description of medical conditions in ehrs varies depending on the location, e.g. 'epistaxis' is a location-specific word describing nose bleedings. specifically, this paper investigates anatomical bias for classification of bleeding and venous thromboembolism (vte) mentions in the narrative text of danish ehrs. automatic extraction of these conditions could be valuable for medical doctors in clinical practice, e.g. to guide diagnostic decision making and treatment options @xcite . previous papers @xcite @xcite @xcite @xcite @xcite have shown promising results for automatic extraction of these medical conditions but they did not investigate the performance of the algorithms across anatomical subgroups. our main contributions are: ## methods to investigate if machine learning algorithms are prone to anatomical bias, we performed two experiments. we (1) investigated the performance of a binary classifier on different anatomical subgroups of a medical condition when that subgroup was left out of the training set, and (2) measured how the performance on an anatomical subgroup varied depending on the amount of samples from that subgroup included in the training set. in addition to the main labels of each dataset (positive and negative for bleeding or vte), we created a subgroup label for the positive samples describing the anatomical location of either the bleeding or vte mention. samples that did not describe the anatomical location or described multiple locations were omitted. for the bleeding dataset, we used the following eight anatomical locations: gastrointestinal, urogenital, internal, otorhinolaryngeal, dermatological, gynecological, cerebral, and ophthalmological. for the vte dataset, we used the following five anatomical locations: lower extremity, lung, liver, cerebral, and upper extremity. the locations included for each medical condition were selected by two medical doctors. for each dataset, we created a balanced training, validation, and test set containing an equal amount of positive and negative samples. moreover, for the bleeding dataset, the positive samples of the training, validation, and test sets were distributed equally between anatomical locations. for the vte dataset, only samples from the lower extremity and lung locations were distributed equally between the train, validation, and test sets. the liver, cerebral, and upper extremity locations were only used for the test set because of a limited number of samples. all samples were preprocessed by removing special characters, superfluous spaces, and duplicate samples. after preprocessing the samples, the bleeding and vte datasets had an average token length of 13.3 and 13.6, respectively. the dataset distributions can be seen in ## analysis of word distributions medical conditions are often described using different words depending on the anatomical location for which the condition occurs. the top-3 words for upper extremity had a low uniqueness score (<0.15). this indicates that the vocabulary used to describe vtes in the upper extremity was also used for other locations -e.g. 'dvt' (deep vein thrombosis) was the most frequent word but the uniqueness score was only 0.07. this might explain why the sensitivity for upper extremity was relatively high, as seen in figure 3 , even when samples from that location were not present in the training set. on the contrary, some of the frequent words from the lower extremity, lung, and cerebral locations were close to unique which could explain why the sensitivity of those locations were low. appendix c shows word frequency and location uniqueness for the bleeding classification dataset which shows similar results. ## discussion this paper has presented evidence that clinical natural language processing (nlp) classification algorithms are prone to anatomical bias, which is unfair algorithmic outcomes against patients with medical conditions in specific anatomical locations. we found that the performance of algorithms for both bleeding and vte classification can vary significantly depending on the anatomical location with differences up to 36.3 pp and 89.1 pp, respectively. moreover, we found that small fluctuations in the training set distribution of anatomical locations can lead to significant performance drops for the underrepresented anatomical locations. for the datasets presented in this study, we showed that the words used to describe medical conditions vary depending on the anatomical location. if classifiers do not learn to properly represent the full vocabulary for describing a medical condition, its performance will decrease for some anatomical locations. we argue that datasets for clinical nlp algo-rithms should be created to be able to carefully measure anatomical bias, e.g. by subdividing each sample into an anatomical location. this is essential to avoid implementing clinical algorithms that might discriminate against specific subgroups of patients. for example, one of the developed vte classifiers in this study performed with sensitivities @xmath0.5% for cerebral vtes. applying such a model in clinical practice or research would provide unfair algorithmic outcomes against patients with cerebral vtes. we also showed that an algorithm not exposed to gynecological bleedings would perform worse on this anatomical location. this would lead to unfair algorithmic outcomes against woman with gynecological bleedings. similarly, because alcoholics have an increased prevalence of gastrointestinal bleedings @xcite , this group of people would have a higher risk of unfair algorithmic outcomes if the algorithm has not been trained on such bleeding locations. to the best of our knowledge, anatomical bias has not been investigated in previous research. however, some studies tried to automatically create datasets distributed between different patient groups by extracting data based on international classification of diseases 10 (icd) codes -e.g. @xcite extract data based on different bleeding disorders. while this approach could, to some degree, mitigate the problem, studies @xcite found that icd codes have low accuracy and, therefore, this does not ensure an evenly distributed dataset. moreover, in order to isolate and measure the performance on different anatomical locations, the dataset should be constructed with a known distribution of these anatomies. our work is closely related to the field of domain adaption. for example, @xcite find that an algorithm trained to extract temporal information from a specific patient population performs worse on another related patient population. their results highlight that it is a challenging task to develop algorithms that can generalise well across domains. the main difference between our study and theirs is that the algorithms described in this paper are not developed to work on different domains. rather, the algorithms are specifically developed to work on a specialised domain in the clinical field, e.g. bleeding detection. as our results have shown, the algorithms perform worse on some subpopulations of the population it is supposed to work on, and therefore, we describe this as a bias issue. ## conclusion this paper presented evidence that clinical nlp algorithms are prone to anatomical bias. we found that the performance of clinical classification algorithms for both bleeding and vte classification can vary significantly depending on the anatomical location of the medical condition. we argue that anatomical bias should be carefully examined when developing clinical text algorithms in order to avoid unfair algorithm performance against patient subgroups.