What Do You See in this Patient? Behavioral Testing of Clinical NLP Models (2022)

Abstract: Decision support systems based on clinical notes have the potential to improve patient care by pointing doctors towards overseen risks. Predicting a patient’s outcome is an essential part of such systems, for which the use of deep neural networks has shown promising results. However, the patterns learned by these networks are mostly opaque and previous work revealed both reproduction of systemic biases and unexpected behavior for out-of-distribution patients. For application in clinical practice it is crucial to be aware of such behavior. We thus introduce a testing framework that evaluates clinical models regarding certain changes in the input. The framework helps to understand learned patterns and their influence on model decisions. In this work, we apply it to analyse the change in behavior with regard to the patient characteristics gender, age and ethnicity. Our evaluation of three current clinical NLP models demonstrates the concrete effects of these characteristics on the models’ decisions. They show that model behavior varies drastically even when fine-tuned on the same data with similar AUROC score. These results exemplify the need for a broader communication of model behavior in the clinical domain.

## introduction outcome prediction from clinical notes. the use of automatic systems in the medical domain is promising due to their potential exposure to large amounts of data from earlier patients. this data can include information that helps doctors make better decisions regarding diagnoses and treatments of a patient at hand. outcome prediction models take patient information as input and then output probabilities for all considered outcomes @xcite . we focus this work on outcome models using natural language in the form of clinical notes as an input, since they are a common source of patient information and contain a multitude of possible variables. the problem of black box models for clinical predictions. recent models show promising results on tasks such as mortality @xcite and diagnosis prediction @xcite . however, since most of these models work as black boxes, it is unclear which features they consider important and how they interpret certain patient characteristics. from earlier work we know that highly parameterized models are prone to emphasize systemic biases in the data @xcite . further, these models have high potential to disadvantage minority groups as their behavior towards out-of-distribution samples is often unpredictable. this behavior is especially dangerous in the clinical domain, since it can lead to underdiagnosis or inappropriate treatment @xcite . thus, understanding models and allocative harms they might cause @xcite is an essential prerequisite for their application in clinical practice. we argue that more in-depth evaluations are needed to know whether models have learned medically meaningful patterns or not. behavioral testing for the clinical domain. as a step towards this goal, we introduce a novel test-ing framework specifically for the clinical domain that enables us to examine the influence of certain patient characteristics on the model predictions. our work is motivated by behavioral testing frameworks for general natural language processing (nlp) tasks @xcite in which model behavior is observed under changing input data. our framework incorporates a number of test cases and is further extendable to the needs of individual data sets and clinical tasks. influence of patient characteristics. as an initial case study we apply the framework to analyse the behavior of models trained on the widely used mimic-iii database @xcite . we analyse how sensitive these models are towards textual indicators of patient characteristics, such as age, gender and ethnicity, in english clinical notes. these characteristics are known to be affected by discrimination in health care @xcite , on the other hand, they can represent important risk factors for certain diseases or conditions. that is why we consider it especially important to understand how these mentions affect model decisions. contributions. in summary, we present the following contributions in this work: 1) we introduce a behavioral testing framework specifically for clinical nlp models. we release the code for applying and extending the framework foot_0 to enable in-depth evaluations. ## behavioral testing of clinical nlp models sample alterations. our goal is to examine how clinical nlp models react to mentions of certain patient characteristics in text. comparable to earlier approaches to behavioral testing we use sample alterations to artificially create different test groups. in our case, a test group is defined by one manifestation of a patient characteristic, such as female as the patient's gender. to ensure that we only measure the influence of this certain characteristic, we keep the rest of the patient case unchanged and apply the alterations to all samples in our test dataset. depending on the original sample, the operations to create a certain test group thus include 1) changing a mention, 2) adding a mention or 3) keeping a mention unchanged (in case of a patient case that is already part of the test group at hand). this results in one newly created dataset per test group, all based on the same patient cases and only different in the patient characteristic under investigation. prediction analysis. after creating the test groups, we collect the models' predictions for all cases in each test group. different from earlier approaches to behavioral testing we do not test whether predictions on the altered samples are true or false with regard to the ground truth. as van aken et al. (2021) pointed out, clinical ground truth must be viewed critically, because the collected data does only show one possible pathway for a patient out of many. further, existing biases in treatments and diagnoses are likely included in our testing data potentially leading to meaningless results. to prevent that, we instead focus on detecting how the model outputs change regardless of the original annotations. this way we can also evaluate very rare mentions (e.g. transgender) and observe their impact on the model predictions reli-ably. figure 2 shows a schematic overview of the functioning of the framework. ## results we present the results on all test cases by averaging the probabilities that a model assigns to each test sample. we then compare the averaged probabilities across test cases to identify which characteristics have a large impact on the model's prediction over the whole test set. the values per diagnosis in the heatmaps shown in figure 3 , 4, 7 and 8 are defined using the following formula: we choose this illustration based on the concept of partial dependence plots @xcite to highlight both positive and negative influence of a characteristic on model behavior. since all test groups are based on the same patients and only differ regarding the characteristic at hand, even small differences in the averaged predictions can point towards general patterns that the model learned to associate with a characteristic. ## discussion model behaviors show large variance. the results described in 5 reveal large differences in the influence of patient characteristics throughout models. the analysis shows that there is no overall best model, but each model has learned both useful patterns (e.g. age as a medical plausible risk factor) and potentially dangerous ones (e.g. decreases in diagnosis risks for minority groups). the large variance is surprising since the models have a shared architecture and are fine-tuned on the same datathey only differ in their pre-training. and while the reported auroc scores for the models ( ## conclusion in this work, we introduced a behavioral testing framework for the clinical domain to understand the effects of textual variations on model predictions. we apply this framework to three current clinical nlp models to examine the impact of cer-tain patient characteristics. our results show that the models-even with very close auroc scoreshave learned very different behavioral patterns, some of them with high potential to disadvantage minority groups. with this work, we want to emphasize the importance of model evaluation beyond common metrics especially in sensitive areas like health care. we recommend to use the results of these evaluations for discussions with medical professionals. being aware of specific model behavior and incorporating this knowledge into clinical decision making is a crucial step towards safe deployment of such models. for future work we consider iterative model fine-tuning with medical professionals in the loop a promising direction to teach models which patterns to stick to and which ones to discard.