XAI for Better Exploitation of Text in Medical Decision Support (2024)

Abstract: In electronic health records, text data is considered a valuable resource as it complements a medical history and may contain information that cannot be easily included in tables. But why does the inclusion of clinical texts as additional input into multimodal models, not always significantly improve the performance of medical decision-support systems? Explainable AI (XAI) might provide the answer. We examine which information in text and structured data influences the performance of models in the context of multimodal decision support for biomedical tasks. Using data from an intensive care unit and targeting a mortality prediction task, we compare information that has been considered relevant by XAI methods to the opinion of a physician.

## introduction electronic health records often contain factual information in short, tabular form, including laboratory values, diagnoses, gender, and age. they also include longer texts in various forms written for many different purposes. depending on the origin and context of the data, the text could be a clinical or nursing note, a discharge summary, or a radiology report, to name a few. the text might provide a high-level interpretation of the current patient situation, taking different kinds and sources of information into account. the text might refer directly to some given structured facts in the database (e.g., a lab value is above borderline) but might also consider additional information such as general impressions, assumptions, and information gathered directly from the patient or other medical personnel (e.g., the patient is not very adherent). for this reason, the texts are generally considered valuable resources in the clinical routine. in the context of machine learning for healthcare, however, the inclusion of such texts has shown in various setups only marginal effects @xcite ; yang and wu, 2021), although one would assume that the additional information and complementary perspective should improve a system's performance. many papers in this area deal with multi-modal data, integrating, for instance, image and text, or structured and unstructured data into one model. @xcite is a popular dataset in this context, as it can be easily accessed by researchers. it contains data from an intensive care unit (icu) of a us hospital, including patient demographics, time series data, or text, such as nursing notes, discharge summaries, or social worker notes. however, while many approaches in other domains do achieve a boost in performance using multimodal (text) data, the performance dif-ference between unimodal and multi-modal models in the medical context can be modest @xcite . in this work, we explore which information is valuable for multi-modal machine learning using mimic data. more precisely, we re-implement two multi-model (mm) approaches for the task of in-hospital mortality prediction. we then introduce an xai approach for the given mm approaches and examine the attributed information according to their faithfulness. finally, we investigate if the attributions are plausible from a physician's perspective. ## related work recent years have seen a surge in leveraging deep learning approaches utilizing diverse clinical data sources for clinical outcome predictions. these include textual clinical notes, longitudinal data, and demographic data. unimodal approaches like cnns @xcite , lstms @xcite , and bert @xcite have laid the groundwork. later expanded to multimodal approaches such as additive fusion @xcite to more sophisticated cross attention fusion @xcite . @xcite and @xcite implemented additive and gated fusionbased multimodal models for tasks like diagnosis prediction, acute respiratory failure prediction, and in-hospital mortality prediction. we extend their work by applying explainability methods to models and evaluating the quality of explanations. explainable ai (xai) enhances transparency and trust in healthcare applications, especially within medical decision support systems @xcite and clinical nlp @xcite . notably, @xcite compared the faithfulness of various explanation methods for models like bert in mortality prediction. additionally, deyoung et al. (2020) introduced a benchmark with human annotations to evaluate nlp models explainability for faithfulness and plausibility. however, previous research has mainly focused on quantitative evaluations of explainability methods for uni models. this study addresses this gap by quantitatively evaluating xai in multimodal models. ## experimental results our first experiment concerns the reproduction of multimodal and unimodal methods and application to the in-hospital mortality task. for the evaluation, we follow a similar methodology to related work ( §2), utilizing roc (area under the receiver operating characteristic curve) and aupr (area under the precision-recall curve). in the second experiment, we apply xai to the models and examine which information is consid-ered by the model as valuable for the prediction. following jacovi and goldberg (2020), we explore faithfulness by replacing the top x attributed token or time point of the time series with a mask token or zero and observe the drop in model performance. finally, we conduct a plausibility test, as suggested by @xcite . here, we directly compare the attributions on text and structured data to the relevant information according to a physician's perspective. only annotation of text data is quantitatively analyzed based on the overlap between annotated tokens and attributed tokens, such overlap matching is not possible for time-series data. as we are particularly interested in examining the benefit of text data, we randomly select 100 patient cases in which a multi-modal approach predicts a higher probability score for mortality than the unimodal lstm approach. likewise, we randomly select 100 cases in which the multi-modal approach predicts a lower probability score for mortality. for those cases, we assume that text data provided additional information to make a stronger prediction assumption (independent of whether the prediction is correct or not). a final-year medical student annotated these 200 cases. the student was asked to identify parts of the text and important time-series features that support the outcome of mortality or survival. in addition, the student was asked to provide their estimation of the patient's survival and whether the text was useful in solving the task. ## discussion the initial results align with findings from related work: text data is a valuable resource for improving predictions, but its benefit varies depending on the task and the text source. for instance, nursing notes led to higher results than admission notes, despite the fact that nursing notes were often truncated due to bert's restricted input length. given the redundancy in clinical texts, it may be beneficial to compress larger texts into shorter documents to accommodate additional text sources. another notable finding is the performance drop when removing height and weight, two features that are often missing and filled with default values. our medical expert confirmed that height and weight do not influence the given task, which may reduce overall trust in our model. however, it is not unusual for machine learning models to consider seemingly irrelevant information as useful. for example, in @xcite , a nephrology outcome prediction model found the number of lab measurements in the last month to be very useful, which may indirectly indicate a patient's deteriorating condition. in our case, the model's reliance on height and weight might be justified by the context in which these features are used. for instance, weight may be measured over time to monitor fluid balance. thus, the model might be capturing an important dependency that is not immediately apparent. in the second experiment evaluating the faithfulness of the attributions, we observed a significant drop in model performance when the top-attributed information was replaced in the input, compared to a random replacement. this stronger decline in performance was particularly pronounced when timeseries data was replaced, indicating that time-series information plays a crucial role in the model's performance for the given task. conversely, it also shows that some tokens, such as medical conditions that are mentioned in the text, have a positive influence on the model. in the third experiment, comparing human and xai annotations of texts suggests that systems can extract relevant information (pre-existing conditions are identified more often than acute conditions). on the other hand, the extracted information is not always humanly plausible. the comparison of human and xai annotated time-series features showed that both the physician and the model consider similar features useful for the given prediction task. however, multimodal quantitative analysis of plausibility remains a bottleneck that should be addressed in future work. ## conclusion we analyzed the relevance of text and structured data in the context of a multimodal decision support system for in-hospital mortality. we found that the source of text influences the model performance (nursing vs admission notes). moreover, sparse information (e.g., patient height and weight) can benefit the performance of models, although such information does seem irrelevant from an expert's perspective. in our experiments, we found that the model performance drops considerably when structured information (time series) is replaced in the input compared to textual inputs. in general text data could provide additional context in a multimodal setup, but its benefit depends on the task (other tasks might lead to more benefits) as our results showed only a marginal boost in performance compared to unimodal models. finally, our comparison between human and xai annotations of the texts indicates that the models can extract relevant information but not always. it seems that for multimodal data such as text and time series, quantitative analysis of plausibility is a bottleneck, and it should be addressed in future work.