Attention Networks for Augmenting Clinical Text with Support Sets for Diagnosis Prediction (2022)

Abstract: Diagnosis prediction on admission notes is a core clinical task. However, these notes may incompletely describe the patient. Also, clinical language models may suffer from idiosyncratic language or imbalanced vocabulary for describing diseases or symptoms. We tackle the task of diagnosis prediction, which consists of predicting future patient diagnoses from clinical texts at the time of admission. We improve the performance on this task by introducing an additional signal from support sets of diagnostic codes from prior admissions or as they emerge during differential diagnosis. To enhance the robustness of diagnosis prediction methods, we propose to augment clinical text with potentially complementary set data from diagnosis codes from previous patient visits or from codes that emerge from the current admission as they become available through diagnostics. We discuss novel attention network architectures and augmentation strategies to solve this problem. Our experiments reveal that support sets improve the performance drastically to predict less common diagnosis codes. Our approach clearly outperforms the previous state-of-the-art PubMedBERT baseline by up 3% points. Furthermore, we find that support sets drastically improve the performance for pregnancy- and gynecology-related diagnoses up to 32.9% points compared to the baseline.

## introduction pre-trained large language models such as clini-calbert @xcite or pubmedbert @xcite are commonly used in the medical domain to predict diagnoses from admission notes @xcite @xcite . admission and discharge notes are valuable information sources about doctors' decisions about patients and the outcomes. however, the vocabulary in these notes is often insufficient to describe the patients' clinical phenotype fully. also, clinical text frequently contains idiosyncratic vocabulary, uncommon abbreviations and differs from clinic to clinic in writing style. moreover, evidence for clinical diseases in the text is imbalanced. in particular for less common or even rare diseases (see also figure 1 ) not much text evidence exists. finally, pre-trained language models can suffer from limited access to training data because of silos or data-privacy concerns. these factors can lead to poor performance in predicting outcomes on clinical text with pre-trained language models. multimodal patient representation. @xcite and @xcite therefore propose to augment text with potential complementary multimodal data into a reusable deep patient representation to improve clinical prediction tasks. for example, recent work surveys to augment text with image data @xcite , with complementary medical text books @xcite , ontologies @xcite or time series data @xcite . ## related work there is a large amount of work focusing on diagnosis prediction from ehr data, especially clinical codes. furthermore, there is an increasing emphasis on incorporating text or multi-modal data from clinical notes. we distinguish ourselves, particularly from work in icd coding, since only information at the time of admission is used for our considered tasks. icd coding, on the other hand, uses all data available at discharge time. diagnosis prediction on codes. @xcite use a reverse time attention mechanism on the diagnosis and procedure codes of the patients' history for the task of heart failure prediction. @xcite use a bidirectional recurrent neural network (rnn) on the diagnosis and procedure codes of the patients' history to predict diagnosis codes for the next admission. later they apply graphbased attention to incorporate the knowledge of a medical knowledge graph to learn medical representations @xcite . @xcite use a self-attention mechanism on the diagnosis codes to capture contextual and temporal relations within the patients' journey to predict the second hierarchy of the icd-9 codes. clinical text for diagnosis prediction. @xcite evaluate the usefulness of different simple text representations for diagnosis prediction and show that the text itself contains valuable information. @xcite use stacked denoising autoencoders combined with a paragraph vector model to learn patient representations. van @xcite simulate patients at admission time by only using parts of the textual descriptions known at admission time, such as "chief complaint" or "medical history". @xcite apply knowledge graphs to retrain and instill attention heads with complementary structured domain knowledge for clinical outcome prediction from text. @xcite seek to embed complementary knowledge to increase the performance on low resource languages by consecutive fine-tuning of multi-lingual models. ## evaluation metrics. we measure the performance of our experiments in macro averaged auroc (area under the receiver operating characteristic curve) and map (mean average precision). because the supplied support set s i in the diagnosis prediction task is part of the target label space s i ⊂ c i , it provides the support set augmented architectures an unfair advantage over the baselines to evaluate on @xmath0. therefore, we only evaluate our approach on @xmath1. ## discussion clever hans problem for readmissions. language models often just learn effective shortcuts of high dimensional data distributions instead of generalizing, which is called clever hans problem @xcite . we expect to find a variant of the clever hans problem for the readmission diagnosis prediction task: here, we expect the model to learn the shortcut of copying codes from the support set into the diagnosis predictions instead of learning novel correlations from the potentially complementary text data. indeed, our model copies in 78.1% of the test cases, on average, 2.12 codes from the support set that are not in the target label set. however, this is only a tiny fraction of the average of 13.45 codes in each support set (s. beneficial and non-beneficial codes. in our evaluation with clinical doctors we observe certain codes that improve the prediction more than others. we find that these codes are typically rare, such as code 188 (s. future improvements. it is interesting to see if graph neural networks can lead to improved representation through updates of related codes. furthermore, it is possible to experiment with negative examples of diagnostic codes and additional encodings to represent the time between two or more admissions. given the sparse training data situation presented in medical data silos, particular focus should be applied to zero-shot or few-shot cases, e.g., codes that occur the first time or are rarely represented in prior admissions. ## conclusion augmenting text with set data is an important problem, in particular in the clinical domain with multimodal patient representations. to solve this problem, we propose novel attention-based network architectures. our results clearly show that in a clinical prediction task, the augmented representation outperforms a language model, particularly for predicting less common diseases. we also observe that complementary data from sets can for compensate shortcomings of language models, such as idiosyncratic language or abbreviations.