Making the Most Out of the Limited Context Length: Predictive Power Varies with Clinical Note Type and Note Section (2023)

Abstract: Recent advances in large language models have led to renewed interest in natural language processing in healthcare using the free text of clinical notes. One distinguishing characteristic of clinical notes is their long time span over multiple long documents. The unique structure of clinical notes creates a new design choice: when the context length for a language model predictor is limited, which part of clinical notes should we choose as the input? Existing studies either choose the inputs with domain knowledge or simply truncate them. We propose a framework to analyze the sections with high predictive power. Using MIMIC-III, we show that: 1) predictive power distribution is different between nursing notes and discharge notes and 2) combining different types of notes could improve performance when the context length is large. Our findings suggest that a carefully selected sampling function could enable more efficient information extraction from clinical notes.

## introduction electronic health records (ehr) enable the development of language model based clinical predictor, which takes in clinical notes to predict patient outcomes. clinical notes in ehr exhibit two unique characteristics. 1) clinical notes cover a long time span (from a few weeks to over a year), which results in their sparsity of information-rich sections. these distinguishing characteristics of clinical notes lead to a new design choice: when the context length is limited due to the constrained compute or model architecture, what parts of clinical notes should we sample to maximize the model's performance? we propose a framework to subsample text sections with high predictive power. empirically, we explore the distribution of predictive power over clinical note types and sections by searching over these variables. we found that 1) the predictive power distribution is different between nursing notes and discharge notes: the predictive power is stronger at the beginning and end of discharge notes, while uniform within nursing notes. 2) the effect of combining sections from different types of notes improves the performance when the context size is large, but harms the performance when the context size is small. more details of task formulation can be found at section 3. our code is publicly available on github foot_0 . ## related work existing methods for subsampling clinical notes for the bert-based model are mostly based on domain knowledge. for instance, @xcite and @xcite choose discharge notes as they summarize patients' visits. @xcite chooses the notes within three days before a cutoff time in consideration of timeliness. while these assumptions are based on domain knowledge, they require human input and may not generalize. thus, we are interested in exploring a data-driven sampling choice without assumptions of expert inputs. another related, but orthogonal approach to the limited context length problem is note aggregation. instead of subsampling notes, @xcite propose to feed everything to the model, one maximum context length at a time, and aggregate the outputs for the final prediction. in their work, notes of one patient are split into a partition of subsequences, and the patient's re-admission risk is obtained by taking a weighted average of probabilities computed from each subsequence. this method's compute cost scales with the aggregated sequence length, which can be expensive for records with long clinical notes. in contrast, our method aims to find one single information-rich segment as input. ## method we formalize our prediction task as follows: given a set of clinical notes x associated with an admission record, we want to predict the class label y which is our patient outcome of interest. ideally, we want to train a classifier f w * to approximate p(y | x). the optimal parameter is we say a sample function s θ has a higher predictive power if m(f w * (s θ (x), y)) is larger. while current works chose s θ based on prior medical knowledge or simply fix it as a truncation function, we propose to explore different sampling functions s θ to make the most out of the limited context length with the highest predictive power. notice that in our work, s and θ are searched manually, instead of using learning algorithms. ## experimental setup we hypothesize that for 30-day all-cause readmission prediction, there exists an alternative sampling function that enables similar or better performance than the commonly used "truncated discharge notes". more formally, we focus on a parameterized sampling function with 2 variables: 1) which section of tokens to include, 2) what type(s) of clinical notes to use.