How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling (2022)

Abstract: Large pre-trained language models (LMs) have been widely adopted in biomedical and clinical domains, introducing many powerful LMs such as bio-lm and BioELECTRA. However, the applicability of these methods to real clinical use cases is hindered, due to the limitation of pre-trained LMs in processing long textual data with thousands of words, which is a common length for a clinical note. In this work, we explore long-range adaptation from such LMs with Longformer, allowing the LMs to capture longer clinical notes context. We conduct experiments on three n2c2 challenges datasets and a longitudinal clinical dataset from Hong Kong Hospital Authority electronic health record (EHR) system to show the effectiveness and generalizability of this concept, achieving ~10% F1-score improvement. Based on our experiments, we conclude that capturing a longer clinical note interval is beneficial to the model performance, but there are different cut-off intervals to achieve the optimal performance for different target variables.

## introduction clinical note is one of the most abundant data available in ehr systems, which records most of the patient interaction with the hospital services, such as consultation with doctors, procedure note, laboratory report, discharge summary, etc. 1 despite retaining rich clinical information, clinical notes are highly unstructured and composed of non-standardized information, which curbs the potential practicality of such information. large pre-trained lms, such as bert @xcite , roberta @xcite , @xcite , etc., have been shown to work well in extracting crucial information from clinical notes by utilizing transfer learning and attention mechanism @xcite @xcite . the adaptation of these models to biomedical and clinical domain emphasizes this success, establishing many new state-of-the-art performances on multiple biomedical and clinical benchmarks @xcite @xcite . while the attention mechanism embedded in the pre-trained models enables them to achieve great performance, it is to be noted that it also causes a quadratic growth in computation cost with respect to input sequence length @xcite @xcite . this makes efficiently processing long documents with pre-trained lms difficult, especially in clinical note modeling, in which a single clinical note tends to consist of hundreds or even thousands of words @xcite @xcite @xcite . current approaches to this problem commonly involve truncation, chunking, or windowing of the long input sequence, preventing the models from acquiring an entire medical record information provided by a whole clinical note. considering that clinical note modeling requires capturing and understanding the underlying long-term dependencies in the clinical notes, this certainly puts a limit on their predictive capability. for this reason, to maximize the models' capability without sacrificing a part of the input clinical notes, we explore the application of long-range adaptation through linear attention mechanism @xcite @xcite , which reduces the computation cost of attention from quadratic to linear in regards to input sequence length. in this work, we focus on assessing the benefit of capturing longer clinical notes on large pre-trained lms to n2c2 (national clinical nlp challenges) 2 clinical tasks by adapting a linear attention mechanism, i.e., longformer @xcite . furthermore, to test the generality of this approach, we evaluate it on a longitudinal clinical note corpus from hong kong hospital authority ehr system, which covers records from 43 hospitals in hong kong. lastly, we hypothesize that modeling longer interval of clinical notes improves the prediction quality of the models on any clinical task. to prove our hypothesis, we conduct our experiment using different context-length, allowing the model to access various intervals of clinical notes. our result suggests that a longer interval of clinical notes increases the prediction quality of the models in most cases, but there is a limit of context length required depending on the target variable. our contributions in this work can be summarized in three-fold: ## long-range clinical note lms on n2c2 challenges we assess the effectiveness of long-range clinical note lms on us-based clinical note datasets from three n2c2 challenges. additionally, we also evaluate six different pre-trained lms without longrange adaptation to benchmark the performance of the biomedical and clinical lms. ## long-range clinical note lms on hong kong longitudinal dataset we assess the generalization and effectiveness of long-range clinical notes lms on hong kong longitudinal clinical note data. we construct a longitudinal dataset with two target variables, i.e., disease risk and mortality risk, and evaluate long-range lms on the dataset. in addition, we add a baseline model, which takes high-level features extracted from the corresponding tabular data provided by the ehr system as the input, to assess the effectiveness of clinical note modeling. ## conclusion in this paper, we show the importance of capturing longer clinical notes for biomedical and clinical large pre-trained lms on 6 clinical nlp tasks on the united states and hong kong clinical note data. our result suggests that utilizing longer clinical notes can significantly increase the performance of lms by ∼5-10% f1-score without the loss of generalization to the unseen data. we also observe that incorporating a longer interval of clinical notes does not always entail performance improvement and there is an optimal cut-off interval depending on the target variable. based on our analysis, we conclude that an interval of ∼2-3 months is the optimal cut-off for mortality risk prediction, while 3.5 years or an even longer interval of clinical notes is required to achieve the optimal performance for disease risk prediction. future work in long-range clinical note modeling would open up opportunities towards a general solution in clinical nlp.