Improving the Transferability of Clinical Note Section Classification Models with BERT and Large Language Model Ensembles (2023)

Abstract: Text in electronic health records is organized into sections, and classifying those sections into section categories is useful for downstream tasks. In this work, we attempt to improve the transferability of section classification models by combining the dataset-specific knowledge in supervised learning models with the world knowledge inside large language models (LLMs). Surprisingly, we find that zero-shot LLMs out-perform supervised BERT-based models applied to out-of-domain data. We also find that their strengths are synergistic, so that a simple ensemble technique leads to additional performance gains.

## introduction the text in electronic health record notes is typically organized into multiple sections. correctly understanding what parts of a note correspond to different section categories has been shown to be useful for a variety of downstream tasks -including abbreviation resolution @xcite , cohort retrieval @xcite , and named entity recognition @xcite . however, documentation of sections is not consistently done across health systems, so building systems to robustly classify clinical text into sections is not trivial. prior work on text classification has shown that systems trained on a dataset from one source perform quite poorly on different sources @xcite . in this work, we extend recent work on section classification @xcite ) that uses the soap ("subjective", "objective", "assessment", "plan") framework @xcite . our previous work @xcite mapped heterogeneous section types across three datasets onto soap categories (plus "other") in order to facilitate cross-domain adaptation. however, despite showing improvements, that work showed that the problem was still challenging for a supervised approach that fine tuned pre-trained bertstyle encoder methods. the insight of this current work is that supervised transformers, while powerful, may overfit to source domain training data. zero-shot methods, on the other hand, have recently gained attention for their sometimes surprising ability to make accurate classification decisions without supervision. in general, for zero-shot classification to work, (1) the pre-training data must contain enough information about the kind of questions it will be asked, and (2) the prompt must be able to precisely represent the meaning of the classification labels. to work on section classification, then, we explore different base models since it is hard to know a priori which models will satisfy (1), and we explore variations in prompts that inject knowledge about the classification task to satisfy (2). therefore, we investigate the following research questions related to the ability of large language models (llms) to do soap section classification: rq1: how do different llms perform on the section classification task in zero-shot and few-shot experiments? rq2: how do llms in the zero-shot setting compare against supervised bert-based models applied across domains in their ability to classify soap sections? rq3: are the strengths of llms and bertbased models complementary so that ensemble methods may be synergistic? ## discussion our results related to rq1 were quite surprising. the best-performing llm, flan-t5-xxl, while being the largest model, has the least overlap with our data genre and was unable to fit any example instances into its prompt. the success of flan-t5-xxl could be attributed to it both being larger in parameter size and having instruction tuning that other models don't have. future work should explore smaller versions of flan-t5 to learn whether the model size or fine tuning is more important, but one interesting hypothesis is that explicit fine tuning on tasks with multiple choice setups may have benefited flan-t5. despite the biomedlm (2.7b) having fewer than half the parameters of the galactica (6.7b) models, performance is not as degraded as we might expect. this could be an indicator that incorporating medical knowledge helps llms recognize medical texts better and thus performs closer to models that are larger when doing section classification. here again, it would be valuable to isolate the model size variable from the pre-training genre variable, but the closest galactica model in size to biomedlm has 1.3 billion parameters -a closer model size but still not a perfect comparison. neither model was seemingly able to take advantage of seeing labeled instances in their prompts. one possible explanation is that, because the output space has five unique labels, and the categories are quite heterogeneous, it is just not able to see enough diversity of each category type to meaningfully generalize. clinical-t5 @xcite , which is trained on mimic @xcite , can be explored in the future too, to examine the effect of pretraining on a more highly aligned domain. however, we note that the pre-training data for clinical-t5 overlaps with the progress dataset we evaluate on here, which makes it difficult to obtain fair zeroshot comparisons. finally, the pace of new releases of llms is quite fast, and models released after this work are potentially quite powerful (e.g. alpaca @xcite and vicuna (team, 2023)). future work can also include assessing those models' capability for section classification. the ensemble model was found to be the best, and a hypothesis can be that llms learn better for the rarer categories and supervised learning learns better on prevalent categories. one explanation for this is that the supervised learner implicitly learns a distribution over label frequency, which may bias it towards frequent categories, while the zero-shot learner only has access to the textual evidence to make its decisions. if this same dynamic holds more generally (as seen in other recent work @xcite ), llms may serve as an important supplement to supervised learning in terms of predicting rare categories. this study estimated the prediction probability for llm by repeating the experiments, and future work can explore additional methods for obtaining the prediction probability. ## conclusion this paper demonstrates the use of llms for section classification and an ensemble method for improving the transferability of section classification models. the supervised learning model and llms are competitive, and when ensembled based on the prediction probabilities, we observed a higher performance. in analyzing the prediction performance by categories, we found llms complemented the supervised learning by performing better on the rare categories, and the supervised method performed better for the most prevalent category. future studies can extend to updated llms and the use of llms for section classification is promising.