Section Classification in Clinical Notes with Multi-task Transformers (2022)

Abstract: Clinical notes are the backbone of electronic health records, often containing vital information not observed in other structured data. Unfortunately, the unstructured nature of clinical notes can lead to critical patient-related information being lost. Algorithms that organize clinical notes into distinct sections are often proposed in order to allow medical professionals to better access information in a given note. These algorithms, however, often assume a given partition over the note, and classify section types given this information. In this paper, we propose a multi-task solution for note sectioning, where a single model identifies context changes and labels each section with its medically-relevant title. Results on in-distribution (MIMIC-III) and out-of-distribution (private held-out) datasets reveal that our approach successfully identifies note sections across different hospital systems.

## introduction the increasing role of free-text narrative in electronic health records (ehr) is both a blessing and a curse. it allows much more nuanced information about patients' conditions being saved and documented @xcite @xcite . however, the unstructured nature of this data can also make it unavailable to medical care givers interested in searching for specific patient-related information @xcite . to better organize free-form clinical notes and allow researchers and practitioners to quickly search over them, many solutions were proposed, mainly focusing on sectioning notes to correspond to headers described within the note @xcite . these solutions were often rule-based @xcite , identifying common section headers in the data. unfortunately, this approach often failed to correctly classify sections across different hospital departments, care providers and ehr systems. for brevity throughout this paper, we refer these as different data sources or distributions interleaving. alternatively, machine learning methods were proposed to classify individual text-spans and map them into a pre-existing list of possible sections. this approach successfully outperformed rule-based approaches, but was often not deployed because of its inability to identify section-boundaries. with the recent success of transformer-based models in natural language understanding, we identify an opportunity to tackle the section boundary detection problem alongside section classification, and propose a unified solution. our approach is based on pre-trained encoder-only transformer models, which were shown to produce superior results on natural language understanding (nlu) tasks broadly @xcite , and specifically on clinically-relevant data @xcite . we start by exploring current section classification methods ( §2). then, we introduce our baseline, a marker-based section header extraction system, and describe how to use it to generate training labels for ml-based methods ( §3). we then pose hypotheses for when should ml systems outperform rule-based approaches, and propose solutions based on the hypotheses ( §4). we continue by proposing a dataset for training multi-task transformers from rule-based labels ( §5) and demonstrate how such models can outperform rule-based approach on in-distribution and out-of-distribution data ( §6). finally, we conclude our work in light of our posed hypotheses ( §7). ## related work identifying section headers in free-form clinical notes is long identified as a crucial task for organizing patient-level data in biomedical informatics @xcite . both ml-based and rule-based solutions were proposed in the last decade to solve the problem @xcite . unfortunately, existing solutions focus on solving the relatively narrowly-defined task of classifying predefined sections into section types, assuming that section borders are already given @xcite @xcite . in practice, however, we often observe complete notes, and are tasked with identifying distinct paragraphs and only then classifying them into individual sections. recently, there has been an influx of research demonstrating the power of pre-trained language models in solving multi-task problems @xcite @xcite , including on long texts @xcite . following this newly-formed conventional wisdom, we embrace this approach here, and propose an ml architecture that attempts to jointly detect section boundaries and classify individual sections. ## marker-based section header extraction we start by developing a marker-based section header extractor. this extractor will then be used for labeling our training data in §5 and as a baseline in §6. in this approach, a marker corresponds to a word that is usually used as the header of section. e.g. pmh is a typical marker word that represents the section past medical history. after examining patterns in the data, we discover hundreds of such markers in the mimic-iii dataset @xcite . lines that start with these markers are extracted and are labeled as section headers. these headers mark the boundary between two sections and the text between two headers is then treated as one single section. during our exploration, we recognized that there exists correlations between the type of the notes and the structure of the sections in the note. with that in regard, we customized our markers to the type of notes and certain markers will only be applied when the type of the note matches our definition. we identified 5 core note types that are most important for our usage: history and physical, progress, discharge summary, consult and operative. building on the mimic-iii dataset, we use an iterative approach to collect markers. a bootstrapping marker set is first developed on a sampled set of notes from the mimic-iii dataset. the marker set is then used to extract sections on the sampled set and the extracted sections are then sent to experienced clinicians for rating. new markers are then added according to the errors collected from the raters and then used on a new set of randomly collected notes. this process is repeated until no more errors are reported from the raters. in practice, we found that this method shows both high precision and high coverage in recognizing the sections. however, this approach does not work well when we try to transfer it to a new dataset where the medical notes come from a different healthcare provider, where we see the recall numbers dropping significantly (see §6 for complete results). by analyzing the errors, we are seeing the following patterns: by comparing with mimic-iii, we observe that while the headers are semantically similar across different healthcare providers, many cases are actually non-identical and can therefore cause recall losses. additionally, this approach does not take the context information into consideration, and is not able to recognize many cases above even if the section contents look similar to each other. ## section classification methods to build solutions that are robust across different distributions or require minimum learning efforts to adapt, we need to understand what is the transferable knowledge that applies. based on our experiences in building the marker-based approach, we have the following hypotheses: for the first hypothesis, we want to understand if we can build source-agnostic solutions by just expanding the markers used in the baseline. for the second hypothesis, we want to check if we can improve the accuracy of section type identification with additional information from the surrounding text of the section titles. for the last hypothesis, we propose to take advantage of the note type information within a multi-task framework. ## data to have enough data for training/evaluation, the output of the baseline system (section 3) is used as the golden data. due to the nature of the baseline algorithm, we can expect the generated data to have high precision/recall for training models on mimic-iii and also high precision but low recall for validation on the held-out private dataset. test data for mimic-iii data, we use the data described above. for the held-out private data, we use the same approach as described above and use all the extracted data as the test data. we randomly selected 500 notes for validation. ## experiments and results we first conducted experiments on mimic-iii dataset and ## conclusion in this work, we explored approaches for recognizing sections in free-form clinical notes. our approach is based on the hypothesis that section content is similar across distributions and can be used to generate a robust section classifier. our results demonstrate that our bert-based model trained on mimic-iii has very good performance on mimic-iii and on our held-out private data, outperforming strong baselines.