An Ensemble Approach for Automatic Structuring of Radiology Reports (2020)

Abstract: Automatic structuring of electronic medical records is of high demand for clinical workflow solutions to facilitate extraction, storage, and querying of patient care information. However, developing a scalable solution is extremely challenging, specifically for radiology reports, as most healthcare institutes use either no template or department/institute specific templates. Moreover, radiologists’ reporting style varies from one to another as sentences are written in a telegraphic format and do not follow general English grammar rules. In this work, we present an ensemble method that consolidates the predictions of three models, capturing various attributes of textual information for automatic labeling of sentences with section labels. These three models are: 1) Focus Sentence model, capturing context of the target sentence; 2) Surrounding Context model, capturing the neighboring context of the target sentence; and finally, 3) Formatting/Layout model, aimed at learning report formatting cues. We utilize Bi-directional LSTMs, followed by sentence encoders, to acquire the context. Furthermore, we define several features that incorporate the structure of reports. We compare our proposed approach against multiple baselines and state-of-the-art approaches on a proprietary dataset as well as 100 manually annotated radiology notes from the MIMIC-III dataset, which we are making publicly available. Our proposed approach significantly outperforms other approaches by achieving 97.1% accuracy.

## introduction electronic medical records (emrs), such as radiology reports, contain patient clinical information and are often in the form of "natural language" written or transcribed by providers @xcite . gathering and disseminating patient information from such notes is required for patient care management. natural language processing (nlp)-driven solutions have been proposed to augment clinical workflows to facilitate such information extraction and structuring processes. segmentation of medical reports into topically cohesive sections @xcite is essential for nlp tasks such as relation extraction, named entity recognition (ner), and question and answering. developing a universal and scalable report segmenting solution is extremely challenging as most healthcare institutes use either no template or institute specific templates. moreover, providers' style of reporting varies from one to another as sentences are written in a telegraphic format and generally do not follow english grammar rules. nonethe-less, in the case of radiology reports, the reports are often composed of similar sections, including the reason for the visit, the performed examination, a summary of observations and findings, and finally, the radiologist's impression and recommendation based on the observations. to extract and structure patient information from notes, most clinical institutes take the approach of developing their specific set of patterns and rules to extract and label the sections within the clinical reports. this requires a substantial amount of effort for defining rules and maintaining them over time. with advancements in machine learning and nlp, researchers have more recently utilized supervised machine learning methods for automatic structuring of radiology reports @xcite @xcite @xcite . these machine learning approaches can be divided into three main themes: 1) methods that solely rely on extracting features from the format of the text and, therefore, are biased on the specific format of the training data @xcite ; 2) more recent efforts that are focused on learning to label based on the context @xcite ; and finally, 3) the hybrid approaches that combine formatting and contextdriven features @xcite . the two latter methods require a reasonably large amount of annotated reports and yet are not scalable solutions as they do not adequately address inter-institute variability unless model training is fine-tuned using annotated data from the target institute. in this work, we frame the structuring of the radiology reports as a multi-class sentence classification problem. more specifically, this work presents a novel framework to identify various sections in the radiology reports and to label all sentences within the note with their corresponding section category. we propose an ensemble approach that takes advantage of formatting cues as well as context-driven features. we incorporate recurrent neural networks (rnn) and sentence encoders accompanied by a set of engineered features from the reports for the task of section labeling in radiology reports. the proposed approach considers the context of the current text span and the surrounding context that helps make more accurate predictions. we were motivated by how a non-expert human self-teaches to perform such a task, paying attention to the context while taking formatting cues into account. we hypothesize that each of the three models learns unique and non-overlapping attributes for solving the problem at hand, and therefore, an ensemble approach seems reasonable. in order to avoid the requirement of access to a large annotated training corpus, we follow a weak learning approach in which we automatically generate the initial training data using generic rules that are implemented using regular expressions and pattern matching. we consider seven types of section categories and label each sentence with one of these categories. our approach is not limited to these specific categories and it can be adapted for any template format and writing style. this is thanks to incorporating a broad set of features that are independent of physicians/institutions. figure 1 depicts a snapshot of the output of our proposed model for automatic labeling of the sentences within a radiology report. the label in front of each line represents the predicted label for the following sentence. we train and evaluate our proposed approach on a large multi-site radiology report corpus from mass general brigham, referred to as mgb. we demonstrate that our proposed solution significantly outperforms common existing methods for automated structuring of radiology reports @xcite as well as several baseline models. moreover, we manually annotated 100 reports from the mimic-iii radiology reports corpus @xcite , and we report performances on this dataset as well. we also make this dataset publicly available to other researchers foot_0 . our main contributions in this study are as follows: the rest of the paper is organized as follows. in section 2, we briefly review current methods for segmenting and structuring clinical reports. next, we describe our proposed pipeline in section 3. in section 4, we present and discuss our results on independent test sets, and finally, the conclusions and potential future work are presented in section 5. ## related work there have been numerous efforts to address the need for automatic structuring of clinical notes via section labeling, including rule-based methods, machine learning-based methods, and hybrid approaches @xcite . @xcite proposed a rule-based system comprising a structural analyzer, lexical analyzer, parser, and a semantic interpreter to identify sections in radiology reports. @xcite developed a hierarchical section header terminology and a statistical model to extract section labels from sentences. radbank was introduced by rubin and desser, which recognizes the structure of radiology reports and extracts the sections for indexing and search, which falls in rule-based methods @xcite . a known shortcoming of rule-based approaches is that they perform well only on reports that follow a specific template and are written following strict structures. as a result, rule-based systems require updating rules/patterns for each new dataset with new formatting and structure. furthermore, rule-based approaches perform poorly on reports lacking a coherent structure and/or are not written using a predefined template. machine learning-based methods solve this problem by training models that can be applied to other datasets without substantial changes as they learn to rely on features beyond formatting and layout. @xcite presented a system based on the naïve bayes classifier to identify sections in radiology reports. @xcite employ maximum entropy to label various sections in discharged summaries and radiology reports. @xcite proposed a hybrid system to extract and label sentences from medical documents. their proposed system is composed of a rule-based module that detects the sections with labels and a machine learning classifier that detects the unlabeled sections. @xcite employed a set of rules for creating a high-confidence training set and applied support vector machines (svm) trained on additional formatting and contextual features to label the sentences from radiology and pathology reports. the main challenge in training such machine learning-based approaches is the need for a relatively large annotated training data. to the best of our knowledge, the most recent work is proposed by @xcite in which they present a system based on an rnn and a bert @xcite model for predicting sections in ehrs. they use sections from the medical literature (e.g., textbooks, journals, web content) with similar content in ehr sections. even though the existing methods address the problems mentioned earlier for the complex task of automatic structuring of radiology reports, an ensemble of several models is shown to yield lower generalization error, as opposed to training individual models @xcite . ## conclusions and future work in this work, we propose an ensemble approach for automatically labeling sentences in radiology reports with section labels. through the proposed ensemble approach, we achieve the state-of-theart performance of 97.1% on a relatively sizeable multi-site test set from mass general brigham. our proposed ensemble method is composed of three parallel models that capture various structural and contextual attributes of radiology reports, including formatting/layout, focus context, and the surrounding context. furthermore, we compared our proposed ensemble model against each of its components and concluded that the combination of all models is more accurate than any individual model. as future work, we plan to incorporate performance calibration in our ensemble model. it adds the importance of individual models to the ensemble model and enables us to obtain higher performance for unseen data. we also plan to extend this work to other types of reports, i.e., pathology reports and discharge summaries. another potential future work is to add conditional random field (crf) to our proposed model.