Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT (2020)

Abstract: The extraction of labels from radiology text reports enables large-scale training of medical imaging models. Existing approaches to report labeling typically rely either on sophisticated feature engineering based on medical domain knowledge or manual annotations by experts. In this work, we introduce a BERT-based approach to medical image report labeling that exploits both the scale of available rule-based systems and the quality of expert annotations. We demonstrate superior performance of a biomedically pretrained BERT model first trained on annotations of a rule-based labeler and then finetuned on a small set of expert annotations augmented with automated backtranslation. We find that our final model, CheXbert, is able to outperform the previous best rules-based labeler with statistical significance, setting a new SOTA for report labeling on one of the largest datasets of chest x-rays.

## introduction the extraction of labels from radiology text reports enables important clinical applications, including large-scale training of medical imaging models @xcite . many natural language processing systems have been designed to label reports using sophisticated feature engineering of medical domain knowledge @xcite . on chest x-rays, the most common radiological exam, rule-based methods have been engineered to label some of the largest available datasets @xcite . while these methods have generated considerable advances, they have been unable to capture the full diversity of complexity, ambiguity and subtlety of natural language in the context of radiology reporting. more recently, transformers have demonstrated success in end-to-end radiology report labeling @xcite . however, these methods have shifted the burden from feature engineering to manual annotation, requiring considerable time and expertise for high quality. moreover, these methods do not take advantage of existing feature-engineered labelers, which represent state-of-the-art on many medical tasks. ## related work many natural language processing systems have been developed to extract structured labels from free-text radiology reports @xcite @xcite @xcite @xcite . in many cases, these methods have relied on heavy feature engineering that include controlled vocabulary and grammatical rules to find and classify properties of radiological findings. negex @xcite , a popular component of rule-based methods, uses simple regular expressions for detecting negation of findings and is often used in combination with ontologies such as the unified medical language system (umls) @xcite . negbio @xcite , an extension to negex, utilizes universal dependencies for pattern definition and subgraph matching for graph traversal search, includes uncertainty detection in addition to negation detection for multiple pathologies in chest x-ray reports, and is used to generate labels for the chestx-ray14 dataset @xcite . the chexpert labeler @xcite improves upon negbio on chest x-ray report classification through more controlled extraction of mentions and an improved nlp pipeline and rule set for uncertainty and negation extraction. the chexpert labeler has been applied to generate labels for the chexpert dataset and mimic-cxr @xcite , which are amongst the largest chest x-ray datasets publicly available. deep learning approaches have also been trained using expert-annotated sets of radiology reports @xcite . in these cases, training set size, often driving the performance of deep learning approaches, is limited by radiologist time and expertise. @xcite trained cnns with glove embeddings @xcite on 1000 radiologist-labeled reports for classification of pulmonary embolism in chest ct reports and improved upon the previous rule-based sota, pefinder @xcite . @xcite trained both recurrent and convolutional networks in combination with attention mechanisms on 27,593 physician-labeled radiology reports and apply their labeler to generate labels. more recently, transformer-based models have also been applied to the task of radiology report labeling. @xcite trained classifiers using bert @xcite and xlnet @xcite on 3,856 radiologist labeled reports to detect normal and abnormal labels. @xcite developed alarm, an mri head report classifier on head mri data using biobert @xcite models trained on 1500 radiologist-labeled reports, and demonstrate improvement over simpler fixed embedding and word2vec-based @xcite models @xcite . our work is closely related to approaches to reduce the number of expert annotations required for training medical report labelers @xcite @xcite . a method of weak supervision known as data programming @xcite has seen successful application to medical report labeling: in this method, users write heuristic labelling functions that programmatically label training data. @xcite used data programming to incorporate labeling functions consisting of regular expressions that look for phrases in radiology reports, developed with the help of a clinical expert in a limited time window, to label for intracranial hemorrhage in head cts. @xcite demonstrated that in under 8 hours of cumulative clinician time, a data programming method can approach the efficacy of large hand-labeled training sets annotated over months or years for training medical imaging models, including chest x-ray classifiers on the task of normal / abnormal detection. beyond data programming approaches, @xcite developed a fully unsupervised approach utilizing a siamese neural network and gaussian mixture models, reporting performance similar to the chexpert labeler without requiring any radiologistlabeled reports on the simplified task of normal / abnormal detection. concurrently developed to our work is the chexpert++ labeler @xcite , which was trained on the outputs of the rule-based chexpert labeler and showed improved performance after a single additional epoch of fine-tuning on expert-labeled report sentences. ## limitations our study has several limitations. first, our hybrid/auto approaches require an already-existing labeler. second, our report labeler has a maximum input token size of 512 tokens, but this may be easily extended to work with longer medical/radiology reports. in the chexpert dataset, we found that only 3 of the 190,460 report impressions were longer than 512 tokens. third, our task is limited to the 14 observations labeled for, and we do not test for the model's ability to label rarer conditions. however, chexbert can mark no finding as blank, which can indicate the presence of another condition if the other 13 conditions are also blank. fourth, the ground truth labels for the mimic-cxr test set were determined by a single board-certified radiologist, and the use of more radiologists could demonstrate a truer comparison to the radiologist benchmark. fifth, while we do test performance on a dataset from an institution unseen in training, additional datasets across institutions could be useful in further establishing the model's ability to generalize. ## conclusion in this study, we propose a simple method for combining existing report labelers with handannotations for accurate radiology report labeling. in this method, a biomedically pretrained bert model is first trained on the outputs of a labeler, and then further fine-tuned on the manual annotations, the set of which is augmented using backtranslation. we report five findings on our resulting model, chexbert. first, we find that chexbert outperforms models trained only on radiologistlabeled reports, or only on the existing labeler's outputs. second, we find that chexbert outperforms the bert-based model not pretrained on biomedical data. third, we find that chexbert outperforms models which do not use backtranslation. fourth, we find that chexbert outperforms the previous best labeler, chexpert (which was rules-based), with an improvement of 0.055 (95% ci 0.039, 0.070) on the f1 metric; we also find that the best model trained only on manually labeled radiology reports (tblue-rad-bt) performs at least as well as the chexpert labeler. fifth, we find that chexbert is 0.007 f1 points from the radiologist performance benchmark, suggesting that the gap to ceiling performance is narrow. we expect this method of training medical report labelers is broadly useful within the medical domain, where collection of expert labels can produce a small set of high quality labels, and existing feature engineered labelers can produce labels at scale. extracting highly accurate labels from medical reports by taking advantage of both sources can enable many important downstream tasks, including the development of more accurate and robust medical imaging models required for clinical deployment.