Catching Misdiagnosed Limb Fractures in the Emergency Department Using Cross-institution Transfer Learning (2023)

Abstract: We investigated the development of a Machine Learning (ML)-based classifier to identify abnormalities in radiology reports from Emergency Departments (EDs) that can help automate the radiology report reconciliation process. Often, radiology reports become available to the ED only after the patient has been treated and discharged, following ED clinician interpretation of the X-ray. However, occasionally ED clinicians misdiagnose or fail to detect subtle abnormalities on X-rays, so they conduct a manual radiology report reconciliation process as a safety net. Previous studies addressed this problem of automated reconciliation using ML-based classification solutions that require data samples from the target institution that is heavily based on feature engineering, implying lower transferability between hospitals. In this paper, we investigated the benefits of using pre-trained BERT models for abnormality classification in a cross-institutional setting where data for fine-tuning was unavailable from the target institution. We also examined how the inclusion of synthetically generated radiology reports from ChatGPT affected the performance of the BERT models. Our findings suggest that BERT-like models outperform previously proposed ML-based methods in cross-institutional scenarios, and that adding ChatGPT-generated labelled radiology reports can improve the classifier’s performance by reducing the number of misdiagnosed discharged patients.

## introduction when a patient presents to the emergency department (ed) with a possible limb fracture, ed clinicians order an x-ray from the radiology department. following imaging, a radiologist authors a report stating the radiological observations and diagnosis, which is then sent back to the ed clinician requesting the procedure. unlike radiology images, radiology reports may not be completed before a patient * conducted this research while affiliated with csiro. leaves the ed. in such cases, ed clinicians interpret radiological images themselves @xcite . occasionally, ed clinicians misdiagnose radiological evidence such as subtle limb abnormalities (e.g., small fractures, dislocations or foreign bodies), resulting in patients being discharged without appropriate treatment @xcite . as a safety net, ed clinicians retrospectively reconcile radiology report findings with ed discharge diagnoses to detect potential misdiagnoses @xcite . since the radiology report reconciliation process is retrospective and performed manually, it may take several days to identify and notify a misdiagnosed patient, exposing them to potentially adverse impacts on their health @xcite . machine learning (ml)-based methods for classifying radiology reports @xcite @xcite @xcite have the potential to streamline and semi-automate the radiology report reconciliation process. however, the development of ml solutions is dependent on the availability of large and diverse labelled datasets from target hospitals for model training @xcite . while radiology reports may be readily available, labelling them requires domain expertise, is time-consuming and costly @xcite . therefore, individual departments or hospitals may not have the capacity to collect sufficiently large datasets of labelled radiology reports to conduct their own model training @xcite . cross-institution transfer learning, in which datasets and model training from one institution are used to start the ml model development at another institution, may solve this problem. however, for cross-institution transfer learning to be useful for developing local ml models for radiology report reconciliation, it must be resilient to interinstitutional variations in reporting styles, lan-guage, and verbosity @xcite . many pre-trained transformer-based language models have achieved state-of-the-art performance in various benchmark datasets @xcite , especially bidirectional encoder representations from transformers (bert) @xcite . in this study, we investigated the benefits of pre-trained bert-like models on a radiology report classification task in cross-institutional environments, where labelled data from the target institution are unavailable. although domain-specific pre-training appears to be effective for in-domain applications @xcite , little is known about the impact of pre-training with different corpora on a radiology report classification task in cross-institution settings. therefore, we focused on answering the following research questions: ## related work common challenges of supervised ml models that support clinical decisions arise from limited clinical data and the lack of their labels, especially when the model is trained with data from a single hospital @xcite . the lack of labelled samples from a target hospital has previously been addressed by using transfer learning @xcite , leveraging training sets with labelled @xcite and unlabelled @xcite data from multiple institutions. however, there are no prior examples of scenarios in which there is no training data at all from the target hospital. in this study, we investigated whether ml automation of the radiology report reconciliation process in a target ed could rely on a training data set that originated from an entirely different hospital. for our purposes, the model architecture of choice must be able to generalise well across institutions. methods relying on feature engineering, such as support vector machine (svm), naïve bayes, or random forest, are not suitable for crossinstitution settings since features engineered for a dataset collected at one institution may not be the best fit for data collected at another institution @xcite . this was also observed by @xcite who found a significant reduction in performance (f1-score) of up to 10-12% in svm-based radiology report classifiers, when the training source institution was different from the target, the test institution. @xcite further demonstrated the dependency on pre-defined feature engineering by showing improved f1-score of 5-10% across hospitals when employing self-feature-extracting cnns with feature adoption transfer. however, to achieve such improvements in performance still required training data from the target institution. unlike svms and cnns, transformer models take advantage of the attention mechanism capable of extracting textual features (location, context, syntactic structure, and semantics), which leads to better performance @xcite . transformer-based models, such as pre-trained bert models, are some of the most successful deep learning (dl) models for natural language processing (nlp) across domains @xcite . therefore, we chose pre-trained bert-like models for the current study. data synthesis is one technique that can mitigate the shortage of labelled training/fine-tuning data. we determined whether chatgpt-generated synthetic radiology reports could be used to augment training or fine-tuning datasets for the purpose of reconciliating radiological findings. additionally, we evaluate the impact of chatgpt-generated reports on the performance of the bert-based abnormal radiology report classifier when chatgptsynthesised reports are included in the fine-tuning dataset. although chatgpt has already been ex-plored for data augmentation @xcite , little has been studied to evaluate the impact of chatgpt-generated radiology reports on increasing the performance of the bert-based classification model, fine-tuned on real samples with and without synthetic reports. ## materials and methods chatgpt. chatgpt foot_0 , recently developed by ope-nai, is one of the largest language models to date (about 175 billion parameters) based on gpt-3 @xcite . chatgpt is a generative language model that is designed to generate natural language according to some input prompt. the quality of its generated language is driven in part by the extensive text it was provided as part of the training process. data. in this study, we used four datasets of freetext limb structure radiology reports; three acquired from the ed of three australian public hospitals (2378 reports), and a synthetic dataset created using chatgpt (100 reports). the hospital-acquired radiology reports comprise anonymised adult, children, and mixed (adult and children) reports from three hospitals located in southeast queensland, australia. ethical approval for the acquisition of these data was granted by the human research ethics committee of the royal brisbane and women's hospital. real free-text radiology reports were manually assessed by two emergency medicine physicians as either "normal" (no fractures, dislocations, or foreign bodies present) or "abnormal" (fractures, dislocations, or foreign bodies present). a software tool was developed to help physicians record their interpretations and highlight the relevant portions of text in the reports. initially, the assessors agreed on the annotations of 2,215 out of 2,378 reports. a senior physician was then asked to act as a third assessor and resolve disagreements. the dataset distribution from three hospitals (rbwh, rch and gch), including the number of reports, the proportion of normal and abnormal cases, the average length of words and the number of unique words in the dataset, are presented in the 100 synthetic radiology reports -50 normal and 50 abnormal -were generated using chatgpt prompts listed in appendix while each model was pre-trained on different corpora, we benchmarked the mentioned models to determine the impact of model pre-training on a classification task on our mixed datasets (rbwh, rch, and gch). since our dataset is relatively small, consisting of only 2378 radiology reports from all three hospitals, we chose to evaluate the pre-trained models under test with 5-fold crossvalidation. each model was fine-tuned for ten epochs per fold, with a learning rate of 9e-6 and randomly selected seed of 112. we compared f scores, precision, recall, and matthew's correlation coefficients (mcc) between the models. rq2 -can we train a model that is transferable between institutions without relying on samples from the target institution? we compare a transformer-based pubmedbert model with the svm and cnn models on the abnormality classification task, in a cross-institutional setting, previously reported in @xcite and @xcite , respectively. we selected pubmedbert since it achieved slightly higher, but not significantly better, performance across all four metrics as a result of answering rq1. to compare pubmedbert with previously proposed methods @xcite , we trained pubmedbert models on data from two out of three hospitals and tested them in the remaining one. in other words, we considered the three fine-tuning/testing splits, namely 1) fine-tuning on rbwh + rch, testing on gch, 2) fine-tuning on rbwh + gch, testing on rch, 3) fine-tuning on rch + gch, testing on rbwh. pubmedbert was fine-tuned for ten epochs, with the learning rate of 9e-6 and the seed value of 112, to keep it consistent with the experimental set-up in rq1. we compared f1 scores between pubmedbert in the current study, and svm and cnn models from previous studies @xcite . rq3 -what is the impact of using chatgpt as an additional data source of synthetic data for classification model fine-tuning? we investigate the benefits of including synthetic reports generated by chatgpt while fine-tuning the pubmedbert on the radiology report abnormality classification task. we fine-tuned six pubmedbert models on three datasets (rbwh, rch and gch) separately with and without synthetic reports generated by chatgpt. the model fine-tuning was performed in consistence with the experimental setup of rq1 and rq2, where each model was fine-tunned for ten epochs, with the learning rate of 9e-6 and the seed value of 112. we evaluated each fine-tuned model on the remaining two real datasets (e.g., the model trained on rbwh we evaluated on rch and gch datasets). the model evaluation consists of an f1 score and a confusion matrix, including the number of true positives (tp), true negatives (tn), false positives (fp) and false negatives (fn) per fine-tuned model. rq4 -how does adding chatgpt synthesised radiology reports to fine-tuning dataset impact the downstream reconciliation task? to assess the impact of chatgpt-generated reports (used in fine-tuning) on patient data reconciliation, the radiology report classification results of both models -pubmedbert fine-tuned with and without chatgpt-generated reports (rq3) -were cross checked with the patient's icd-10 discharge diagnosis of the ed. since some of the icd-10 codes were unavailable or missing from the data received from the ed, we performed patient data reconciliation on available 1429/1480 rbwh, 495/498 rch and 329/400 gch records. following the experimental design used to address rq3, we evaluated two groups of pubmedbert models, fine-tuned on records from a single hospital with and without chatgpt-generated reports, on the downstream task of automatic reconciliation of radiology reports and discharge diagnoses. the evaluation of these two fine-tuned pubmedbert model groups was performed on the datasets from the remaining two hospitals. based on the classification results, there were four possible combinations of the radiology report classification / ed discharge diagnosis results: 1) both abnormal; 2) both normal; 3) radiology abnormal but ed normal; and 4) radiology normal but ed abnormal. ## experiments and results rq1 -what is the impact of pre-training on the abnormal radiology report classification task? figure 1 shows the fine-tuned means and standard deviations for f-score, precision, recall and mcc across 5-folds for each of the six pre-trained bert-based models. both bioelectra models, the base and the large models were excluded from the comparison since the models did not converge and always predicted the same (abnormal) class. figure 1 shows that the pubmedbert model achieves the highest performance across all four metrics (f1-score, precision, recall and mcc). to determine the significance of the difference in performance between models, we calculated twosided 95% wilson confidence intervals (figure 1 -right). models with confidence intervals that do not overlap are regarded significantly different at @xmath0.05. overlapping of the wilson confidence intervals suggests that the performances of pub-medbert, bert, bioclinicalbert, bluebertbase and linkbert (base and large) were not significantly different from each other; however, all those models performed significantly better than the bluebert-large model. rq2 -can we train a model that is transferable between institutions without relying on samples from the target institution? the results of the abnormal report classification performance achieved by pubmedbert models and their comparison with the earlier reported performance of the svm and cnn models are presented in although these trade-offs do not manifest as a clear improvement in metrics such as the f1-score ( rq4 -how does adding chatgpt synthesised radiology reports to fine-tuning dataset impact the downstream reconciliation task? the results obtained, detailed in ## discussion and conclusion we determined that pubmedbert was the bestperforming of six pre-trained bert-like models for classifying free-text radiology reports of x-rays for suspected limb fractures in ed patients. compared to svm and cnn models, pubmedbert had better performance (measured by f1-score) for classifying radiology reports when training data and testing data were from different hospitals, suggesting that pubmedbert has better transferability in cross-institution settings, especially in a low-data regime where the data from the target hospital is unavailable. we also found that pubmedbert models, which included some chatgpt-generated synthetic radiology reports in fine-tuning, resulted in higher numbers of true positives and false positives and lower numbers of true negatives and false negatives than models without synthetic reports. the trade-off in detecting more true positives, using the model enhanced by chatgpt data, is that there were also more false positives. while this implies that more patients with misdiagnoses would be identified, it also increases the number of reports that must be manually reconciled. this is an important observation in the reconciliation process since the higher number of fps has less severe consequences on reconciliation than the higher number of fns. this is because every fp-classified radiology report would require manual clarification, and every fn-classified report stands for a misdiagnosed case. nevertheless, if all radiology reports are requried to be reviewed, as is done in current practice, our approach to reconciliation can allow patient cases to be prioritised for clinical follow-up such that suspected misdiagnosed cases would be prioritised for manual review. to address the issue of data imbalance, it is common in the literature to perform over-or undersampling when developing prediction models @xcite 2022). using synthetic data generated from chat-gpt can be viewed as another approach to augment modeling by changing the data distribution. we demonstrate that using synthetic data reduces the number of unwanted predictions, such as false negatives. this shows that augmenting with chatgpt has a similar effect to balancing the data distribution by increasing the sample size of rare classes, in this case the abnormal diagnoses. overall, when developing a solution for automated reconciliation of radiology reports and discharge diagnoses in a setting where labelled radiology reports from the target institution are unavailable, pre-trained transformer models such as pub-medbert fine-tuned on available labelled reports from partner institutions, together with chatgptsynthesised radiology reports can boost the automatic reconciliation performance. as we showed the promise of using nlp models to facilitate diagnosis reconciliation for ed clinicians, more works may investigate similar approaches to streamline the manual review process, flag mismatches, and explore workflow integration.