Leveraging Prompt-Learning for Structured Information Extraction from Crohn’s Disease Radiology Reports in a Low-Resource Language (2024)

Abstract: Automatic conversion of free-text radiology reports into structured data using Natural Language Processing (NLP) techniques is crucial for analyzing diseases on a large scale. While effective for tasks in widely spoken languages like English, generative large language models (LLMs) typically underperform with less common languages and can pose potential risks to patient privacy. Fine-tuning local NLP models is hindered by the skewed nature of real-world medical datasets, where rare findings represent a significant data imbalance. We introduce SMP-BERT, a novel prompt learning method that leverages the structured nature of reports to overcome these challenges. In our studies involving a substantial collection of Crohn’s disease radiology reports in Hebrew (over 8,000 patients and 10,000 reports), SMP-BERT greatly surpassed traditional fine-tuning methods in performance, notably in detecting infrequent conditions (AUC: 0.99 vs 0.94, F1: 0.84 vs 0.34). SMP-BERT empowers more accurate AI diagnostics available for low-resource languages.

## introduction medical imaging, particularly computed tomography (ct) and magnetic resonance imaging (mri), emerges as a key element in the management of complex conditions such as crohn's disease (cd) @xcite serving as a cornerstone for diagnosis, monitoring, and guiding treatment decisions @xcite . large-scale analyses of imaging data in cd hold promise for advancing research on the inflammatory burden in the bowel and developing predictive models of disease progression @xcite . the critical clinical information extracted from these images is typically embedded in free-text radiology reports, presenting a significant challenge for large-scale analysis. manually extracting phenotypes and other pertinent information from radiology reports is laborintensive and requires domain-specific expertise in radiology. furthermore, cd exhibits high heterogeneity in the disease course, necessitating manual evaluation of a wide range of potential conditions @xcite . this task's time-consuming nature and impracticality for large-scale applications pose significant challenges in achieving efficient and accurate data extraction. recent attempts to automate this extraction process have utilized generative large language models (llms) such as gpt-4, which leverage free-text instructions instead of requiring annotated data for training @xcite . while these models hold promise, concerns regarding low-resource languages and data privacy remain a challenge. other approaches have involved directly finetuning open-source language models on a manually labeled subset of the data @xcite et al., 2022). however, fine-tuning performance suffers from significant data imbalance, a common challenge in medical datasets and particularly in the case of cd, which features some rare conditions. to address these limitations, we propose smp-bert, a novel prompt learning method built upon the "pre-train, prompt, and predict" framework @xcite , specifically tailored for the structured nature of radiology reports. smp-bert leverages a new pre-training task called section matching prediction (smp). this task leverages the structured format of radiology reports, where key findings reside in some "impression" section. by pre-training on this task, smp-bert can infer in a zero-shot setting and also further fine-tune using a relatively small amount of annotated data. this approach not only mitigates the challenge of data imbalance but also eliminates the need for massive training corpora during pre-training. this advantage makes smp-bert readily applicable to low-resource languages, paving the way for a more inclusive and efficient method of extracting information from radiology reports. ## results to account for the inherent class imbalance in our dataset, we focused our analysis on phenotypes with at least 10 positive samples, ensuring the reliability of our findings. further analysis presented in figure 7 of model performance relative to the count of positive instances exhibited the strength of smp-bert + tuning, particularly for labels with sparse positives in the training set. for example, with only 19 positive cases for "rectum bowel wall thickening," smp-bert + tuning achieved a significantly higher f1score (0.74) compared to the standard model (0.1). this demonstrates its superior ability to generalize well from limited data. however, both models performed well when dealing with abundant positive instances. for example, with 137 positives for "ileum bowel wall thickening" (almost half the dataset), both models achieved good results, with smp-bert + tuning maintaining a decent gap (f1-score 0.97 vs. 0.915 for the standard model). the graph shown in figure 7 indicates that the performance gap between the models decreases with an increase in the number of positive instances. this suggests that while smp-bert + tuning shines with limited data, it still performs better when more data is available. we also analyzed how the size of the training set impacts model performance. as shown in figure 6 , the smp-bert + tuning model exhibits superior adaptability. notably, it achieves good performance even with limited training data (50-100 samples). the standard fine-tuning model exhibits a trend of broadening iqrs and decrease of median score. this could suggests an improving performance for common phenotypes (like ileum bowel wall thickening) but potentially decreasing performance for rarer ones due to increased data imbalance. ## discussion this study examined the efficacy of smp-bert, a novel prompt-learning approach, in extracting detailed information from hebrew radiology reports of cd patients. our results reveal that smp-bert, especially the fine-tuned version (smp-bert + tuning), significantly outperforms the standard finetuning approach, , achieving an improvement of 49% in median f1 score and 5% in median auc. our study highlights the significant improvement of smp-bert + tuning, achieving superior f1scores and aucs compared to standard fine-tuning across all analyzed phenotypes. notably, the model performs well even with a low amount of annotated data. this improvement is particularly notable for rarer phenotypes, demonstrating the model's ability to handle imbalanced datasets, a common challenge in the medical domain. this robustness is crucial for advancing research in cd and other conditions with diverse clinical presentations. furthermore, this study contributes to the growing exploration of prompt learning for nlp tasks in healthcare. unlike traditional fine-tuning approaches, which require substantial labeled data, smp-bert leverages pre-training on the "section matching prediction" task and further smp-tuning to achieve exceptional performance even with limited data. this opens exciting possibilities for applying prompt learning in scenarios with limited annotated data, imbalanced data, or low-resource languages, pushing the boundaries of nlp applications in healthcare.