Can Rule-Based Insights Enhance LLM s for Radiology Report Classification? Introducing the R ad P rompt Methodology. (2024)

Abstract: Developing imaging models capable of detecting pathologies from chest X-rays can be cost and time-prohibitive for large datasets as it requires supervision to attain state-of-the-art performance. Instead, labels extracted from radiology reports may serve as distant supervision since these are routinely generated as part of clinical practice. Despite their widespread use, current rule-based methods for label extraction rely on extensive rule sets that are limited in their robustness to syntactic variability. To alleviate these limitations, we introduce RadPert, a rule-based system that integrates an uncertainty-aware information schema with a streamlined set of rules, enhancing performance. Additionally, we have developed RadPrompt, a multi-turn prompting strategy that leverages RadPert to bolster the zero-shot predictive capabilities of large language models, achieving a statistically significant improvement in weighted average F1 score over GPT-4 Turbo. Most notably, RadPrompt surpasses both its underlying models, showcasing the synergistic potential of LLMs with rule-based models. We have evaluated our methods on two English Corpora: the MIMIC-CXR gold-standard test set and a gold-standard dataset collected from the Cambridge University Hospitals.

## introduction supervised deep learning for medical imaging classification has accomplished significant milestones. in the chest x-ray (cxr) domain, such models have exhibited predictive capabilities on par with expert physicians @xcite and are being utilized in collaborative * equal contribution. annotating medical images, however, is expensive and arduous: it requires a committee of expert radiologists to resolve the inherently high degree of annotator variance and subjectivity @xcite . this issue is particularly problematic considering the global shortage of radiologists @xcite @xcite . instead, we often have access to a form of distant supervision: the radiology report. radiology reports are semi-structured free-text interpretations of an x-ray image and are generated as a routine part of clinical practice to communicate findings. in the past, rule-based models @xcite have been used to extract structured labels from radiology reports in various imaging datasets, including chestx-ray14 @xcite , chexpert @xcite , mimic-cxr @xcite and brax @xcite . however, those rule-based methods are often based on elementary techniques and, thus, exhibit limited robustness to syntactic variation. naturally, supervised deep learning models offer superior performance through their robustness to syntactic variability @xcite ). in contrast, large language models (llms) represent a significant improvement over rule-based models in an unsupervised setting and have achieved impressive performance in the field of radiology @xcite @xcite . in this paper, we present radpert, a rule-based model built on the radgraph knowledge graph @xcite . radpert leverages entity-level uncertainty labels from radgraph, reducing the need for a comprehensive rule set and enhancing its resilience to syntactic variations. we have evaluated radpert internally on mimic-cxr and externally on a dataset collected from the cambridge university hospitals (cuh). radpert surpasses chexpert, the former rule-based state-of-the-art (sota), by achieving statistically significant improvement in weighted average f1 score. furthermore, we explore the collaborative potential of llms with rule-based models through radprompt. radprompt is a multi-turn prompting strategy that employs radpert as an implicit means of encoding medical knowledge (figure 1 ). in fact, radprompt, based on gpt-4 turbo, manages to outperform both its underlying models in a zero-shot setting. ## related work numerous natural language processing methods have been developed to derive structured predictions from radiology reports @xcite @xcite @xcite . many of those approaches are designed for the multitask classification of radiology reports, written in english, into labels representing prevalent pathologies from cxrs. each such label can exhibit one of four output classes: null, positive, negative and uncertain. chexpert @xcite , the rule-based sota, follows an approach based on regular expression matching and the universal dependency graph (udg) of a radiology report. due to the rudimentary regular expression matching, however, chexpert is sensitive to syntactic variation. thus, multiple over-generalized rules are used in an attempt to alleviate these shortcomings. furthermore, the udg is a type of information extraction that does not explicitly identify negation and uncertainty. therefore, its ability to detect uncertainty in complex phrases is hampered despite the extensive rule set. extensions of chexpert have been developed for brazilian portuguese @xcite and german @xcite . chexbert @xcite is a semi-supervised model pretrained on automatically extracted labels from the chexpert model, fine-tuned on manually annotated reports, and evaluated on 687 mimic-cxr goldstandard test set reports. however, the published model weights 1 of chexbert differ from the original model. this discrepancy complicates compar-isons on the mimic-cxr dataset as the published model is fine-tuned on unspecified mimic-cxr manually annotated reports, which can potentially overlap with the mimic-cxr gold-standard test set. recent work has also explored the adoption of llms for radiology report classification. @xcite examine the zero and few-shot capabilities of llms. however, they mainly treat the task as a binary classification for each pathology. namely, for multitask classification, they only report the few-shot results on an unpublished institutional dataset. chex-gpt @xcite utilizes zero-shot gpt-4 labels as a distant supervision to fine-tune a bert-based model. nonetheless, they also simplify the task into binary classification. alternative approaches to the classification of chest x-rays (cxrs) explore moving away from the distantly supervised paradigm of training unimodal vision models on classifying structured labels extracted from radiology reports. in lieu of structured prediction, vision-language (vl) models are trained to align the embedding representations of cxrs with the representations of the corresponding radiology reports via self-supervised contrastive learning objectives @xcite @xcite @xcite . this alignment task is transformed into cxr classification through the cosine similarity of cxr embeddings to the embeddings of textual prompts representing the existence or absence of pathologies. however, vision models trained with the structured prediction paradigm outperform vl models such as chexzero @xcite , even when the latter utilizes an expert-annotated validation set for selecting optimal classification thresholds. in this paper, we will focus on improving the unsupervised sota for the multitask classification of radiology reports. ## limitations while this study demonstrates promising improvements in radiology report classification using the radprompt methodology, several limitations must be considered. radpert and radprompt are exclusively developed and tested for the english language. the study also centers around a list of pathologies typical of chest x-rays. as such, the extension of our methodologies to other languages, types of medical imaging, and additional pathologies was not verified. furthermore, previous studies have highlighted discrepancies between labels from radiology report annotations and those from the corresponding imaging study annotations @xcite . the source of such inconsistencies includes incomplete radiology report impressions, hierarchical relationships within labels, and the undeniable uncertainty of the task. in future work, we aim to study this effect within the cuh test set. due to ethical considerations, we are currently unable to perform inference for the cuh test set through third-party apis. thus, we have not evaluated radprompt externally for sota llms. we expect to overcome this limitation after the planned release of the cuh dataset. additionally, we cannot estimate the computational cost and carbon footprint for gpt-4-based radprompt due to a lack of specific metrics. in the appendix, we provide carbon footprint estimates for the llama-2-based radprompt, which is significantly higher than radpert and chexpert. nonetheless, radpert delivers performance comparable to gpt-4 while operating on a commercial cpu with minimal carbon emissions, underscoring its benefits in resource-limited environments. finally, there is an inherent degree of ambiguity in classifying radiology reports, especially as it pertains to the uncertainty labels. we aim to extend current datasets with labels from multiple annota- ## conclusions this paper introduced radpert, a rule-based system enhanced by the radgraph information schema, demonstrating significant improvements in the classification of radiology reports. by leveraging entitylevel uncertainty labels, radpert reduces reliance on comprehensive rule sets. our evaluations show that radpert surpasses chexpert, the previous rulebased sota, by achieving an 8.0% (95% ci: 5.5%, 10.8%) increase in f1 score, with confidence intervals strongly supporting this improvement. further extending the application of radpert, we developed radprompt, a multi-turn prompting strategy that utilizes insights from radpert to enhance the zero-shot prediction capabilities of large language models. radprompt demonstrated a 2.1% (95% ci: 0.3%, 4.1%) improvement in f1 score over gpt-4 turbo, indicating its potential to refine predictions in clinical settings. these results highlight the growing synergy between structured rule-based systems and large language models, offering a promising direction for future research in biomedical natural language processing. as we continue to refine these tools, future work will focus on expanding the existing datasets and addressing the discrepancies between goldstandard image labels and those extracted from radiology reports.