Exploring the Boundaries of GPT -4 in Radiology (2023)

Abstract: The recent success of general-domain large language models (LLMs) has significantly changed the natural language processing paradigm towards a unified foundation model across domains and applications. In this paper, we focus on assessing the performance of GPT-4, the most capable LLM so far, on the text-based applications for radiology reports, comparing against state-of-the-art (SOTA) radiology-specific models. Exploring various prompting strategies, we evaluated GPT-4 on a diverse range of common radiology tasks and we found GPT-4 either outperforms or is on par with current SOTA radiology models. With zero-shot prompting, GPT-4 already obtains substantial gains ( \approx 10% absolute improvement) over radiology models in temporal sentence similarity classification (accuracy) and natural language inference ( F_1 ). For tasks that require learning dataset-specific style or schema (e.g. findings summarisation), GPT-4 improves with example-based prompting and matches supervised SOTA. Our extensive error analysis with a board-certified radiologist shows GPT-4 has a sufficient level of radiology knowledge with only occasional errors in complex context that require nuanced domain knowledge. For findings summarisation, GPT-4 outputs are found to be overall comparable with existing manually-written impressions.

## introduction recently, the emergence of large language models (llms) has pushed forward ai performance in many domains; with many gpt-4 (openai, 2023) powered applications achieving and even surpassing human performance in many tasks @xcite . there is a shift in paradigm towards using a unified general-domain foundation llm to replace domain-and taskspecific models. general-domain llms enable * corresponding author: jaalvare@microsoft.com a wider range of customised tasks without the need to extensively collect human labels or to perform specialised domain training. also, with off-theshelf prompting, applying llms is easier than the traditional training pipeline for supervised models. while contemporary studies @xcite @xcite have started to explore the use of gpt-4 in the clinical domain, the readiness of gpt-4 in the radiology workflow remains to be rigorously and systematically tested. in this study, we set out the following research questions: (1) how can we evaluate gpt-4 on its ability to process and understand radiology reports? (2) how can we apply common prompting strategies for gpt-4 across different radiology tasks? (3) how does gpt-4 compare against sota radiology-specific models? to answer these questions, we established a rigorous evaluation framework to evaluate gpt-4 on a diverse range of common radiology tasks including both language understanding and generation. the evaluation covers sentence-level semantics (natural language inference, sentence similarity classification), structured information extraction (including entity extraction, disease classification and disease progression classification), and a direct application of findings summarisation. we explored various prompting strategies including zero-shot, few-shot, chain-of-thought (cot) @xcite , example selection @xcite , and iterative refinement @xcite , and we further experimented with adding self-consistency @xcite and asking gpt-4 to defer handling uncertain cases to improve the reliability of gpt-4. for each task, we benchmarked gpt-4 with prior gpt-3.5 models (text-davinci-003 and chatgpt) and the respective state-of-the-art (sota) radiology models. apart from reporting metric scores, we performed extensive qualitative analysis with a board-certified radiologist to understand the model errors by categorising them as ambiguous, label noise, or gen-uine model mistakes. we highlight the particular importance of qualitative analysis for open-ended generation tasks such as findings summariastion where gpt-4 may provide alternative solutions. to sum up, our key contributions and findings (in italics) are: gpt-4 shows a significant level of radiology knowledge. the majority of detected errors are either ambiguous or label noise, with a few model mistakes requiring nuanced domain knowledge. for findings summarisation, gpt-4 outputs are often comparable to existing manually-written impressions. ## related work there have been extensive efforts to benchmark and analyse llms in the general-domain. liang et al. (2023) benchmarks llms across broad nlp scenarios with diverse metrics. hendrycks et al. (2021) measures llms' multitask accuracy across disciplines. zheng et al. (2023) explores using llms as judge for open-ended questions. bubeck et al. the evaluation of gpt-4 has also begun to garner interest in the medical field. for example, @xcite discusses the potential advantages and drawbacks of using gpt-4 as an ai chatbot in the medical field. @xcite investigates possible applications of gpt-4 in biomedical engineering. @xcite evaluates gpt-4 for medical competency examinations and shows gpt-4 performance is well above the passing score. there have also been a few recent studies that evaluate gpt-4 in the radiology domain: @xcite show that gpt-4 significantly outperforms gpt-3.5 and exceeds the passing scores on radiology board exams. other studies have shown great potential from gpt-4 in various radiology applications such as simplifying clinical reports for clinical education @xcite , extracting structures from radiology reports @xcite , natural language inference (nli) @xcite , and generating reports @xcite . while most of these studies focus on a specific application, our study aims for an extensive evaluation to compare gpt-4 against sota radiology models, covering diverse tasks and various prompting techniques. beyond prompting gpt-4, continued efforts are being made to adapt llms to the medical domain via fine-tuning. med-palm and med-palm-2 @xcite improve over palm @xcite and palm-2 @xcite with medical-domain fine-tuning. @xcite and @xcite further fine-tune the open-source llama model @xcite with medical-domain data. @xcite adapts llms to radiology data with parameter efficient fine-tuning. while these models offer lightweight alternatives, our study focuses on gpt-4 as it is still by far the best-performing model across many domains and represents the frontier of artificial intelligence @xcite . ## conclusion this study evaluates gpt-4 on a diverse range of common radiology text-based tasks. we found gpt-4 either outperforms or is on par with taskspecific radiology models. gpt-4 requires the least prompting effort for the 'learn-by-instruction' tasks where requirements can be clearly defined in the instruction. our extensive error analysis shows that although it occasionally fails to surface domain knowledge, gpt-4 has substantial capability in the processing and analysis of radiology text, achieving near-ceiling performance in many tasks.