Zero-Shot Information Extraction for Clinical Meta-Analysis using Large Language Models (2023)

Abstract: Meta-analysis of randomized clinical trials (RCTs) plays a crucial role in evidence-based medicine but can be labor-intensive and error-prone. This study explores the use of large language models to enhance the efficiency of aggregating results from randomized clinical trials (RCTs) at scale. We perform a detailed comparison of the performance of these models in zero-shot prompt-based information extraction from a diverse set of RCTs to traditional manual annotation methods. We analyze the results for two different meta-analyses aimed at drug repurposing in cancer therapy pharmacovigilience in chronic myeloid leukemia. Our findings reveal that the best model for the two demonstrated tasks, ChatGPT can generally extract correct information and identify when the desired information is missing from an article. We additionally conduct a systematic error analysis, documenting the prevalence of diverse error types encountered during the process of prompt-based information extraction.

## introduction meta-analysis is a statistical method that combines and analyzes data from multiple studies to obtain an overall effect size or estimate of treatment effect. it is widely used in healthcare research, particularly in clinical trials, to provide a comprehensive and robust summary of the available evidence @xcite . the importance of meta-analysis lies in its ability to increase statistical power and reduce bias, thereby improving the accuracy and reliability of the findings. meta-analysis also allows for the identification of important subgroups of patients and provides insights into the potential sources of heterogeneity in the results of different studies @xcite . in the context of clinical trials, meta-analysis plays a crucial role in the evaluation of new treatments and interventions @xcite @xcite . by combining data from multiple studies, researchers can obtain a more precise estimate of the effectiveness of treatment and identify any potential adverse effects. while clinical meta-analysis is essential to establishing guidelines for clinical best-practice @xcite , curating data is timeconsuming for medical professionals. a recent survey found that most clinical meta-analyses require 6-10 months of data gathering and analysis for 5 individuals @xcite , which does not account for research development time spent before registering meta-analyses on prospero registry of systematic reviews @xcite . moreover, most meta-analyses seek to answer very targeted questions about specific diseases or drugs, making it difficult to adapt existing datasets for the automatic or semi-automatic extraction of needed data. a recent review @xcite ) examined founation models/large language models (llms), such as chatgpt, and opined that while there is evidence that clinical foundation models improve accuracy, there has been minimal work to validate other potential benefits, such as reducing data labeling burden, enabling new clinical applications, and offering novel human-ai interfaces. however, foundation models also present significant risks, including data privacy and security concerns, interpretability challenges, high up-front costs, and biases @xcite . this paper advocates for the development of new evaluation tasks, metrics, and datasets to better understand how foundation models perform on clinical tasks. to this end, it contributes the following to the development and use of foundation models in clinical and biomedical research: ## related work recent surveys and case studies have shown that natural language processing can powerfully accelerate the identification of candidates for drug repurposing @xcite and the identification of adverse effects in pharmacovigilance efforts @xcite . substantial work in clinical evidence extraction has been in the collection of patients, interventions, comparators, and outcomes (pico) from clinical trials. nye et al. (2018) and zlabinger et al. (2020) both present a corpora with pico annotations aimed at tagging the high-level characteristics of clinical trials. biomedical transformer models are currently stateof-the-art on pico tagging (alrowili and shanker, @xcite @xcite . additional improvements can be made by augmenting pico data with weak supervision from external dictionaries and hand-crafted rules @xcite . a number of works have built models to improve the screening and extracting evidence from clinical trials for meta-analysis and systematic review. @xcite automatically constructs a dataset focused on summarizing systematic reviews using the "conclusion" section of structured abstracts, and this work was extended in a recent shared task on multi-document summarization for scientific literature reviews @xcite . @xcite takes a different approach by producing an end-to-end system for screening and summarizing individual clinical cohort studies based on pico elements and clinically-relevant details annotated by epidemiologists during data curation. @xcite creates a novel, hierarchical framework for structuring evidence in clinical trials for better usability and understanding. while existing work offers many valuable resources for evidence extraction across a diverse array of tasks, it is limited to specific, narrowlyscoped tasks and requires substantial annotation efforts to work effectively. in contrast, many critical clinical meta-analyses answer very specific clinical questions which are out of the scope of existing data and require manual curation. this work differs from existing work by evaluating the effectiveness of foundation nlp models for zeroshot extraction of information for highly-specific clinical questions, enabling a much broader range of applications. ## problem setup and methodology two prior works facilitated the development of this zero-shot information extraction work, the remedy database @xcite and cml dataset registered on prospero for an ongoing metaanalysis @xcite . each was created through manual annotations of clinical trial and cohort study articles. lists of the data columns included in each dataset are included in tables 1 and 2 . ## conclusion in conclusion, this study demonstrates the potential of large language models in enhancing the efficiency of clinical meta-analyses of randomized clinical trials. by comparing its performance with traditional manual annotation methods, we provide valuable insights into the advantages and challenges of implementing ai-based solutions in evidence-based medicine. the results of our research indicate that llms can contribute to more streamlined, transparent, and reproducible results in clinical research. it also reveals that they still make significant errors and should be used cautiously with additional quality checks when used as a tool to extract data from clinical research.