Appraising the Potential Uses and Harms of LLM s for Medical Systematic Reviews (2023)

Abstract: Medical systematic reviews play a vital role in healthcare decision making and policy. However, their production is time-consuming, limiting the availability of high-quality and up-to-date evidence summaries. Recent advancements in LLMs offer the potential to automatically generate literature reviews on demand, addressing this issue. However, LLMs sometimes generate inaccurate (and potentially misleading) texts by hallucination or omission. In healthcare, this can make LLMs unusable at best and dangerous at worst. We conducted 16 interviews with international systematic review experts to characterize the perceived utility and risks of LLMs in the specific context of medical evidence reviews. Experts indicated that LLMs can assist in the writing process by drafting summaries, generating templates, distilling information, and crosschecking information. They also raised concerns regarding confidently composed but inaccurate LLM outputs and other potential downstream harms, including decreased accountability and proliferation of low-quality reviews. Informed by this qualitative analysis, we identify criteria for rigorous evaluation of biomedical LLMs aligned with domain expert views.

## introduction in the fall of 2022, meta (formerly facebook) unveiled galactica, foot_0 a large language model (llm) touted as being able to "store, combine and reason about scientific knowledge" @xcite . the prototype allowed users to enter (natural language) questions, and the model would then generate confident, scientific-sounding outputs ostensibly backed by evidence and published literature. nevertheless, like all current llms, galactica was prone to factual inaccuracies @xcite and could easily be induced to produce plainly absurd and arguably harmful outputs. for example, prompted to produce an article on "the benefits of eating crushed glass", the model fabricated a confidently written and scientificsounding human subjects study purporting to test the effectiveness of eating crushed glass to prevent the stomach from making too much acid. it asserted that this evidence indicates it is relatively beneficial to eat crushed glass: "the results of the study showed that the glass meal was the most effective at lowering stomach acid output, and the wheat bran meal was the least effective." a swift backlash ensued on social media, with individuals posting outputs featuring confidently written but scientifically inaccurate prose @xcite . such examples were widely characterized as potentially harmful and unsafe @xcite . such discourse around llms tends toward extreme positions-either hyping llms and their ability to seamlessly synthesize knowledge on-demand, or characterizing them as uniformly useless at best and harmful at worst. we argue that assessment of the potential uses and harms of llms is only meaningful when rooted in a particular context: when are llm outputs potentially dangerous, exactly, and to whom? conversely, what advantages might they confer, and for what tasks? in this work, we ground the consideration of such questions in the important context of medical systematic reviews (srs). medical systematic reviews, evidence-based medicine, and llms. one of the touted strengths of galactica (and by implication, other llms) is its ability to "synthesize knowledge by generating secondary content automatically: such as literature reviews..." @xcite . systematic literature reviews are a critical tool of evidence based medicine (ebm; @xcite . such comprehensive synopses of published findings are considered the strongest form of evidence and inform healthcare practice @xcite @xcite . however, the medical evidence base is voluminous and continues to expand at a rapid pace, which makes producing high-quality reviews of the evidence onerous @xcite . even once published, such synopses quickly go stale as new, relevant evidence accumulates @xcite . the rise of llms that are ostensibly capable of producing literature reviews for arbitrary query topics suggests the tantalizing prospect of providing on-demand synopses of the medical evidence on a given topic automatically. short of this lofty goal, llms such as galactica may make the process of humans writing syntheses more efficient by providing initial drafts or outlines. despite the excitement around llms, critical questions remain regarding the extent to which domain experts will find them useful in practice, and the degree to which the benefits outweigh the anticipated risks. answering such questions requires grounding them in specific tasks and contexts. here we focus on the important task of medical literature reviews and soliciting the opinions of experts in the field. we address the following research questions: (1) what are the perspectives of domain experts with respect to the potential utility of llms to aid the production of medical systematic reviews? (2) do domain experts anticipate any potential risks from the use of llms in this context? (3) what can we learn from domain experts which might inform criteria for rigorous evaluation of biomedical llms? as far as we are aware, this is the first effort to qualitatively characterize expert views on llms for the task of drafting medical systematic reviews. our hope is that this study will inform the development and evaluation of llms that can effectively assist experts in writing systematic reviews. ## related work prior work has sought to expedite the production of systematic medical literature reviews using ml and nlp, for example, by helping to identify relevant studies @xcite @xcite @xcite or automating data extraction @xcite @xcite . more recent work has pointed out how llms like chatgpt show some potential promise for scientific writing and aiding smaller systematic review-related tasks such as screening titles for relevance or formulat-ing structured review questions @xcite @xcite . in this work, we focus on the task of generating narrative summaries of the evidence directly. most work on this task has treated it as a standard multi-document summarization task, assuming inputs (abstracts of articles describing relevant trials) are given @xcite @xcite . here, we focus on the more audacious approach of asking the model to generate a review on a given topic without external references (i.e., on the basis of its pre-trained weights alone). this is the sort of functionality that galactica @xcite ostensibly offers and is representative of the broader trend in how llms are being used. moreover, assuming all relevant trial reports are provided as input is unrealistic in practice, as it would first require completing the time-consuming, rigorous process of a search and citation screening to identify this set. one of the promises of llms is that they might be able to implicitly perform such a search by generating a synopsis of relevant studies (ingested during pre-training) directly. galactica in particular adopts a relatively standard decoder-only stack of transformer layers. it is trained on over 48 million papers, textbooks, lecture notes, reference materials, text representations of compounds and proteins, scientific websites, and other sources of scientific knowledge @xcite . in addition to galactica (6.7b parameters), we consider two other representative models. the first is biomedlm (formerly pubmedgpt; @xcite , which is a smaller model (2.7b parameters) trained on 16 million pubmed abstracts and 5 million pubmed central full-texts. we also consider chatgpt (february 13 and march 23 versions; openai 2022), which while not trained explicitly for biomedical tasks has demonstrated considerable flexibility and is performant across domains. ## methods we applied upstream stakeholder engagement @xcite @xcite which involves eliciting responses from domain experts prior to implementing a system. we took a qualitative approach because we aim to characterize general views on llms held by domain experts, and surveys would have overly constrained responses. qualitative research allows for richer, more detailed analysis of unstructured data from a smaller number of participants. we used an intentional sampling approach to recruit interview participants, aiming to include experts with diverse experience in medical systematic reviewing (including methodologists, practitioners, clinical researchers, journal editors, and publishers in research synthesis, and clinical guideline experts who use such reviews). we detail the recruitment process and participant background in appendix appendix a. during interviews, we shared with expert participants samples of outputs from llms prompted to generate evidence summaries based on a query to act as probes @xcite to spark discussion. we used the following representative llms: galactica 6.7b @xcite , biomedlm 2.7b @xcite , and chatgpt @xcite . a schematic of the entire process we took for this qualitative study is provided in figure 1 . ## results below we present findings from our qualitative analysis, also summarized in figure 2 . ## discussion in this work, we sought to answer the question: what are the perceived potential uses and harms of using llms for aiding the production of medical systematic reviews? by engaging with varied domain experts about this question in interviews, we identified some consistent viewpoints. first, interviewees largely agreed that llms are likely to be useful for producing medical systematic reviews either as a writing tool (e.g. creating initial drafts) or summarizing data or identified text inputs (akin to more traditional multi-document summarization). however, participants also expressed concerns about the potential downstream clinical harms of such generations, e.g., individuals being misled by confidently composed but inaccurate synopses. exacerbating these issues is the lack of transparency of llms as they produce overviews of findings on topics often without explicit references (or sometimes accompanied by hallucinated references). we discussed with participating experts potential ways to improve llms for aiding medical systematic reviews, including user interface (ui) choices to clarify that model outputs are intended as drafts for editing and using llms to scaffold evidence overviews. longer-term goals include enhancing llm transparency for evidence synthesis through semi-parametric (i.e. retrieval-augmented) approaches and addressing model hallucination. our discussions with evidence synthesis experts emphasized the importance of human validation and editing of model outputs.