Summarizing, Simplifying, and Synthesizing Medical Evidence using GPT -3 (with Varying Success) (2023)

Abstract: Large language models, particularly GPT-3, are able to produce high quality summaries ofgeneral domain news articles in few- and zero-shot settings. However, it is unclear if such models are similarly capable in more specialized domains such as biomedicine. In this paper we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given no supervision. We consider bothsingle- and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in thelatter, we assess the degree to which GPT-3 is able to synthesize evidence reported acrossa collection of articles. We design an annotation scheme for evaluating model outputs, withan emphasis on assessing the factual accuracy of generated summaries. We find that whileGPT-3 is able to summarize and simplify single biomedical articles faithfully, it strugglesto provide accurate aggregations of findings over multiple documents. We release all data,code, and annotations used in this work.

## introduction large language models have been shown to be capable of producing high-quality and reasonably accurate summaries in zero-shot settings @xcite , with gpt-3 besting fully supervised models in generic news summarization, according to human judgments @xcite . in this work we evaluate if such models are similarly able to summarize medical literature, a highstakes domain that demands factual accuracy. specifically, we use the newest iteration of gpt-3 (text-davinci-003; gpt3-d3 from here) to generate summaries of (a) individual articles describing individual randomized controlled trials (rcts) 1 https://github.com/cshaib/ summarizing-medical-evidence ## single document summarization data we sample 100 articles describing randomized control trials (rcts) indexed in the trialstreamer database @xcite , which also provides automatically extracted "key results" foot_0 alongside titles and abstracts. we search for trials published after november 28 2022, following the release date of gpt3-d3, to ensure the model has not seen any of the studies during pre-training. experimental setup using the rct data described above, we evaluate the ability of gpt3-d3 to faithfully summarize and simplify biomedical texts in a zero-shot setting. we also compare gpt3-d3 summaries to summaries generated using flan-t5 @xcite , but qualitatively find that gpt3-d3 summaries are much higher quality. we provide results of this comparison in appendix f.3. specifically, we prompt gpt3-d3 to separately produce: (i) a technical summary, and, (ii) a plain language summary @xcite . see appendix c for all prompts. ## multiple document summarization and evidence synthesis data for multi-document summarization, we download meta-analyses from the cochrane library (these are reviews of medical evidence, usually rcts). 5 our final sample contains 50 multidocument studies comprising meta-review titles, reference abstracts (inputs), and target conclusions (target summaries) written by domain experts, 10 of which were published post-gpt3-d3 release. 6 experimental setup because inputs comprise multiple abstracts, these (together with generated tokens) often exceed the token capacity of gpt3-d3. in our dataset, about 41% of the samples exceeded this upper-bound. we report information about our data, including average length, in appendix b. to address the upper-bound problem, we adopt a simple two-phase strategy for multi-document summarization. first, we generate independent summaries for each abstract, using the single-document summarization prompt described in section 2. then, we include all the generated single-document summaries in our multi-document synthesis prompt 7 (examples in appendix c). ## results rq1: does gpt3-d3 produce faithful summaries of medical articles? in the single document setting, we find that gpt3-d3 generates summaries of biomedical abstracts that are fairly highquality. figure 2 (a) shows that annotators rated a majority of the summaries as being coherent, useful, and capturing "key results". when gpt3-d3 does err, it tends to make minor mistakes or omit details. the latter is more common than the former, as shown in figure 3 (a) . rq2: can gpt3-d3 accurately simplify while summarizing medical texts? shown in figure 2 (b), gpt3-d3 produces simplified summaries that are similarly deemed to be coherent and useful, and which appear to contain key results. simplified outputs are scored highly in terms of readability, indicating that these summaries would be understood by someone without medical training. in comparison to the technical summaries, figure 3 (b) shows that there are fewer omissions but a slightly higher amount of errors. these may be problematic, but -importantly -some omissions are expected in a simplified summary, as certain details that are important for an accurate summary for a technical audience may not be necessary to convey key information to a more general audience. rq3: can gpt3-d3 synthesize findings presented in multiple input articles in a way that accurately reflects the totality of the evidence? we now evaluate gpt3-d3's performance on multidocument summarization, i.e., its ability to synthesize evidence @xcite . figure 4 shows that most summaries generated by gpt3-d3 in this setting are supported by the inputs. this is consistent with our findings in rq1: gpt3-d3 is able to summarize faithfully with respect to given input. however, we find that generated summaries do not consistently agree with the target summaries. indeed, figure 4 shows that generated summaries disagree with the targets in over half of cases. this discrepancy suggests that human-written summaries in the biomedical domain require a level of synthesis that is not captured by gpt3-d3 . rq4: what sort of factual mistakes does gpt3-d3 make and what are the risks? in rq1, we reported that gpt3-d3 sometimes omits key information. figure 5 characterizes the types of omissions and errors made, with respect to pico elements. gpt3-d3 tends to underspecify elements in the summary more often than generating inaccuracies. appendix f provides further details regarding underspecification. in the simplification task, gpt3-d3 capably simplifies most technical terms in the generated output (figure 6 ). regarding rq3, we showed that there are often discrepancies between generated and target summaries, despite the former being supported by the inputs. human-written summaries of trials may be number of summaries all mostly all only a few none figure 6: in the simplification case, the model usually replaces complex terms with simpler ones. overall, we find that gpt3-d3 copies frequently from inputs. this results in summaries that are often faithful to the input. it may also be one reason that summaries tend to have more omissions (rather than errors) in the single document case, and it may also explain how summaries in the multi-document case often disagree with the reference synopsis while also being supported by (some subset of) the inputs. we calculate the degree of overlap and similarity between inputs and generated summaries from gpt3-d3 for both single-document and multidocument summarization at the sentence level (figure 8). gpt3-d3 often copies sentences verbatim. in other cases, it changes phrasings but only very slightly (see appendix f for examples). further, figure 8 shows how many sentences in each summary have a bleu score @xmath0; which indicates the sentences are highly aligned. over 70% of the summaries have at least a quarter of the sentences copied from the input. appendix f shows some examples of highly similar summaries and sentence pairs. ## related work more broadly in summarization, several efforts have called for increased emphasis on human (rather than automated) evaluation of generated texts, increased deployment of human-centered systems for text generation evaluation @xcite , and greater focus on building benchmarks that incorporate human preferences @xcite . and indeed, @xcite find that summaries produced by gpt3-d3 are often preferred by humans over alternative model outputs even when automated metrics disagree. such findings have motivated the manual analysis we conduct for this work. as far as we know, there has not been any work that assess the degree to which gpt-3 is proficient at summarizing biomedical and clinical data in both single-document and multi-document cases. our analysis of summarization in the biomedical space complements recent work analyzing the question answering capabilities of such models in this domain @xcite and the degree to which they encode medical knowledge implicitly @xcite . other work has considered using summarization of biomedical texts as assistive tools for reading @xcite . ## conclusions we evaluate the ability of gpt3-d3 to faithfully summarize and simplify medical literature. the expert annotations we collect indicate that gpt3-d3 performs single-document tasks quite well, but struggles with multi-document summarization. this highlights the ability to aggregate across documents as a direction for future work. we release all data and annotations to facilitate such work in the medical space going forward.