Beyond Abstracts: A New Dataset, Prompt Design Strategy and Method for Biomedical Synthesis Generation (2024)

Abstract: The biomedical field relies on cost and time intensive systematic reviews of papers to enable practitioners to keep up to date with research. Impressive recent advances in large language models (LLMs) have made the task of automating at least part of the systematic review process feasible, but progress is slow. This paper identifies some factors that may have been holding research back, and proposes a new, enhanced dataset and prompting-based method for automatic synthesis generation, the most challenging step for automation. We test different models and types of information from and about biomedical studies for their usefulness in obtaining high-quality results.We find that, surprisingly, inclusion of paper abstracts can worsens results. Instead, study summary information, and system instructions informed by domain knowledge, are key to producing high-quality syntheses.

## introduction medical practitioners need to keep up to date with the latest medical research, but the ever increasing volume of studies makes it difficult to separate signal from noise. the goal of systematic reviews is to synthesise all relevant evidence for a clinical query @xcite and provide clear, up-to-date answers based on high-quality research. systematic reviews are considered the most reliable form of evidence in the biomedical field. consequently, they have a huge influence on the medical decisions made by doctors, health authorities and individuals. producing systematic reviews is a slow and costly process. a study in 2019 estimated that the average cost of producing a biomedical systematic review was $141,194.80 @xcite . the high cost is due to reviewers having to sift through hundreds or thousands of potentially relevant studies to find the high quality studies that * equal contribution are included in their final analysis which must then undergo rigorous statistical analysis before a final conclusion is reached. unsurprisingly, there is a lot of interest in automating different steps in the process, and recent advancements in llms offer a promising avenue to do just this. prior work in this area has tended to take an endto-end approach to the task and to use limited information about reviews and included studies in the input which does not reflect a deep enough understanding of the systematic review process. below we start by setting out this process and the information collected in repositories like the cochrane library (section 2), followed by an overview of previous work where we identify important details not included in prior work that may be useful in solving the task (section 3). we use these insights to create a new, richer dataset for the biomedical synthesis task (section 4), and a new promptingbased approach to generating biomedical scientific syntheses (section 5). we show the promise of this new approach via evaluation with diverse metrics and discuss key observations (section 8). we make our dataset and code available on github. foot_0 ## background the cochrane library is one of the most highly respected institutions for creating systematic medical reviews, which it collects as the cochrane reviews, a public repository of systematic reviews. the following are identified foot_1 as the key steps in creating a cochrane review: (1) identification of relevant studies; (2) selection of studies for inclusion / evaluation of their strengths and limitations; our focus in the work presented here is on the fourth step where experts review the data to form the final conclusion of the systematic review. conclusions are typically provided in both quantitative and qualitative forms. our focus is on automating the qualitative analysis of systematic reviews. we leave the automation of pico (population, intervention, comparator and outcome) extraction and quantitative analysis (see 4.6) to future work. figure 1 provides an overview of the entire process, with the portion enclosed within the dashed box indicating the part that our approach focuses on automating. ## a new enriched dataset for systematic review synthesis in this section we describe our new dataset and how it differs from datasets used in previous work. ## biomedical synthesis generation via llm prompting the complete biomedical synthesis generation task takes as input (a) a research question, and (b) a repository of papers, and produces as output a text representing the answer to the question based on the scientific consensus as evidenced by the papers in the repository. in the work presented here, we address part of the complete task. rather than starting from the raw papers, we avail of the meta-information available in the cochrane library, to test what performance can be achieved when such information is available in high-quality form (here human produced). our basic approach is to put (a) the research question, and (b) information representing each included study we wish to use as evidence to an llm in a prompt, and interpret the llm response as the answer to the question. more specifically, for each systematic review in our dataset, we use its title (e.g. care delivery and self-management strategies for children with epilepsy) as the question, and the key information from all included studies as the evidence set (as illustrated in figure 2 and described in section 5.1). to evaluate the quality of the answer (synthesis) generated by the model, we compare it to the human-authored synthesis (the author conclusion section) from the systematic review. our aim is to improve over previous approaches by including more complete, more detailed and higher quality information about each included study (as set out in section 4), along with detailed instructions based on textbook guidelines about how to conduct a systematic review, in the prompt to the llm. below we describe prompt composition (section 5.1), and the llms we test (section 5.2). we outline each of the above prompt components below. base prompt (part 1): part 1 of our base prompt is as follows: "you are a systematic reviewer tasked with synthesizing information from multiple clinical studies. below is the data you need to review. the title of the systematic review is: {systematic review title}". included study information: the included study information is made up of four components which are concatenated together: included study reference, included study title, included study abstract and included study pico (which includes risk of bias as described in section 4.5). ## evaluation methods as outlined in section 4.2, we use the authors' conclusions from the abstract of the target systematic review as our reference text to assess the performance of our system. the aim is to quantify the agreement between conclusions drawn in our 5 https://www.anthropic.com/news/ claude-3-family 6 training data contained information up to august 2023 for haiku and sonnet. training data contained information up to april 2024 for sonnet 3.5. generated biomedical syntheses with those in this reference text. below we describe metrics we used to evaluate the agreement between the two texts. llm judge. we use llm-as-judge as our primary approach to evaluation, as metrics based on it have been shown to have the highest correlation with human judgements in multiple studies @xcite @xcite . we use the most recent version of gpt (gpt-4o) 7 as our llm-as-judge. more specifically, the judge llm is provided with the reference text and the generated text, and is instructed to determine whether the generated text agrees or disagrees with the conclusions in the reference text. it is instructed to set out its reasoning first, and then give a score as a number between 1 and 4, with the following meaning: 1 (strongly disagree), 2 (disagree), 3 (agree), 4 (strongly agree). this scoring is similar to the scale used in @xcite . we report both the average llm judge score and agreement percentage which is the percentage of generated syntheses that are scored agree and strongly agree ( we tuned the prompt for our llm-as-judge metric to make sure it would give the appropriate response for a set of four synthetic examples that we designed to match the four different scores above. we iterated on the prompt design until the scores assigned by the llm matched what we expected to see for each of our synthetic inputs. the temperature for the llm-as-judge calls was set to 0, intended to ensure reproducibility. however, in our experiments we noted that rerunning with the same prompt and temperature 0 does not always produce the same response. therefore, we run each call three times for each generated summary and assign the majority score. if the model assigned three different scores, we instead assign a zero score indicating lack of agreement. this happened four times in 45 systematic reviews over the 15 experiments listed in the final prompt was as follows: "you are to judge the quality of the output of an automatically generated 'author's conclusion' section for a biomedical systematic review. the user will provide the gold standard reference text and the generated text. you will use the submit_analysis tool to provide your analysis. reference summary: {refer-ence} generated summary: {generated}" the model is required to submit its response in a structured json format using the function calling feature of the openai api. 8 the model must submit the reasoning for its answer first and then a score between 1-4 as described above. the description of the function given to the model is: "accepts analysis of generated text against reference text." the description of the reasoning parameter is "the reasoning of the reviewer about whether the generated text agrees or disagrees with the conclusions in the reference text." the description of the score parameter is "give the result as a number 1-4 meaning: 1: strongly disagree, 2: disagree, 3: agree, 4: strongly agree." we require the model to fill out the reasoning parameter first to avail of the benefits of chain-of-thought reasoning @xcite and also to aid in analysis of the model's ultimate decisions. other automatic metrics. following established practice, we also employ a range of stringsimilarity metrics to assess the quality of our generated texts, specifically: bleu @xcite , rouge-1, rouge-2, rouge-l @xcite , and chrf score @xcite . ## experimental results overall results. our 'kitchen sink' experiment which includes all of the components of our input described in section 5.1 (abstracts, included study pico, comparison pico, base prompt, guidelines and examples) achieved a 51% llm agreement percentage with the haiku model (row 3), 47% with sonnet (row 5), and 44% with sonnet 3.5 (row 7). however, in all cases, better results in terms of this metric are achieved when leaving out abstracts, as illustrated further in figure 3 . with haiku and sonnet 3.5, leaving out the abstract produced the overall best result of 53% agreement percentage (row 1, 2); for sonnet it was 49% (row 4). initially, we hypothesised that this could be due to the fact that our dataset only has 81% coverage of abstracts. we wanted to test whether the same effect would happen if there was 100% coverage of abstracts. we proceeded to conduct two more experiments to further research this surprising result. note that these experiments are not listed in regarding the strength of the model, in all cases, there is only a slight difference in performance between models. this is despite the fact the claude 3.5 sonnet is generally considered to be a far stronger model (section8). in terms of inclusion of guidelines and examples, comparison with @xcite . we observed notable improvements when using our highquality dataset compared to the dataset used by shaib et al. (2023), as illustrated in figure 5. the dataset in shaib et al. (2023) contains summaries of abstracts from the same 45 systematic reviews as contained in our datasets. these summaries were automatically generated using an llm. as noted above, the shaib et al. (2023) dataset has much lower coverage of included study abstracts than our own dataset. using our base prompt only, along with the abstract summaries from shaib et al.'s (2023) dataset, the system reaches a 7% agreement percentage. when we substitute the full abstracts from our dataset this rises to 9%. however, when we applied our full prompt (base prompt + guidelines + examples), the improvement was much more substantial, with the agreement percentage jumping from 13% to 31%. this demonstrates the combined effect of a high-quality dataset and a comprehensive prompt in enhancing the synthesis process. correlations between metrics. we found that there was little correlation between the agreement percentage scores produced by the llm, and the other automatic metrics that we used. for the full correlation matrix, see appendix j. dataset quality. our results show that our improved prompting strategy has a small impact when applied to prior datasets with much lower coverage of abstracts from included studies. however, when applied to our more comprehensive dataset, the same prompts are more effective. with our best prompt and the abstracts from our dataset, the agreement percentage is 31%, compared to 13% with the shaib et al. (2023) dataset. ## conclusion in the study reported in this paper, we leveraged llms to generate biomedical scientific syntheses by incorporating diverse types of crucial information from included studies as input. we evaluated our approach using a carefully constructed dataset that addresses limitations of existing datasets. our results show that we can improve over previous approaches and guide models to produce higherquality output by providing them with included study pico information, as well as crafting structured prompts incorporating instructions informed by domain knowledge gleaned from textbooks. it seems likely that further performance improvement can be achieved by further developing the prompt design. however, an important focus for future research will need to be the confident automatic extraction of relevant information from studies and papers for incorporation into such prompts.