Automatically Summarizing Evidence from Clinical Trials: A Prototype Highlighting Current Challenges (2023)

Abstract: In this work we present TrialsSummarizer, a system that aims to automatically summarize evidence presented in the set of randomized controlled trials most relevant to a given query. Building on prior work, the system retrieves trial publications matching a query specifying a combination of condition, intervention(s), and outcome(s), and ranks these according to sample size and estimated study quality. The top-k such studies are passed through a neural multi-document summarization system, yielding a synopsis of these trials. We consider two architectures: A standard sequence-to-sequence model based on BART, and a multi-headed architecture intended to provide greater transparency and controllability to end-users. Both models produce fluent and relevant summaries of evidence retrieved for queries, but their tendency to introduce unsupported statements render them inappropriate for use in this domain at present. The proposed architecture may help users verify outputs allowing users to trace generated tokens back to inputs. The demonstration video can be found at https://vimeo.com/735605060The prototype, source code, and model weights are available at: https://sanjanaramprasad.github.io/trials-summarizer/

## introduction patient treatment decisions would ideally be informed by all available relevant evidence. however, realizing this aim of evidence-based care has become increasingly difficult as the medical literature (already vast) has continued to rapidly expand @xcite . well over 100 new rct reports are now published every day @xcite . language technologies -specifically automatic summarization methods -have the potential to provide concise overviews of all evidence relevant to a given clinical question, providing a kind of systematic review on demand @xcite @xcite . we describe a demonstration system, trialssummarizer, which combines retrieval over clinical trials literature with a summarization model to provide narrative overviews of current published evidence relevant to clinical questions. figure 1 shows an illustrative query run in our system and the resultant output. a system capable of producing accurate summaries of the medical evidence on any given topic could dramatically improve the ability of caregivers to consult the whole of the evidence base to inform care. however, current neural summarization systems are prone to inserting inaccuracies into outputs @xcite @xcite @xcite . this has been shown specifically to be a problem in the context of medical literature summarization @xcite , where there is a heightened need for factual accuracy. a system that produces plausible but often misleading summaries of comparative treatment efficacy is useless without an efficient means for users to assess the validity of outputs. motivated by this need for transparency when summarizing clinical trials, we implement a summarization architecture and interface designed to permit interactions that might instill trust in outputs. specifically, the model associates each token in a generated summary with a particular source "aspect" extracted from inputs. this in turn allows one to trace output text back to (snippets of) inputs, permitting a form of verification. the architecture also provides functionality to "in-fill" pre-defined template summaries, providing a compromise between the control afforded by templates and the flexibility of abstractive summarization. we realize this functionality in our system demonstration. ## related work the (lack of) factuality of neural summarization systems is an active area of research (chen et al., figure 1 : an example query (regarding use of statins to reduce risk of stroke) and output summary provided by the system. in this example, the summary accurately reflects the evidence, but this is not always the case. 2021; @xcite @xcite @xcite @xcite . this demo paper considers this issue in the context of a specific domain and application. we also explored controllability to permit interaction, in part via templates. this follows prior work on hybrid template/neural summarization @xcite @xcite . we also note that this work draws upon prior work on visualizing summarization system outputs @xcite @xcite and biomedical literature summarization @xcite @xcite @xcite . however, to our knowledge this is the first working prototype to attempt to generate (draft) evidence reviews that are both interpretable and editable on demand. ## system overview our interface is built on top of trialstreamer @xcite ), an automated system that identifies new reports of randomized controlled trials (rcts) in humans and then extracts and stores salient information from these in a database of all published trial information. our system works by identifying rct reports relevant to a given query using a straightforward retrieval technique (section 3.1), and then passing the top-k of these through a multi-document summarization model (section 3.2). for the latter component we consider both a standard sequence-to-sequence approach and a aspect structured architecture (section 3.3) intended to provide greater transparency. ## dataset and training details we aim to consume collections of titles and abstracts that describe rcts addressing the same clinical question to abstractive summaries that synthesize the evidence presented in these. we train all models on an rct summarization dataset @xcite where we extract clinically salient elements -i.e., our aspects -from each of the (unstructured) inputs as a pre-processing step using existing models @xcite . training we use the huggingface transformers library @xcite to implement both models. we initalize both models to bart-base @xcite . we fine-tune the models with a batch size of 2 for 3 epochs, using the adam optimizer (kingma and ba, 2014) with a learning rate of 3e-5. ## case study: verification and controllability to demonstrate the potential usefulness of the interface (and the architecture which enables it), we walk through two case studies. we highlight the type of interpretability for verification our proposed approach provides, also demonstrate the ability to perform controllable summarization to show how this might be useful. the queries used in these case studies along with the investigation were performed by a co-author ijm, a medical doctor with substantial experience in evidence-based medicine. we also compare the models and report automatic scores for rouge and factuality in the appendix section a and find that the two models perform comparably. ## conclusions we have described trialssummarizer, a prototype system for automatically summarizing rcts relevant to a given query. neural summarization models produce summaries that are readable and (mostly) relevant, but their tendency to introduce unsupported or incorrect information into outputs means they are not yet ready for use in this domain. we implement a multi-headed architecture intended to provide greater transparency. we provided qualitative examples intended to highlight its potential to permit faster verification and controllable generation. future work is needed to test the utility of this functionality in a user trial, and to inform new architectures that would further increase the accuracy and transparency of models for summarizing biomedical evidence.