T rialstreamer: Mapping and Browsing Medical Evidence in Real-Time (2020)

Abstract: We introduce Trialstreamer, a living database of clinical trial reports. Here we mainly describe the evidence extraction component; this extracts from biomedical abstracts key pieces of information that clinicians need when appraising the literature, and also the relations between these. Specifically, the system extracts descriptions of trial participants, the treatments compared in each arm (the interventions), and which outcomes were measured. The system then attempts to infer which interventions were reported to work best by determining their relationship with identified trial outcome measures. In addition to summarizing individual trials, these extracted data elements allow automatic synthesis of results across many trials on the same topic. We apply the system at scale to all reports of randomized controlled trials indexed in MEDLINE, powering the automatic generation of evidence maps, which provide a global view of the efficacy of different interventions combining data from all relevant clinical trials on a topic. We make all code and models freely available alongside a demonstration of the web interface.

## introduction and motivation the highest-quality evidence to inform healthcare practice comes from randomized controlled trials (rcts). the results of the vast majority of these trials are communicated in the form of unstructured text in journal articles. such results accumulate quickly, with over 100 articles describing rcts published daily, on average. it is difficult for healthcare providers and patients to make sense of and keep up with this torrent of unstructured literature. consider a patient who has been newly diagnosed with diabetes. she would like to con-1 https://github.com/bepnye/evidence extraction/ 2 http://bit.ly/trialstreamer figure 1 : a portion of an example evidence mapping interventions and their inferred efficacy for outcomes, given the condition (or population) of type ii diabetes. these maps are generated automatically using the nlp system we describe in this work. trialstreamer is a first attempt to solve this problem, making evidence more browseable via nlp technologies. figure 1 shows one of the key features of the system: an automatically generated evidence map that displays treatments (vertical axis) and outcomes (horizontal) identified for a condition specified by the user (here, migraines). we elaborate on this particular example to illustrate the use of the system in section 3. trialstreamer aims to facilitate efficient evidence mapping with a user friendly method of presenting a search across a broad field (here, being a clinical condition) @xcite . we use nlp technologies to provide browseable, interactive overviews of large volumes of literature, on-demand. these may then inform subsequent, formal syntheses, or they may simply guide ex-ploration of the primary literature. in this work we describe an open-source prototype that enables evidence mapping, using nlp to generate interactive overviews and visualizations of all rct reports indexed by medline (and accessible via pubmed). when mapping the evidence one is generally interested in the following basic questions: in the remainder of this paper we describe a prototype system that facilitates interactive exploration and mapping of the evidence base, with an emphasis on answering the above questions. the trialstreamer mapping interface allows structured search over study populations, interventions/comparators, and outcomes -collectively referred to as pico elements @xcite . it then displays key clinical attributes automatically extracted from the set of retrieved trials. this is made possible via nlp modules trained on recently released corpora @xcite , described below. ## system overview the evidence extraction pipeline is composed of four primary phases. first, text snippets that convey information about the trial's treatments (or interventions), outcome measures, and results are extracted from abstracts. relations between these snippets are then inferred to identify which treatments were compared against each other, and which outcomes were measured for these comparisons. the extracted relations and evidence statements are then used to infer an overall conclusion about the comparative efficacy of the trial's interventions. finally, the clinical concepts expressed in the extracted spans are normalized to a structured vocabulary in order to ground them in an existing knowledge base and allow for aggregations across trials. a typical rct report would pertain to a single clinical condition (the population), but might report multiple numerical results, each concerning a particular intervention, comparator, and outcome measure (which we describe as an ico triplet). because the end-to-end task combines nlp subtasks that are supported by different datasets, we collected new development and test sets -160 abstracts in all, exhaustively annotated -in order to evaluate the overall performance of our system. two medical doctors foot_0 annotated these documents with the all of the expressed entities, their mentions in the text, the relations between them, the conclusions reported for each ico triplet and the sentence that contains the supporting evidence for this @xcite . we were unable to obtain normalized concept labels for the ico triplets due to the excessive difficulty of the task for the annotators. modeling decisions were informed by the 60 document development set, and we present evaluations of the first four information extraction modules with regard to the 100 documents in the unseen test set. ## illustrative example to illustrate the envisioned use of our automatic mapping system, we return to the example we began with at the outset of this paper: seeking evidence concerning treatment of type ii diabetes. to begin, the user specifies a condition (population) of interest. we rely on medical subject headings (mesh) terms, foot_1 which as discussed above is a structured vocabularly maintained by the nlm. we allow users to enter a search string and provide auto-complete options from the mesh vocabulary. users can additionally provide interventions or outcomes of interest to further narrow the search. we show an example of a constructed set of filters in figure 3 . once a set of search terms is specified, relevant rcts are retrieved from the comprehensive and up-to-date database. 6 the interface then displays counts of unique interventions and outcomes covered by the retrieved trials. each bar in these plots can be clicked to explicitly include that concept in the search terms, allowing for a data-driven approach to building up the search parameters via iterative refinement. at this point, the evidence map shown in figure 1 is also displayed, providing a summary of the evidence available for the effectiveness of the selected interventions with respect to their cooccurring outcomes. the user can mouse-over plot elements to view tooltips that include snippets of contributing evidence from the underlying abstracts, or click through to browse these texts annotated with all of the extracted information, as shown in figure 4 . to evaluate the system's utility for a real-world task, we provided the tool to a team of researchers at cures within reach for cancer (cwr4c). 7 domain experts reviewed the extracted ico conclusions and automatically generated plots for a randomly selected subset of documents pertaining to cancer trials, a domain that is particularly challenging given the prevalence of complex compound interventions that often share individual components between trial arms. the reviewers were asked to evaluate the types of mistakes made by the system as well as the overall precision and recall of the extracted conclusions for each document. across 21 documents average precision was 54% and average recall was 75%, and the team expressed excitement about the efficacy of the system for their purposes. cwr4c has continued to work with this tool as a source of information about cancer-related clinical trials. ## conclusions we have presented the evidence extraction component of trialstreamer, an open-source prototype that performs end-to-end identification of published rct reports, extracts key elements from the texts (intervention and outcomes descriptions), and performs relation extraction between these, i.e., attempts to determine which intervention was reported to work for which outcomes. we use this pipeline to provide fast, on-demand overviews of all published evidence pertaining to a condition of interest. moving forward, we hope to refine the linking of extracted snippets to structured vocabularies to run a more comprehensive user-study to evaluate the use of the system in practice by different types of users. we also hope to develop a joint extraction and inference model, rather than relying on the current pipelined approach.