Evidence Inference 2.0: More Data, Better Models (2020)

Abstract: How do we most effectively treat a disease or condition? Ideally, we could consult a database of evidence gleaned from clinical trials to answer such questions. Unfortunately, no such database exists; clinical trial results are instead disseminated primarily via lengthy natural language articles. Perusing all such articles would be prohibitively time-consuming for healthcare practitioners; they instead tend to depend on manually compiled systematic reviews of medical literature to inform care. NLP may speed this process up, and eventually facilitate immediate consult of published evidence. The Evidence Inference dataset was recently released to facilitate research toward this end. This task entails inferring the comparative performance of two treatments, with respect to a given outcome, from a particular article (describing a clinical trial) and identifying supporting evidence. For instance: Does this article report that chemotherapy performed better than surgery for five-year survival rates of operable cancers? In this paper, we collect additional annotations to expand the Evidence Inference dataset by 25%, provide stronger baseline models, systematically inspect the errors that these make, and probe dataset quality. We also release an abstract only (as opposed to full-texts) version of the task for rapid model prototyping. The updated corpus, documentation, and code for new baselines and evaluations are available at http://evidence-inference.ebm-nlp.com/ .

## introduction as reports of clinical trials continue to amass at rapid pace, staying on top of all current literature to inform evidence-based practice is next to impossible. as of 2010, about seventy clinical trial reports were published daily, on average @xcite . this has risen to over one hundred thirty trials per day. 1 motivated by the rapid growth in clinical trial publications, there now exist a plethora of tools to partially automate the systematic review task @xcite . however, efforts at fully integrating the pico framework into this process have been limited @xcite . what if we could build a database of participants, 2 interventions, comparisons, and outcomes studied in these trials, and the findings reported concerning these? if done accurately, this would provide direct access to which treatments the evidence supports. in the near-term, such technologies may mitigate the tedious work necessary for manual synthesis. recent efforts in this direction include the ebm-nlp project @xcite , and evidence inference @xcite , both of which comprise annotations collected on reports of randomized control trials (rcts) from pubmed. 3 here we build upon the latter, which tasks systems with inferring findings in full-text reports of rcts with respect to particular interventions and outcomes, and extracting evidence snippets supporting these. we expand the evidence inference dataset and evaluate transformer-based models @xcite on the task. concretely, our contributions are: ## annotation in the evidence inference task @xcite , a model is provided with a full-text article describing a randomized controlled trial (rct) and a 'prompt' that specifies an intervention (e.g., aspirin), a comparator (e.g., placebo), and an outcome (e.g., duration of headache). we refer to these as ico prompts. the task then is to infer whether a given article reports that the intervention resulted in a significant increase, significant decrease, or produced no significant difference in the outcome, as compared to the comparator. our annotation process largely follows that outlined in @xcite ; we summarize this briefly here. data collection comprises three steps: combining this new data with the dataset introduced in @xcite yields in total 12,616 unique prompts stemming from 3,346 unique articles, increasing the original dataset by 25%. 5 to acquire the new annotations, we hired 11 doctors: 1 for prompt generation, 6 for prompt annotation, and 4 for verification. ## models we consider a simple bert-based @xcite pipeline comprising two independent models, as depicted in figure 1 . the first identifies evidence bearing sentences within an article for a given ico. the second model then classifies the reported findings for an ico prompt using the evidence extracted by this first model. these models place a dense layer on top of representations yielded from @xcite ), foot_6 a variant of roberta @xcite pre-trained over scientific corpora, foot_7 followed by a softmax. specifically, we first perform sentence segmentation over full-text articles using scispacy (neumann et al., 2019). we use this segmentation to recover evidence bearing sentences. we train an evidence identifier by learning to discriminate between evidence bearing sentences and randomly sampled non-evidence sentences. 9 we then train an evidence classifier over the evidence bearing sentences to characterize the trial's finding as reporting that the intervention significantly decreased, did not significantly change, or significantly increased the outcome compared to the comparator in an ico. when making a prediction for an (ico, document) pair we use the highest scoring evidence sentence from the identifier, feeding this to the evidence classifier for a final result. note that the evidence classifier is conditioned on the ico frame; we prepend the ico embedding (from biomed roberta) to the embedding of the identified evidence snippet. reassuringly, removing this signal degrades performance ( for all models we fine-tuned the underlying bert parameters. we trained all models using the adam optimizer (kingma and ba, 2014) with a bert learning rate 2e-5. we train these models for 10 epochs, keeping the best performing version on a nested held-out set with respect to macro-averaged f-scores. when training the evidence identifier, we experiment with different numbers of random samples per positive instance. we used scikit-learn @xcite for evaluation and diagnostics, and implemented all models in pytorch @xcite . we additionally reproduce the end-to-end system from @xcite : a gated recurrent unit @xcite to encode the document, attention @xcite conditioned on the ico, with the resultant vector (plus the ico) fed into an mlp for a final significance decision. ## experiments and results our main results are reported in impact of negative sampling as negative sampling is a crucial part of the pipeline, we vary the number of samples and evaluate performance. we provide detailed results in appendix a, but to summarize briefly: we find that two to four negative samples (per positive) performs the best for the end-to-end task, with little change in both auroc and accuracy of the best fit evidence sentence. this is likely because the model needs only to maximize discriminative capability, rather than calibration. distribution shift in addition to comparable krippendorf-α values computed above, we measure the impact of the new data on pipeline performance. we compare performance of the pipeline with all data "biomed roberta (br) pipeline" vs. just the old data "biomed roberta (br) bert pipeline 1.0" in ## conclusions and future work we have introduced an expanded version of the evidence inference dataset. we have proposed and evaluated bert-based models for the evidence inference task (which entails identifying snippets of evidence for particular ico prompts in long documents and then classifying the reported finding on the basis of these), achieving state of the art results on this task.