Predicting Clinical Trial Results by Implicit Evidence Integration (2020)

Abstract: Clinical trials provide essential guidance for practicing Evidence-Based Medicine, though often accompanying with unendurable costs and risks. To optimize the design of clinical trials, we introduce a novel Clinical Trial Result Prediction (CTRP) task. In the CTRP framework, a model takes a PICO-formatted clinical trial proposal with its background as input and predicts the result, i.e. how the Intervention group compares with the Comparison group in terms of the measured Outcome in the studied Population. While structured clinical evidence is prohibitively expensive for manual collection, we exploit large-scale unstructured sentences from medical literature that implicitly contain PICOs and results as evidence. Specifically, we pre-train a model to predict the disentangled results from such implicit evidence and fine-tune the model with limited data on the downstream datasets. Experiments on the benchmark Evidence Integration dataset show that the proposed model outperforms the baselines by large margins, e.g., with a 10.7% relative gain over BioBERT in macro-F1. Moreover, the performance improvement is also validated on another dataset composed of clinical trials related to COVID-19.

## introduction shall covid-19 patients be treated with hydroxychloroquine? in the era of evidence-based medicine (ebm, sackett 1997), medical practice should be guided by well-designed and wellconducted clinical research, such as randomized controlled trials. however, conducting clinical trials is expensive and time-consuming. furthermore, inappropriately designed studies can be devastating in a pandemic: a high-profile remdesivir clinical trial fails to achieve statistically significant conclusions @xcite , partially because it does not attain the predetermined sample size when "competing with" other inappropriately designed trials that are unlikely to succeed or not so urgent to test (e.g.: physical exercises and dietary treatments). therefore, it is crucial to carefully design and evaluate clinical trials before conducting them. proposing new clinical trials requires support from previous evidence in medical literature or practice. for example, the world health organization (who) has launched a global megatrial, solidarity (who, 2020), to prioritize clinical resources by recommending only four most promising therapies foot_0 . the rationale for this suggestion comes from the integration of evidence that they might be effective against coronaviruses or other related organisms in laboratory or clinical studies @xcite @xcite . however, manual integration of evidence is far from satisfying, as one study reports that about 86.2% of clinical trials fail @xcite and even some of the solidarity therapies do not get expected results @xcite . to assist clinical trial designing, we introduce a novel task: clinical trial result prediction (ctrp), i.e. predicting the results of clinical trials without actually doing them ( §3). figure 1 shows the architecture of the ctrp task. we define the input to be a clinical trial proposal foot_1 , which contains free-texts of a population (e.g.: "covid-19 patients with severe symptoms"), an intervention (e.g.: "active remdesivir (i.v.)"), a comparator (e.g.: "placebos matched remdesivir") and an outcome (e.g.:"time to clinical improvement"), i.e. a pico-formatted query @xcite , and the background of the proposed trial. the output is the trial result, denoting how (higher, lower, or no difference) i compares to c in terms of o for p. one particular challenge of this task is that evidence is entangled with other free-texts in the literature. prior works have explored explicit methods for evidence integration through a pipeline of retrieval, extraction and inference on structured {p,i,c,o,r} evidence @xcite @xcite @xcite @xcite @xcite . however, they are limited in scale since getting domain-specific supervision for all clinical evidence is prohibitively expensive. in this work, we propose to implicitly learn from such evidence by pre-training, instead of relying on explicit evidence with purely supervised learning. there are more than 30 million articles in pubmed 3 , which stores almost all available medical evidence and thus is an ideal source for learning. we collect 12 million sentences from pubmed abstracts and pubmed central 4 (pmc) articles with comparative semantics, which is commonly used to express clinical evidence ( §4.1). p, i, c, o, and r are entangled with other free-texts in such sentences, which we denote as implicit evidence. unlike previous efforts that seek to disentangle all of pico and r, we only disentangle r out of the implicit evidence using simple heuristics ( §4.2). for better learning the ordering function of i/c conditioned on p and o, we also use adversarial examples generated by reversing both the entangled pico and the r in the pre-training ( §4.3). then, we pre-train a transformer encoder @xcite to predict the disentangled r from the implicit evidence, which still contains pico ( §5.1). the model is named ebm-net to reflect its utility for evidence-based 3 https://pubmed.ncbi.nlm.nih.gov/ 4 https://www.ncbi.nlm.nih.gov/pmc/ medicine. finally, we fine-tune the pre-trained ebm-net on downstream datasets of the ctrp task ( §5.2), which are typically small in scale ( §6). to evaluate model performance, we introduce a benchmark dataset, evidence integration ( §6.1), by re-purposing the evidence inference dataset @xcite . experiments show that our pre-trained ebm-net outperforms the baselines ( §6.2) by large margins ( §6.3). clustering analyses indicate that ebm-net can effectively learn quantitative comparison results ( §6.4). in addition, the ebm-net model is further validated on a dataset composed of covid-19 related clinical trials ( §6.5). our contribution is two-fold. first, we propose a novel and meaningful task, ctrp, to predict clinical trial results before conducting them. second, unlike previous efforts that depend on structured data to understand the totality of clinical evidence, we heuristically collect unstructured textual data, i.e. implicit evidence, and utilize large-scale pretraining to tackle the proposed ctrp task. the datasets and codes are publicly available at https: //github.com/alibaba-nlp/ebm-net . explicit evidence integration: it depends on the existence of structured evidence, i.e.: {p, i, c, o, r} @xcite . consequently, collecting such explicit evidence is vital for further analyses, and is also the objective for most relevant works: some seek to find relevant papers through retrieval @xcite ; many works are aimed at extracting pico elements from published literature @xcite @xcite @xcite ; the evidence inference task extracts r for a given ico query using the corresponding clinical trial report @xcite . however, since getting expert annotations is expensive, these works are typically limited in scale, with only thousands of labeled instances. few works have been done to utilize the automatically collected structured data for analyses. in this paper, we adopt an end-to-end approach, where we use large-scale pre-training to implicitly learn from free-text clinical evidence. ## the ctrp task the ctrp task is motivated to evaluate clinical trial proposals by predicting their results before actually conducting them, as discussed in §1. therefore, we formulate the task to take as input exactly the information required for proposing a new clinical trial: free-texts of a background description and a pico query to be investigated. formally, we denote the strings of the input background as b and pico elements as p, i, c, and o, respectively. the task output is defined as one of the three possible comparison results: higher (↑), no difference (→), or lower (↓) measurement o in intervention group i than in comparison group c for population p. we denote the result as r, and: ## implicit evidence integration in this section, we introduce the implicit evidence integration, which is used to collect pre-training data for comparative language modeling ( §5.1). instead of collecting explicit evidence with structured {b, p, i, c, o -r} information, we utilize a simple observation to collect evidence implicitly: clinical evidence is naturally expressed by comparisons, e.g.: "blood oxygen is higher in the intervention group than in the placebo group". free-texts of p, i, c, o and r are entangled with other functional words that connect these elements in such comparative sentences, where r is a free-text version of the structured result r (e.g.: @xmath0... than" translates into @xmath1. we call these sentences entangled implicit evidence and denote them as e @xmath2. then, we disentangle r out of the e ent by heuristics, getting r and the left e @xmath3. we also include adversarial instances generated from the original ones. several examples are shown in details of implicit evidence collection, disentanglement, and adversarial data generation are introduced in §4.1, §4.2 and §4.3, respectively. ## ebm-net we introduce the ebm-net model in this section. similar to bert @xcite , ebm-net is essentially a transformer encoder @xcite , and follows the pre-training -fine-tuning approach: we pre-train ebm-net by comparative language modeling (clm, §5.1) that is designed to learn the conditional ordering function of i/c. the pre-trained ebm-net is fine-tuned to solve the ctrp task on downstream datasets ( §5.2). ## experiments 6.1 the evidence integration dataset the evidence integration dataset serves as a benchmark for our task. we collect this dataset by repurposing the evidence inference dataset @xcite , which is essentially a machine reading comprehension task for extracting the structured result (i.e.: r) of a given structured ico query foot_3 from the corresponding clinical trial report article. since clinical trial reports already contain free-text result descriptions (i.e.: r) of the given ico, solving the original task does not require the integration of previous clinical evidence. to test such capability for our proposed ctrp task, we remove the result/conclusion part and only keep the background/method part in the input clinical trial report. 34.6% tokens of the original abstracts are removed on average and the remained are used as the clinical trial backgrounds. specifically, input of the evidence integration dataset includes free texts of ico elements i, c and o which are the same as the original evidence inference dataset, and their clinical trial backgrounds b. the output is the comparison result r. following the original dataset split, there are 8,164 instances for training, 1,002 for validation, and 965 for test. we also do experiments under the adversarial setting, where adversarial examples generated by reversing both the i/c order and the r label (similar to §5.1) are added. this setting is used to test model robustness under adversarial attack. ## conclusions in this paper, we introduce a novel task, ctrp, to predict clinical trial results without actually doing them. instead of using structured evidence that is prohibitively expensive to annotate, we heuristically collect 12m unstructured sentences as implicit evidence, and use large-scale clm pretraining to learn the conditional ordering function required for solving the ctrp task. our ebm-net model outperforms other strong baselines on the evidence integration dataset and is also validated on covid-19 clinical trials. tengteng zhang, yiqin yu, jing mei, zefang tang, xiang zhang, and shaochun li. 2020. unlocking the power of deep pico extraction: step-wise medical ner identification. arxiv preprint arxiv:2005.06601.