Predicting Intervention Approval in Clinical Trials through Multi-Document Summarization (2022)

Abstract: Clinical trials offer a fundamental opportunity to discover new treatments and advance the medical knowledge. However, the uncertainty of the outcome of a trial can lead to unforeseen costs and setbacks. In this study, we propose a new method to predict the effectiveness of an intervention in a clinical trial. Our method relies on generating an informative summary from multiple documents available in the literature about the intervention under study. Specifically, our method first gathers all the abstracts of PubMed articles related to the intervention. Then, an evidence sentence, which conveys information about the effectiveness of the intervention, is extracted automatically from each abstract. Based on the set of evidence sentences extracted from the abstracts, a short summary about the intervention is constructed. Finally, the produced summaries are used to train a BERT-based classifier, in order to infer the effectiveness of an intervention. To evaluate our proposed method, we introduce a new dataset which is a collection of clinical trials together with their associated PubMed articles. Our experiments, demonstrate the effectiveness of producing short informative summaries and using them to predict the effectiveness of an intervention.

## introduction clinical trials (ct) present the basic evidencebased clinical research tool for assessing the effectiveness of health interventions. nevertheless, only a small number of interventions make it successfully through the process of clinical testing. approximately, 39%-64% of interventions actually advance to the next step of each phase of clinical trials @xcite . the uncertainty of a ct outcome could lead to increased costs, prolonged drug development and ineffective treatment for the participants. at the same time, the volume of published scientific literature is rapidly growing and offers the opportunity to explore a valuable knowledge. therefore, there is a need to develop new tools which can i) integrate such information and ii) enhance the process of intervention approval in ct. predicting the approval of an intervention, a task that describes the ability of a system to predict whether an intervention will reach the final stage of clinical testing, is a topic that has been studied before @xcite . the majority of these studies use various traditional machine learning methods and rely on structured data from various sources, including biomedical, chemical or drug databases @xcite . however, only a few studies take into account the textual information that is available online, and mostly in a supplementary manner @xcite . in fact, employing natural language processing (nlp) techniques to address the outcome prediction task has been hardly explored. recognising this lack of related studies, the work presented here addresses the task of predicting intervention approval with the use of nlp. particularly, we relied on generating concise and informative summaries from multiple texts that are relevant to the intervention under evaluation. in a sense, we built an intervention-specific narrative which combines key information from multiple inter-connected documents. the benefit of using multiple articles to generate summaries is that they can cover the inherently multi-faceted nature of an intervention's clinical background. more precisely, given an intervention, our system retrieves all pubmed abstracts that are relevant to the intervention and refer to a clinical study. it then extracts the evidence sentences from each abstract using a bert-based evidence sentence classifier, in a similar fashion to @xcite . this set of evidence sentences, which captures the consolidated narrative about the intervention, can grow gradually, as new articles become available. thus, further analysis is necessary in order to select the most important information. using the set of evidence sentences for each intervention, we generate short summaries by leveraging the power of language models (bert or bart). the resulted summaries are then fed to a bert-based binary sequence classifier which makes a prediction about the likely approval or not of the intervention. overall, the main contributions of the paper are the following: intervention success prediction the prediction of intervention approval belongs to a broader category of medical prediction tasks. relevant work includes clinical trial outcome prediction @xcite @xcite , drug approval @xcite @xcite , clinical trial termination @xcite @xcite , predicting phase transition @xcite . all these studies rely either on specific types of structured data or on combining structured data with limited unstructured data. differently from this line of work, the authors of @xcite proposed an approach that employs nlp to infer the relation between an intervention and the outcome of a specific clinical trial. their method is based on extracting evidence sentences from unstructured text. an extension of this work suggests the use of bert-based language models for the same task @xcite . another closely related study @xcite , performs a large-scale pre-training on unstructured text data to infer the outcome of a clinical trial. our approach builds upon this related work, aiming to incorporate information from multiple articles. this extension is motivated by the assumption that the inter-connected clinical knowledge, coming from multiple sources can provide a more holistic picture of the intervention, facilitating more precise analysis and accurate prediction. although all these prior efforts tackle, more or less, the problem of intervention approval, none of them attempted to predict the effectiveness of an intervention using summarization methods. ## task overview according to the u.s. food and drug administration (fda), a ct addresses one of five phases of clinical assessment: early phase 1 (former phase 0), phase 1, phase 2, phase 3 and phase 4. each phase is defined by the study's objective, the interventions under evaluation, the number of participants, and other characteristics foot_0 . notably, phase 4 clinical trials take place after fda has approved a drug for marketing. therefore, we can assume that a ct in phase 4 assesses effective intervention. on this basis, our task is to predict whether an intervention will advance to the final stage of clinical testing (phase 4), as shown in figure 1 . we model the task of predicting the success or failure of an intervention as a binary classification task. all data relevant to phase 4 are omitted from the training stage. ## data in this work, we introduce a new dataset foot_1 for the task of predicting intervention approval. the dataset is a collection of structured and unstructured data in english derived from clinicaltrials.gov and pubmed during may-june 2021. as a first step in the construction of the dataset, we retrieve all available ct studies from clinicaltrials.gov that satisfy some criteria. then, we associate each ct with pubmed articles based on the ct study identifier. following some cleaning process (i.e. deduplication and entity resolution) we generate the final dataset. ## methodology in figure 2 , we illustrate the proposed approach, which consists of three main steps. initially, we use the abstracts of the intervention's clinical trial record to extract evidence sentences. these sentences are then used to generate a short summary that contains information about the efficacy of the intervention. the summary is then processed by a bert-based sequence classifier to make the final decision about the intervention. each of the three steps is detailed in the following subsections. ## results and analysis we followed different training approaches for the different trainable components of our pipeline. for the evidence sentence selection and the abstractive summarization models we split the data into development and test and then split the development set further into training (90%) and validation (10%). we kept the model that performed best on the validation set and evaluated it on the held-out test set of each task respectively, averaged over three random data splits. considering the small size of the interventions dataset, we applied a 10-fold cross validation for the final classification task. for this task, we report macro averages of the evaluation metrics over the ten folds. ## conclusion predicting intervention approval in clinical trials is a major challenge with significant impact in healthcare. in this paper, we have proposed a new pipeline to address this problem, based on stateof-the-art nlp techniques. the proposed method consists of three steps. first, it identifies evidence sentences from multiple abstracts related to an intervention. then, these sentences are used to produce short summaries. finally, a classifier is trained on the generated summaries in order to predict the approval or not of an intervention. moreover, we introduced a new dataset for this task which contains 704 interventions associated with 15,800 abstracts. this data was used to evaluate our pipeline against other baseline models. the experimental results verified the effectiveness of our approach in predicting the approval of an intervention and the contribution of each step of the proposed pipeline to the final result. further evaluation on predicting phase transitions, showed that our model can assist in all stages of a clinical trial. besides, the generated multi-document summaries can be naturally used to explain the predictions of the model. there are multiple ways to extend this work. in terms of multi-document summarization, there is room to explore more advanced summarization models, quality and performance metrics as well as better explainability assessment. in the bigger picture, we shall also consider to expand the dataset by extending its size and incorporating different types of resources (e.g. drug interaction networks). finally, we are interested in enhancing the proposed method to incorporate temporal information associated with the cts to maintain the history of clinical changes. yuhao zhang, derek merck, emily bao tsai, christopher d manning, and curtis p langlotz. 2019. optimizing the factual correctness of a summary: a study of summarizing radiology reports. arxiv preprint arxiv:1911.02541.