NLI 4 CT : Multi-Evidence Natural Language Inference for Clinical Trial Reports (2023)

Abstract: How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets fail to capture the full complexity of inference over CTRs. In this work, we present a novel resource to advance research on NLI for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Secondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. Baselines on this corpus expose the limitations of existing NLI approaches, with 6 state-of-the-art NLI models achieving a maximum F1 score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, competition leaderboard, and website, available on CodaLab, and code to replicate the baseline experiments on GitHub.

## introduction clinical trials are research studies performed to test the efficacy and safety of novel treatments, and they are indispensable for the progression of experimental medicine @xcite . ctrs are documents that detail the methodology and results 1 email: mael.jullien@manchester.ac.uk of a particular trial. clinical practitioners use the information in these ctrs to design personalized and targeted interventions, matching therapeutic agents to each patient's tumor biomarkers. however, there are over 400,000 ctrs available, being continually published at an ever-increasing rate @xcite . consequently, it has become unfeasible to manually carry out comprehensive evaluations of all the relevant literature when designing treatment protocols @xcite . natural language inference (nli) offers a potential solution for the large-scale interpretation and retrieval of medical evidence, connecting the latest evidence to support personalized care @xcite . in this paper, we propose two different tasks, on breast cancer ctrs. firstly, to determine the inference relation between a natural language statement and a ctr, as shown in figure 1 . secondly, to retrieve supporting facts from the ctr(s) to justify the predicted relation. with respect to healthcare, these inference tasks are a move towards automatic claim verification and evidence extraction, over biomedical corpora. these proposed tasks represent several challenges for existing nli systems. the inference task requires substantial amounts of quantitative reasoning and numerical operations, as seen in figure 1 , on which many current state-of-the-art nli models perform poorly @xcite . additionally, much of the inference for these tasks revolves around domain-specific terminology (see figure 1 ). many state-of-the-art nli models fail to effectively surmount the word distribution shift from general domain corpora to biomedical corpora through transfer learning @xcite . this phenomenon is well-represented in the proposed dataset. this task differentiates itself from other biomedical nli tasks by considering the full scope of ctrs, not limited to results and interventions, instead including eligibility and adverse events (ae) as well @xcite . the proposed task also provides a significant variance in the types of inference required to solve each instance, with minimal, if any, repetition in inference chains. the contributions of this paper are as follows: to the best of our knowledge, we are the first to design a task that covers the full scope of ctrs, combining together the complexity of biomedical and numerical nli. solving these tasks would be a significant advancement toward efficient retrieval, synthesis, and inference of published literature to support evidence-based experimental medicine. ## related work there are a multitude of expert-annotated resources for clinical nlp, examples are shown in neural models continue to perform poorly on quantitative reasoning and numerical operations within nli @xcite . prior works such as @xcite , @xcite , and @xcite experiment with biomedical pre-training strategies. @xcite presents exact, an automatic information extraction system for clinical trial variables such as drug dosage, sample size, and primary outcomes from ctrs. while there are many proficient evidence retrieval models, there are no existing systems that are able to effectively carry out biomedical and numerical nli simultaneously. in a related vein, @xcite introduced an automated approach to cluster clinical trials based on similar eligibility features. using data from clinicaltrials.gov, they extracted semantic features from the eligibility criteria of trials and employed the jaccard similarity coefficient for clustering. their method, which generated 8,806 centerbased clusters, offers potential utility in understanding knowledge reuse patterns in clinical trial designs and enhancing recruitment strategies. @xcite presented the clinical trial knowledge base (ctkb), a comprehensive database that houses discrete clinical trial eligibility criteria. developed using advanced nlp techniques, ctkb transforms free-text criteria from clinicaltrials.gov as the biomedical domain continues to generate vast amounts of textual data, the importance of efficient concept recognition grows. @xcite introduced the mantra gsc, a multilingual gold-standard corpus designed for biomedical concept recognition across five languages: english, french, german, spanish, and dutch. this corpus, based on parallel biomedical texts from sources like medline and drug labels, provides annotations grounded in a subset of the unified medical language system. such endeavors underscore the global push towards more multilingual biomedical information extraction tools, catering to the diverse linguistic landscape of the biomedical community. in the pursuit of enhancing access to evidencebased medicine, campillos-llanos et al. (2021) introduced the clinical trials for evidence-based medicine in spanish (ct-ebm-sp) corpus. this corpus, comprising 1200 texts about clinical trials, is annotated with entities from the unified medical language system, covering areas such as anatomy, pharmacological substances, pathologies, and diagnostic or therapeutic procedures. their work emphasizes the potential of natural language processing in bridging the gap between vast medical literature and the need for healthcare professionals to access relevant, evidence-based information efficiently. ## clinical trial reports clinical trials are research studies on human volunteers used to evaluate an intervention @xcite ). an intervention may be medical, surgical, or a behavioral protocol. these interventions are often compared with a standard treatment, placebo, or control protocol. the investigators use various outcome measurements from the volunteers to ascertain the effectiveness of the intervention, as well as the likelihood and severity of aes. for nli4ct we retrieved 1000 publicly available breast cancer ctrs, with published results from clinicaltrials.gov, on the 22nd of december 2021. this data is developed by the u.s. national library of medicine and covered by the hipaa privacy rule, protecting personally identifiable information (additional ethical considerations can be found in section 11). we separate the ctrs into 4 sections: eligibility criteria: a set of conditions for patients to be allowed to take part in the clinical trial. intervention: the type, dosage, frequency, and duration of treatments being studied. results: number of participants in the trial, outcome measures, units, and the results. adverse events: these are signs and symptoms eligibility criteria inclusion: • her2 + diagnosis • 18 years of age or older exclusion: • presence of central nervous system or brain metastases. intervention gemcitabine 1500 mg/m2 body surface area (bsa) intravenously (iv) over 30 minutes (+/-5 minutes) on days 1 and 15 of each cycle. results variable: progression free survival number of participants: 29 median (months): 10.4 (5.6 to 15.2) adverse events total: 3/29 • neutropenic fever 1 / 29 (3.45%) • peripheral neuropathy 1 / 29 (3.45%) • seizure/syncope 1 / 29 (3.45%) ## task definition we define two different tasks on the nli4ct dataset, task 1, a textual entailment task, and task 2, an evidence selection task. each instance in nli4ct contains a ctr premise and a statement. the ctr premise comprises one of the four sections of a ctr, with both a mean token length and a standard deviation of 230. the statements are sentences ranging from 10-35 tokens (see example in figure 1 ). on average there are 7.74 pieces of relevant evidence that need to be selected out of a total of 21.67 facts per ctr premise. there are two types of instances in nli4ct, in single type instances the ctr premise is the primary ctr, and the statement makes a claim about the information in this ctr. whereas in the comparison type the premise contains a primary and a secondary trial, and the statement will make a claim comparing and contrasting the two trials. to summarize: task 1 requires determining the entailment relation between the ctr premise and the statement, therefore the output is either an entailment or contradiction label, as shown in figure 1 . task 2 requires outputting a set of supporting facts, extracted from the ctr premise, necessary to justify the label predicted in task 1. ## evaluation metrics task 1 is a binary classification task, the statement is either labeled as entailment or contradiction. we evaluate performance on task 1 by computing the precision, recall, and macro f1-score of the predicted labels against the annotated gold labels (entailment/contradiction). for task 2 the output is a subset of the facts within the ctr premise. we can frame the evidence selection problem as a ranking problem @xcite . models assign a score to each fact in a ctr premise, which can be used to rank the facts. in this framework, solving the task requires the model to rank all the facts in the gold evidence higher than the irrelevant facts. we evaluate performance on this task using mean average precision map @ k @xcite . ## conclusions & future work we present 2 tasks, textual entailment of claims over clinical trial reports (ctrs), and extracting evidence from ctrs to support or contradict these claims. we provide a corpus of 2400 expertannotated instances to test and train models on these tasks. if models could be developed to solve these tasks, it would progress the field toward the efficient retrieval and synthesis of published literature to support evidence-based medicine. additionally, it would address several important nlu challenges related to numerical reasoning, biomedical nli, and inference over long texts. our baselines outline the weaknesses of existing nli models, particularly with regard to numerical inference. we tested 6 sota nli models on task 1 and reported a maximum f1 score of 0.644. additionally, we show that the limiting factor for current models is not evidence selection, but rather leveraging that evidence, and using numerical and biomedical inference for predictions. in future works, we plan to generate a subset of adversarial instances via a controlled set of perturbations of words/numbers in statements. allowing us to perform interventional and causal analysis of models' behavior for numerical and multi-hop inference @xcite , in the context of biomedical applications, as well as to extend the size of the dataset. our corpus, competition leaderboard, code, and website are available on codalab and github.