Sebis at S em E val-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports (2023)

Abstract: With the increasing number of clinical trial reports generated every day, it is becoming hard to keep up with novel discoveries that inform evidence-based healthcare recommendations. To help automate this process and assist medical experts, NLP solutions are being developed. This motivated the SemEval-2023 Task 7, where the goal was to develop an NLP system for two tasks: evidence retrieval and natural language inference from clinical trial data. In this paper, we describe our two developed systems. The first one is a pipeline system that models the two tasks separately, while the second one is a joint system that learns the two tasks simultaneously with a shared representation and a multi-task learning approach. The final system combines their outputs in an ensemble system. We formalize the models, present their characteristics and challenges, and provide an analysis of achieved results. Our system ranked 3rd out of 40 participants with a final submission.

## introduction clinical trials are research studies carried out in human subjects to assess the effectiveness and safety of medical, surgical, or behavioral interventions @xcite . these investigations constitute the main approach that medical researchers use to determine whether a novel treatment such as a new medication, procedure, diet, or medical device is safe and efficient in humans. clinical trials are often used to compare the effectiveness of a new treatment against the standard treatment or placebo treatment and to assess its adverse effects. when performed rigorously, clinical trials present the most valuable resource for informing evidence-based healthcare decisions. on average, more than 100 reports of cts are published every day @xcite . keeping up with all their results and novel discoveries is a time-consuming and often practically impossible endeavor. this has brought to light dedicated organizations, such as cochrane, foot_0 that manually synthesize these clinical outcomes @xcite . nevertheless, they struggle to keep up with the ever-increasing amount of literature and new studies. for this purpose, automated approaches based on machine learning (ml) and natural language processing (nlp) can be developed to facilitate the process of inferring new knowledge and finding evidence from clinical trial reports. the semeval-2023 task 7, titled multi-evidence natural language inference for clinical trial data (nli4ct), focused on developing nlp systems for making conclusions and finding evidence in clinical trial reports (ctrs) @xcite . it featured two subtasks that were strongly coupled together. the first subtask was, given a clinical trial document and a claim, to develop a model that infers a logical relation between them, namely either entailment or contradiction. the second subtask was, given all the sentences in the clinical trial document, to develop a model that selects a subset of those sentences that serve as evidence for making a decision on the logical entailment relation between the claim and the document. the task is illustrated in figure 1 . it shows two different claims, each coupled with an excerpt from a clinical trial report and a gold final label. the gold evidence sentences are highlighted for emphasis. for this task, we developed two competitive nlp systems. the first one is a pipeline system, which learns to perform the tasks of evidence retrieval and textual entailment separately and takes into account one sentence at a time during the evidence selection process. the second approach is a joint system, which jointly learns the two subtasks with a unified representation of a claim and the whole document, and uses a shared training procedure. the joint system performed better on the test set, but the final system is an ensemble system that utilizes the power of both systems by combining their outputs with an averaging function. our final system achieved the 3rd place in the competition out of 40 submitting participants. we provide the code of our system in a github repository. foot_1 ## related work the task of natural language inference (nli) consists of inferring whether there is logical entailment or contradiction between two pieces of text -a premise and a hypothesis. it has been researched since the early days of nlp when it was mostly tackled with rule-based and linguistically informed approaches @xcite . a big surge of interest in the task occurred after the release of the snli dataset @xcite , as well as follow-up datasets like multinli (mnli) @xcite and adversarial-nli (anli) @xcite . the task has also been researched in the clinical domain -the dataset mednli @xcite features more than 14,000 pairs of claims from clinical notes and patient records, extracted from the mimic-iii database @xcite . the task of evidence retrieval, or more precisely evidence sentence selection, has been studied as a step in common nlp tasks like machine reading comprehension (mrc) @xcite and question answering (qa) @xcite . its main purpose is to improve the explainability and interpretability of models' predictions but also to aid its reasoning process for making decisions. combining the steps of evidence retrieval and textual entailment recognition comprises the task of automated fact-checking or claim verification, where the goal is to assess a claim's veracity based on the evidence related to it @xcite . this task has mostly been concerned with verifying claims related to news events, society, and politics, but has been increasingly so researched in scientific, biomedical, and clinical domains @xcite . ## dataset the clinical trials used for constructing the dataset originate from clinicaltrials.gov, foot_2 a database of privately and publicly funded clinical studies conducted around the world, run by the u.s. national library of medicine. all the clinical trial reports in the dataset are related to breast cancer and are written in english. there is a total of 1,200 clinical trials and 2,400 corresponding claims related to these ctrs, which were written and annotated by clinical domain experts, clinical trial organizers, and research oncologists. for the task, each ctr may contain 1-2 patient groups, called cohorts or arms, and these groups may receive different treatments, or have different baseline characteristics. the dataset consists of a total of 2,400 claims and was split into a training dataset of 1,700 claims, a development dataset of 200 claims, and a hidden test dataset of 500 claims. there are two types of claims, those related only to a single ctr and other ones related to two ctrs, which usually make some form of comparison between the two reports. each of the claims is related to only one of the following four sections: ## system description in this chapter, we will describe the architecture of the developed systems and the choice of base models comprising the systems. even though the task features two standalone subtasks and participating in each of them was optional, we found the best synergy is achieved by combining them. the selection of appropriate evidence sentences in subtask 2 is an important prerequisite for recognizing textual entailment in subtask 1. this is because most textual entailment datasets used for training the nli systems are modeled to recognize entailment between two sentences only, so the performance between a sentence and a whole document is still underwhelming @xcite . therefore, it was important to narrow down the whole document to a shorter span of text with only the relevant evidence sentences. ## results & analysis in this section, we report on results achieved by our systems and provide a qualitative error analysis with challenging examples from the dataset. ## conclusion in this paper, we describe our solution for the semeval-2023 task 7, dealing with natural language inference and evidence retrieval from clinical trial reports. we motivate the task, discuss related work, provide formal definitions of the developed systems, present the results, analyze the performance of models, and discuss some challenges in the process. we developed two types of systems -a pipeline system, which learns evidence retrieval and textual entailment sequentially, and a joint system, which learns the two tasks simultaneously. the final system combines them into an ensemble and achieved the 3rd place in the competition out of 40 teams with a final submission. we anticipate this system will be useful for facilitating the work of medical experts in synthesizing the results and outcomes of the ever-increasing amount of clinical trial reports, as well as be used by the nlp community for the related tasks of biomedical question answering, automated claim verification, and recognizing textual entailment. the system could be improved in the future by overcoming the challenges related to commonsense, numerical, and multi-hop reasoning, or by injecting additional medical and world knowledge to it.