I 2 R at S em E val-2023 Task 7: Explanations-driven Ensemble Approach for Natural Language Inference over Clinical Trial Data (2023)

Abstract: In this paper, we describe our system for SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. Given a CTR premise, and a statement, this task involves 2 sub-tasks (i) identifying the inference relation between CTR - statement pairs (Task 1: Textual Entailment), and (ii) extracting a set of supporting facts, from the premise, to justify the label predicted in Task 1 (Task 2: Evidence Retrieval). We adopt an explanations driven NLI approach to tackle the tasks. Given a statement to verify, the idea is to first identify relevant evidence from the target CTR(s), perform evidence level inferences and then ensemble them to arrive at the final inference. We have experimented with various BERT based models and T5 models. Our final model uses T5 base that achieved better performance compared to BERT models. In summary, our system achieves F1 score of 70.1% for Task 1 and 80.2% for Task 2. We ranked 8th respectively under both the tasks. Moreover, ours was one of the 5 systems that ranked within the Top 10 under both tasks.

## introduction natural language inference (nli) is the task of detecting inferential relationships between a premise text and a hypothesis text @xcite , which is considered fundamental in natural language understanding (nlu) research @xcite . the objective is to determine whether hypothesis h is true ('entailment'), false ('contradiction'), or un-determined ('neutral') given the premise p. this task, formerly known as recognizing textual entailment (rte) @xcite has long been a popular task among researchers. moreover, contribution of datasets from past shared tasks @xcite , and recent research @xcite have pushed the boundaries for this seemingly simple, but challenging problem. nli brings an opportunity to support the largescale interpretation and retrieval of medical evidence. understanding the contextual evidence will support personalized care for patients @xcite , e.g., analysis of clinical trial reports (ctrs). this is especially useful as, in the past few years, the number of publications of ctrs has increased exponentially and it has become impracticable for clinical practitioners to stay updated @xcite . semeval-2023 task 7: multi-evidence natural language inference for clinical trial data @xcite attempts to capture and investigate this opportunity via two shared tasks namely (i) task 1: textual entailment: given a statement, which make some type of claim about the information contained in one of the sections in the ctr premise, identify the inference relation (either entailment or contradiction) between ctrstatement pairs. (ii) task 2: given a ctr premise, and a statement, extract a set of supporting facts, from the premise, to justify the label predicted in task 1. several nli systems have been proposed in the literature. @xcite @xcite @xcite @xcite however, most of the successful ## experiments in the nli4ct challenge dataset, there are 2400 instances provided, and the labels are evenly split across train/dev/test (1700/200/500) examples. our model, as described in section 3, was initialized from the pre-trained t5 base trained on a variety of general text on mlm scheme. we have used this as a starting point to fine tune the model for the given dataset and adjusted the hyperparameters based on the best performance. we have also used the feature relevancy to categorize the relevant sentences. during t5 training, we set the number of beams as 50 and the number of returned sequences as 5. we randomly split the instances into 80% training and 20%, repeated validation five times and report the average performance. the experiments were executed on nvidia-geforce rtx 2080 series with eight cores of gpu machines with 8*12 gb of memory for all our experiments. also, to train t5 large models, we have used nvidia-geforce tesla v100 series sxm2-32gb with 5 cores of gpu machines. models were trained for 3-5 hours for training and reasoning. the pretrained weights for the transformers prior to fine-tuning were from the huggingface nlp library. the results are presented below, where performance metrics are averaged over 10 runs and quoted in % for easier interpretation, unless stated otherwise. ## results for task 1, our system achieved low precision and high recall compared to other teams that participated in the task. the model performance in task 2 seems to be better compared to all metrics.t5 models works better for both task 1 and task 2. ## conclusions in this paper, we described our system for semeval-2023 task 7: multi-evidence natural language inference for clinical trial data. we adopted an explanation driven nli approach to develop a methodology to tackle both the tasks. we have experimented with various bert based models and t5 models, and our final model uses t5 base that achieved better performance. our system achieves f1 score of 70.1% for task 1 and 80.2% for task 2. we ranked 8th respectively under both the tasks. moreover, ours was one of the 5 systems that ranked within the top 10 under both tasks. we noted that the t5 base model for natural language inference task performs reasonably well for clinical trial dataset. some interesting research questions for further investigation are: 1) generative text to text framework t5 models could perform well for nli along with arithmetic reasoning. 2) pre-training data with domain specific clinical corpora might increase performance. using pre-trained models trained on large biomedical corpora such as scifive @xcite and biobert @xcite models are steps along this direction.