Saama AI Research at S em E val-2023 Task 7: Exploring the Capabilities of Flan-T5 for Multi-evidence Natural Language Inference in Clinical Trial Data (2023)

Abstract: The goal of the NLI4CT task is to build a Natural Language Inference system for Clinical Trial Reports that will be used for evidence interpretation and retrieval. Large Language models have demonstrated state-of-the-art performance in various natural language processing tasks across multiple domains. We suggest using an instruction-finetuned Large Language Models (LLMs) to take on this particular task in light of these developments. We have evaluated the publicly available LLMs under zeroshot setting, and finetuned the best performing Flan-T5 model for this task. On the leaderboard, our system ranked second, with an F1 Score of 0.834 on the official test set.

## introduction building a natural language inference (nli) system for clinical trial reports (ctrs) is the objective of the multi-evidence natural language inference for clinical trial data (nli4ct) task @xcite , which focuses on the interpretation and retrieval of medical evidence. clinical practitioners struggle to keep up with the large number of ctrs published yearly to provide individualised, evidence-based care @xcite . the main task is separated into two subtasks. task 1 is to determine the relationship of the claim to the information in a single ctr or to compare two ctrs to see if there is entailment or contradiction. the supporting factor required to back up the predicted label in task 1 must be extracted from the ctrs in task 2. english is the only language used in the task. @xcite , @xcite , and @xcite models revolutionised the field of natural language processing by introducing a new technique of pretraining on language modelling and finetuning on task-specific data using supervised data, resulting in state-of-the-art results on a variety of datasets and benchmarks @xcite , @xcite . when applying the same model architecture to domains such as biomedical, they also achieved state-of-the-art results on domain specific tasks; domain specific trained models like biobert @xcite , @xcite , bioelectra raj @xcite further improved performance. furthermore, the introduction of large language models (llms) @xcite , palm @xcite , @xcite , @xcite , @xcite ushered in a new strategy known as prompt-based learning @xcite , which allows these models to be more effectively adapted to tasks. prompt-based learning is a technique for using llm models to predict what will happen next in a sentence. the models fill in the blanks to form the final sentence, which can then be used to determine the correct answer. with the implementation of prompt-based learning, the model can now carry out few-shot or even zeroshot learning, enabling it to adjust to new circumstances with either a limited amount of labelled data or none at all. llms' potential for zeroshot success on unseen tasks was improved by supervised finetuning their instruction-following skills across multiple tasks. instruction tuning applies to both decoder only models like gpt-3, opt and encoder-decoder models like @xcite . in this paper, we investigate the effectiveness of flan- @xcite models for the nli4ct task. flan-t5 is an instruction tuned t5 model. in the zeroshot setting, we assessed flan-t5 models with various instructions for this task. on the official test set, our finetuned flan-t5 model on the nli4ct task received an f1 score of 0.834, placing it second on the leader-board. the code to reproduce the experiments mentioned in this paper is publicly available. the annotated statements are sentences that make some claim about the information contained in one of the sections in the ctr (premise). task 1 is to predict the entailment vs contradiction between ctr-statement pairs. the statements may make claims about a single ctr or compare two ctrs. for the "single" type, all evidence will be contained in the primary ctr, while for the "comparison" type, evidence will have to be retrieved from both ctrs same section. task 2 is to identify the supporting factor extracted from the ctrs (premise) to justify the prediction from task 1, given ctr, annotated statement, and prediction. there are 999 breast cancer ctrs in the dataset. the datasets, which are divided into train, development, and test sets, contain a total of 2400 annotated statements. the distribution of labels between the train and development sets is even. eligibility sections are used most often in single annotated statements, while intervention is used in comparison statements. detailed statistics are in ## system overview it has been demonstrated that finetuning the llms to follow natural language instructions improved the models' ability to perform better on previously unseen tasks in zeroshot and few-shot settings. for the ctrs nli task, we will use flan-t5, an instruction tuned t5 model. the flan-t5 model was chosen based on its zeroshot performance in this particular task. t0pp sanh et al. (2021 2022 , instruct-gpt-175b were the other models considered for this experiment. figure 1 depicts the zeroshot performance of these models. for all the models, instructions are adapted to their instruction tuning style. there is no single instruction that works across all models efficiently. the expected model labels are also passed to the model via the same instruction to get the proper output from the model. under the zeroshot settings, the flan-t5 xxl(11b) and xl(3b) have the best f1 scores. the t5 is an encoder-decoder sutskever et al. ( 2014 ) model transforms all tasks into a text-to-text format. the model has parameters ranging from 60 million to 11 billion. a denoising objective is used to pre-train the t5 model. the flan-t5 model is trained from @xcite language modeling objective adapted t5 model. the original t5 model is trained with a maximum sequence length of 512, whereas the flan-t5 model is trained with a maximum encoder length of 1024 and a decoder length of 256. the flan-t5 model has been finetuned on 1836 tasks with 15m examples. for zeroshot evaluation and finetuning experiments, we construct the input to the model using an instruction template, ctrs data, and the statement as shown in figure 2 . given the following instruction to the model, the model generates the entailment or contradiction label. ## conclusion in our work, we have evaluated various instruction tuned large language models under the zeroshot setting and finetuned the best-performing instruction tuned t5, flan-t5-xxl and achieved an f1 score of 0.834 and finished second position in the nli4ct task. we observed that instruction tuned models are better for datasets with fewer training samples. for flan-t5, the model performance steadily increased as the model size increased. in the zeroshot setting, we also observe that the in-struction/prompt given to the model significantly affects its performance. we open-source instruction templates, code and pre-trained models for the reproducibility of our work.