B p H igh at S em E val-2023 Task 7: Can Fine-tuned Cross-encoders Outperform GPT -3.5 in NLI Tasks on Clinical Trial Data? (2023)

Abstract: Many nations and organizations have begun collecting and storing clinical trial records for storage and analytical purposes so that medical and clinical practitioners can refer to them on a centralized database over the internet and stay updated with the current clinical information. The amount of clinical trial records have gone through the roof, making it difficult for many medical and clinical practitioners to stay updated with the latest information. To help and support medical and clinical practitioners, there is a need to build intelligent systems that can update them with the latest information in a byte-sized condensed format and, at the same time, leverage their understanding capabilities to help them make decisions. This paper describes our contribution to SemEval 2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT). Our results show that there is still a need to build domain-specific models as smaller transformer-based models can be finetuned on that data and outperform foundational large language models like GPT-3.5. We also demonstrate how the performance of GPT-3.5 can be increased using few-shot prompting by leveraging the semantic similarity of the text samples and the few-shot train snippets. We will also release our code and our models on open source hosting platforms, GitHub and HuggingFace.

## introduction the multi-evidence natural language inference for clinical trial data (nli4ct) shared task @xcite aims at building clinical support systems on top of clinical trial reports to help clinical professionals. clinical trials are studies where the researchers test new ways to prevent, diagnose or treat a disease/disorder or find newer ways to improve the quality of life for people with chronic diseases. these studies are available as clinical trial reports ( ctrs ), providing detailed study methods and results. clinical decision support systems can aid clinical and medical professionals in their day-to-day tasks and provide meaningful insights to complement their productivity. clinical decision support systems consist of common frameworks and analytical software built on ehrs(electronic health reports). however, the growth and development in ai and nlp opens new doors and opportunities which entrepreneurs, researchers, and engineers can leverage to build next-gen intelligent systems which can leverage the contextual information of ehrs and ctrs. nevertheless, one must be vigilant and take extra precautions while building intelligent systems for clinical domains, as it is a complex domain @xcite . recently many researchers have started building domain-specific versions of famous transformer architectures like bert, gpt, and t5. @xcite demonstrate how domain-specific clinical bert, clinical biobert outperform standard bert model on mednli natural language inference task @xcite . however, there are certain limitations that such models face, which are primarily due to the data such domainspecific models are trained on. as the data to train clinical models is usually small and usually belongs to a single institution, group of institutions, or a single region, it cannot generalize to the differences in clinical practice in other institutions and regions. there is a need for creating more datasets in the clinical nlp domain and also for consideration to be given to including data from different regions and institutions. creating large multi-lingual datasets containing clinical data of multiple geographies can help create more robust and generalized models. many researchers also believe that more stress should be given to building generalized large language models trained on a large corpus of open data that performs well in the clinical domain instead of pre-training and fine-tuning domain-specific models. after the release of @xcite , many researchers have also tried to experiment and observe whether gpt-3 can be used for clinical systems and how it might benefit clinical practitioners and patients @xcite . the most significant pitfalls they find are the interpretability and hallucination in generative models. many countries have also realized why it is necessary to have clinical datasets and store clinical trial records digitally that can represent the experiments on diseases and illnesses more relevant to their population. 1 the nli4ct shared task also uses one such database by the u.s. national library of medicine with clinical trial records of clinical investigations carried out globally by private and public entities. 2 the organizers of the task have created a set of breast cancer clinical trial records, statements, explanations, and labels that have been annotated by experts who belong to the clinical field. the nli4ct shared task consists of two subtasks, subtask 1 is a textual entailment task, whereas subtask 2 is a retrieval-based task. we participate in subtask 1 only and demonstrate how cross-encoder models fine-tuned on small training datasets can outperform gpt-3.5 @xcite in zero-shot settings. we fine-tune several models based on sentence transformer and cross-encoder architecture for sentence pair modeling. we also compare our models with gpt-3.5 davinci (text-davinci-003 according to the openai platform) in zero-shot and few-shot settings. we also show the effect freezing of base transformer layers has while training cross-encoders and on their performance. we will release all our code on github foot_2 and fine-tuned models on huggingface. 4 ## dataset description the primary dataset consists of a set of ctrs(clinical trial reports) condensed into 4 sections. the subtask-specific dataset for subtask 1 consists of a set of statements which are sentences that act as the premise and are based on the hypothesis, which consists of information present in a particular section of 1 ctr or a comparison of 2 ctrs. the subtask 1 dataset also provides a label whether a given statement-hypothesis pair is an entailment or contradiction. we explain the 4 sections present in the primary dataset below with the help of the clinical trial report with id-nct00003782 :-1. eligibility criteria:-it acts as a guard which decides which subjects to enroll in the study based on its shoulder angels the inclusion and exclusion criteria. the inclusion criteria -lays down statements, and volunteers who fit in those statements are chosen for the study. the exclusion criteria -provide statements that, if present, eliminate the volunteers from participating in the study. for example, in the above-mentioned report, the following inclusion criteria are used-"no ulceration, erythema, infiltration of the skin or underlying chest wall (complete fixation), peau d'orange, or skin edema of any magnitude(tethering or dimpling of the skin or nipple inversion allowed ) " suppose the volunteer is found to have ulceration /erythema/ infiltration/ edema/ peau d' orange over skin or chest wall infiltration. in that case, the breast cancer is no longer stage iii a and has already progressed to stage iii b or a higher stage. dimpling of the skin or nipple inversion is allowed as it is not considered skin involvement and hence will not change the staging. for example, in the above-mentioned report, intervention is done by giving chemotherapy using drugs like doxorubicin, cyclophosphamide, and docetaxel. these anticancer drugs act during different stages of cell division to stop cancer cells from proliferating. doxorubicin and cyclophosphamide act by destabilizing dna structure. docetaxel affects the microtubular system in cells, which helps in equal dna distribution during cell division. for example, in the above-mentioned report, the 8-year survival was maximum ( 83 % ) in subjects who received doxorubicin with cyclophosphamide followed by docetaxel as chemotherapy. for example, serious adverse effects like febrile neutropenia, thromboembolic events, etc., were reported in the above-mentioned report. febrile neutropenia is fever with lab reports suggestive of reduced neutrophil counts, cells protecting the body against infections. thromboembolic events include blood clot formation in deep veins and complications that may arise due to it. ## methodology this section discusses our approaches for tackling subtask 1 of the nli4ct task. for all further references, whenever we refer to a model which has been fine-tuned by using the nli4ct subtask 1 training dataset, we prepend 'nli4ct-' to the pretrained model's name. using the sentence transformers package foot_4 , we fine-tune one sentencetransformer model and multiple cross-encoder models. we also compare the performance of our finetuned models with gpt-3.5 davinci (text-davinci-003 according to the openai platform) in zero-shot and few-shot settings. ## results the results for all approaches on the dev and test dataset are presented in ## conclusion we map the textual entailment task as a sentencepair classification task, and we study the literature to find all approaches that can be used to solve this task. we also study clinical literature to grasp our dataset and understand the definition of whereas properties properly. we also try to find pretrained models already trained on some clinical or biomed-ical domain that can be finetuned further for our task. we agree on specific approaches/pretrained models and finetune our models based on those pretrained models. to get a good grasp of the performance of our models, we leverage very large language models and compare performance with gpt-3.5 davinci state-of-the-art models in zeroshot and few-shot settings. we devise a few-shot strategy based on semantic similarity to find the top two few-shot train snippets for each test sample. we observe that finetuned cross-encoders perform well compared to zero-shot gpt-3.5, and some models also perform comparably to the few-shot gpt-3.5. we also observe that freezing the base transformers' layers while training cross-encoders considerably affects model performance on downstream tasks. further, we would explore how to increase our model performance and achieve similar performances as unfrozen models by freezing specific layers while training crossencoders to see their effects. we would also experiment to train cross-encoders based on more pretrained models like biolinkbert @xcite .