IITK at S em E val-2024 Task 2: Exploring the Capabilities of LLM s for Safe Biomedical Natural Language Inference for Clinical Trials (2024)

Abstract: Large Language models (LLMs) have demonstrated state-of-the-art performance in various natural language processing (NLP) tasks across multiple domains, yet they are prone to shortcut learning and factual inconsistencies. This research investigates LLMs’ robustness, consistency, and faithful reasoning when performing Natural Language Inference (NLI) on breast cancer Clinical Trial Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We examine the reasoning capabilities of LLMs and their adeptness at logical problem-solving. A comparative analysis is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro under zero-shot settings using Retrieval-Augmented Generation (RAG) framework, integrating various reasoning chains. The evaluation yields an F1 score of 0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test dataset.

## introduction clinical trials serve as essential endeavors to evaluate the effectiveness and safety of new medical treatments, playing a pivotal role in advancing experimental medicine. clinical trial reports (ctrs) detail the methodologies and outcomes of these trials, serving as vital resources for healthcare professionals in designing and prescribing treatments. however, the sheer volume of ctrs (e.g., exceeding 400,000 and proliferating) presents a challenge for comprehensive literature assessment when developing treatments @xcite . natural language inference (nli) @xcite emerges as a promising avenue for large-scale interpretation and retrieval of medical evidence bridging recent findings to facilitate personalized care @xcite . the semeval 2024 task 2 on the natural language inference for clinical trials (nli4ct) @xcite revolves around annotating statements extracted from breast cancer ctrs foot_0 and determining the inference relation between these statements and corresponding sections of the ctrs, such as eligibility criteria, intervention, results, and adverse events. by systematically intervening in the statements, targeting numerical, vocabulary, syntax, and semantic reasoning, the task aims to investigate large language models (llm)s' consistency and faithful reasoning capabilities. in this paper, we experiment with gemini pro @xcite , gpt-3.5 @xcite , flan-t5 @xcite and several pre-trained language models (plms) trained on biomedical datasets, namely biolinkbert @xcite , scibert @xcite , clinicalbert @xcite . we conducted zero-shot evaluations of gemini pro and gpt-3.5, employing retrieval augmented generation (rag) framework @xcite integrating tree of thoughts (tot) reasoning @xcite facilitating multiple reasoning paths. our experiments involved applying various instruction templates to guide the generation process. these templates were refined through manual comparison of the labels within the training dataset against those generated by the models. the plms were fine-tuned on the provided training dataset, while the flan-t5 model was assessed under zero-shot conditions. gemini pro emerged as the top-performing model among all the experimented models, achieving an f1 score of 0.69, with consistency and faithfulness scores of 0.71 and 0.90, respectively, on the official test dataset. notably, a comparative analysis between gpt-3.5 and gemini pro revealed shortcomings in gpt-3.5's performance, particularly in instances requiring numerical reasoning. for detailed examination of such instances, please clinical trial report 1 eligibility criterion ... intervention ... single arm of healthy postmenopausal women to have two breast mri (baseline and post-treatment) ... results ... adverse events adverse events 1: • total: 69/258 (26.74%) • anaemia 3/258 (1.16%) ... adverse events 2: • total: 64/224 (28.57%) • anaemia 2/224 (0.89%) ... clinical trial report 2 eligibility criterion ... intervention ... healthy women will be screened for magnetic resonance imaging (mri) contraindications, and then undergo contrast injection, and swift acquisition. ... ## system overview llms such as gpt-3 @xcite ) and gemini pro @xcite have shown remarkable performances across various tasks. for the nli4ct task, we have experimented with gemini pro, gpt-3.5, flan-t5 @xcite , biolinkbert @xcite , scibert @xcite , clinicalbert @xcite and clinicaltrialbiobert-nli4ct 7 . the performance of the different models is shown in figure 7 . zero-shot evaluation was done on gemini pro and gpt-3.5, flan-t5 was instruction fine-tuned following kanakarajan and sankarasubbu (2023), and the rest of the models were trained on the given train and development dataset. gemini pro and gpt-3.5 were considered for further experimentation because of their superior performance. the proposed system utilizes structured instruction templates and multi-turn conversation techniques to generate explanations and labels for the statements provided as input, as shown in figure 2 . reasoning is an essential ability required by an llm to solve complex problems (qiao et al., 7 https://huggingface.co/domenicrosati/clinicaltrialbiobertnli4ct 2022). tree of thoughts (tot) framework @xcite and chain-of-thought (cot) reasoning @xcite is integrated into the models, facilitating multiple reasoning paths. ## results the zero-shot evaluation of gemini pro yielded an f1 score of 0.69, with a consistency of 0.71 and a faithfulness score of 0.90 on the official test dataset. our system achieved a fifth-place ranking based on the faithfulness score, a sixteenth-place ranking based on the consistency score, and a twenty-firstplace ranking based on the f1 score. gemini pro outperforms gpt-3.5 with an improvement in f1 score by +1.9%, while maintaining almost similar consistency score. additionally, the faithfulness score of gemini pro improves by +3.5% compared to gpt-3.5, as illustrated in ## conclusion this paper presents an evaluation of several pretrained language models (plms), and gpt-3.5, gemini pro, under zero-shot conditions. our analysis focuses on assessing the reasoning capabilities of gpt-3.5 and gemini pro and their adeptness at logical problem-solving. in the nli4ct task, we achieved an f1 score of 0.691, consistency of 0.71, and faithfulness of 0.90. additionally, our findings underscore that prompt engineering is crucial for large language models (llms). we have made our instruction templates and code publicly available to facilitate reproducibility.