USMBA - NLP at S em E val-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials using Bert (2024)

Abstract: This paper presents the application of BERT inSemEval 2024 Task 2, Safe Biomedical Natu-ral Language Inference for Clinical Trials. Themain objectives of this task were: First, to in-vestigate the consistency of BERT in its rep-resentation of semantic phenomena necessaryfor complex inference in clinical NLI settings.Second, to investigate the ability of BERT toperform faithful reasoning, i.e., make correctpredictions for the correct reasons. The submit-ted model is fine-tuned on the NLI4CT dataset,which is enhanced with a novel contrast set,using binary cross entropy loss.

## introduction nli stands for natural language inference. it is a task in natural language processing (nlp) where the goal is to determine the relationship between two text segments: a premise and a hypothesis. the task typically involves classifying whether the hypothesis is entailed, contradicted, or neutral with respect to the premise. nli has emerged as a beacon of hope for clinical trial reports (ctrs). its ability to handle vast amounts of medical evidence could revolutionize the interpretation and retrieval of ctrs. clinical trials stand as pillars in experimental medicine, scrutinizing the efficacy and safety of novel treatments @xcite . ctrs meticulously outline trial methodologies and findings, guiding the development of targeted interventions for patients. yet, the staggering quantity of published ctrs renders manual review impractical for devising new treatment protocols @xcite . the proposed textual entailment task aims to advance the understanding of models' behavior and enhance existing evaluation methodologies for clinical nli. by systematically applying controlled interventions, each engineered to probe a specific semantic phenomenon in natural language and numerical inference, the study seeks to assess the robustness, consistency, and faithfulness of models in clinical settings, thereby investigating their reasoning capabilities. in this paper, we present our findings on se-meval 2024 task 2, safe biomedical natural language inference for clinical trials @xcite . the aim of our method is to assess the robustness, consistency, and faithfulness of bert @xcite , pre-training of deep bidirectional transformers for language understanding, on the clinical nli. our method follows to steps: the first step is fine-tuning bert on the multievidence natural language inference for clinical trial data (nli4ct) @xcite which is enhanced with a novel contrast set. then, the prediction step, consists on the determining the inference relation (entailment vs contradiction) between ctr -statement pairs. the rest of the paper is structured in the following manner: section 2 provides the main objective of the task. section 3 describes our system. section 4 details the experiments. and finally, section 5 concludes this paper. ## task description this paper focuses on the task of textual entailment within the domain of clinical trial data analysis, specifically targeting clinical trial reports (ctrs). ctrs serve as comprehensive documents containing essential information regarding various aspects of clinical trials, including eligibility criteria, interventions, results, and adverse events. automating the analysis of ctrs through natural language processing techniques can significantly facilitate researchers' understanding and decision-making processes. the task of nli4ct involves analyzing annotated statements and determining their inference relation with the information contained in the ctr premises. these statements, characterized by an average length of 19.5 tokens, make claims about various sections of the ctrs, including: the nli4ct task presents several challenges inherent to clinical trial data analysis, including numerical and quantitative reasoning, vocabulary and syntax variations, and comprehension of complex semantic structures. to address these challenges, interventions have been implemented targeting the following aspects: ## system description to evaluate bert on the semeval 2024 task 2: safe biomedical natural language inference for clinical trials, we have fine-tuned bert model on the nli4ct dataset. we follow standard procedures for fine-tuning transformer-based models on natural language inference tasks. here's a description of the process: ## experimental results we experimented our model on on the semeval 2024 task 2: safe biomedical natural language inference for clinical trials. the experiment has been conducted in google colab environment foot_0 , the following libraries: transformers -hugging face foot_1 @xcite , and tensorflow foot_2 were used to train and to assess the performance of the model. ## conclusion in this paper, an investigation is conducted into the utilization of bert for nli4ct, which underscores the complex nature of textual entailment tasks within the medical domain. the described approach tackles semeval 2024 task 2: safe biomedical natural language inference for clinical trials. the model secured the 27th position among a total of 32 teams. despite challenges such as domain-specific terminology and nuanced semantics, our study reveals the potential for advancements in automated analysis of clinical trial reports. by recognizing the need for domain-specific approaches and leveraging the models, we pave the way for more accurate and reliable models tailored to the intricacies of medical data. ultimately, our findings advocate for continued research and development efforts aimed at enhancing natural language processing techniques for clinical applications, thereby contributing to improved healthcare outcomes and medical decisionmaking processes.