C ares AI at S em E val-2024 Task 2: Improving Natural Language Inference in Clinical Trial Data using Model Ensemble and Data Explanation (2024)

Abstract: Large language models (LLMs) have demonstrated state-of-the-art performance across multiple domains in various natural language tasks. Entailment tasks, however, are more difficult to achieve with a high-performance model. The task is to use safe natural language models to conclude biomedical clinical trial reports (CTRs). The Natural Language Inference for Clinical Trial Data (NLI4CT) task aims to define a given entailment and hypothesis based on CTRs. This paper aims to address the challenges of medical abbreviations and numerical data that can be logically inferred from one another due to acronyms, using different data pre-processing techniques to explain such data. This paper presents a model for NLI4CT SemEval 2024 task 2 that trains the data with DeBERTa, BioLink, BERT, GPT2, BioGPT, and Clinical BERT using the best training approaches, such as fine-tuning, prompt tuning, and contrastive learning. Furthermore, to validate these models, different experiments have been carried out. Our best system is built on an ensemble of different models with different training settings, which achieves an F1 score of 0.77, a faithfulness score of 0.76, and a consistency score of 0.75 and secures the sixth rank in the official leaderboard. In conclusion, this paper has addressed challenges in medical text analysis by exploring various NLP techniques, evaluating multiple advanced natural languagemodels(NLM) models and achieving good results with the ensemble model. Additionally, this project has contributed to the advancement of safe and effective NLMs for analysing complex medical data in CTRs.

## introduction clinical trials play a crucial role in advancing medical knowledge, evaluating the safety and efficacy of new treatments, and improving patient care @xcite which are essential for the development of new drugs, therapies, and medical interventions. most importantly, they involve systematic investigations that aim to answer specific research questions and provide evidence-based guidance for medical decision-making @xcite . moreover, clinical trial reports (ctrs) have been published at an accelerated rate due to the rapid development of digital health. currently, there are more than 10,000 ctrs just for breast cancer @xcite . also, medical professionals have developed evidence-based clinical diagnoses through the increasing number of clinical trial reports (ctrs) @xcite , which serve as a broad source of factual and scientific information. despite these ctrs, drawing valuable conclusions from these reports can be an uphill task due to the different medical domains and the unstructured nature of the report. recent improvements in natural language processing (nlp) systems, on the other hand, have led to the idea of using multiple language models that have already been trained in the medical field to efficiently carry out medical nlp tasks. the growth of ctrs has also made it possible for a natural language inference (nli) system to be created that can help with medical interpretation and finding evidence for individualized evidence-based therapy. @xcite used instructgpt with zero-shot and few-shot settings to extract information from clinical text. in addition, the authors introduced new datasets for benchmarking for few-shot clinical information extraction. the work in @xcite introduced a new tool, the acta automated tool, to support evidence-based clinical decision-making. the authors in @xcite proposed a new model, linkbert, that incorporates document link knowledge for medical domains. despite substantial research on the use of advanced nlp approaches in the medical domain, evaluation benchmarks remain inadequate. the semeval 2024 task 2: safe biomedical natural language inference for clinical trials (nli4ct) task is proposed by @xcite by building an efficient evaluation benchmark using a set of statements, explanations, and ctrs for breast cancer. this task is an extension of the previous year's shared task multi-evidence natural language inference for clinical trials. the purpose of the nli4ct task is to entail a statement based on one or multiple clinical reports. nli4ct is challenging because hypothesis verification sometimes requires integrating multiple pieces of data from the premise. in some instances, validating a hypothesis necessitates a comparison of two distinct premise ctrs. validating hypotheses based on each premise type demands varying levels of inference skills @xcite . this paper presents work done in the nli4ct to address these challenges owing to the complexity of the medical domain and text structure. the objective of this task is to develop a system capable of deducing conclusions or implications about various ctrs. the system consists of an ensemble of different experiments using different training approaches. the rest of the papers go as follows: section 3 discusses the proposed methods, section 4 shows experimental results, and section 5 concludes the paper. ## background the main goal of the task is to determine the validity of a claim (hypothesis) based on a single section from one or multiple clinical trial reports (ctrs) of breast cancer (premises). there are two possible inferential relations for each statement: entailment and contradiction. the dataset 1 used is provided by the task organizers and it is divided into two parts: the first part is derived from a compilation of ctrs, which is categorized into four sections.: a) eligibility criteria required for participation in the clinical trial; b) intervention detailing the treatment type, dosage, frequency, and duration; c) results showing participant numbers, outcome measures, units, and findings; and d) adverse effects observed in patients during the trial. the second part compromises the claim about the information contained in a single section, either in one or two ctrs and information about which ctrs are targeted and which section. the dataset consists of 1700 training samples and 200 validation samples. the dataset supplied has an equal 1 https://github.com/ai-systems/ task-2-semeval-2024 distribution of labels. ## system overview this section presents the various approaches used while developing the final models. this includes techniques for preprocessing and ingesting data. moreover, it includes the paradigms used for training as well as the experimental setup. deberta and bert-based models are fine-tuned using a weighted ensemble of refined iterations, as well as prompt-based fine-tuning @xcite for deberta final models. ## results in this section, the performance of the proposed models is reported based on the official metric during the dev-phase and test-phase. error analysis @xcite was conducted to identify the weaknesses of the proposed models. for the task, the official metric is based on the f1 score and the average faithfulness foot_0 and consistency foot_1 scores. ## conclusion the study tested different ways to prepare and load data, as well as more advanced nlm models. it came to the conclusion that the ensemble