i ML at S em E val-2024 Task 2: Safe Biomedical Natural Language Interference for Clinical Trials with LLM Based Ensemble Inferencing (2024)

Abstract: We engaged in the shared task 2 at SenEval-2024, employing a diverse set of solutions with a particular emphasis on leveraging a Large Language Model (LLM) based zero-shot inference approach to address the challenge.

## introduction clinical nli is a specialized application of natural language processing (nlp) that focuses on understanding and inferring information from text within the healthcare domain. it involves analyzing and drawing conclusions from clinical narratives, such as electronic health records (ehrs), doctor's notes, medical transcripts, clinical trials and other forms of medical documentation @xcite . the goal of clinical nli is to determine the logical relationship between premises and hypotheses (conclusions) in clinical text. by inferring information from clinical text, nli can assist healthcare providers in making informed decisions by providing evidence-based recommendations and alerts. in addition, clinical nli can be used to identify patient cohorts for clinical trials or research studies by inferring patient eligibility based on inclusion and exclusion criteria mentioned in clinical records. applications of clinical nli are not limited to the ones mentioned and there are lots of other usages in which clinical nli can be useful @xcite . nli for clinical trials faces unique challenges due to the complexity of medical language, the need for domain-specific knowledge, and the sensitivity and privacy concerns associated with health data. however, advancements in nlp and specifically large language models (llms) are continuously improving the accuracy and applicability of clinical nli, making it an increasingly valuable tool in the healthcare industry. to foster collaboration and dissemination of novel insights within this field, semeval 2024 @xcite has established a shared task exclusively devoted to clinical nli. a publicly accessible dataset, annotated by humans, has been made available to facilitate the comparison of solutions proposed by different researchers. to address the challenge, we developed an ensemble-oriented solution that combines various large language models (llms) based models within the framework of prompting and fine-tuned classification. our primary goals were to first understand the comparative performance of generative models versus classification models. subsequently, we explored whether the use of automatic summarization models to condense the premises would influence the efficacy of both classifiers and generative models. ultimately, our approach sought to facilitate synergistic interactions among the different models, leveraging their respective strengths to mitigate individual inference limitations. nevertheless, despite conducting a variety of experiments that involved combining summarization, fine-tuning classifiers, prompting, and more, the results demonstrated a clear superiority of generative models in comparison to the others, even when used independently. the remainder of this paper is organized as follows: section 2 provides a brief review of related work. the proposed model and its constituent modules are detailed in section 3. sections 4 and 5 discuss the experiments conducted and the corresponding results. finally, we conclude the paper in section 6. ## past work recent literature underscores the need for sophisticated models that can accurately capture the semantics of clinical narratives and support reasoning in line with medical knowledge. @xcite , introduced a shared task on nli for clinical trials (nli4ct), providing a dataset of annotated clinical trials and inviting researchers to develop models to tackle the associated challenges. the shared task comprises two sub-tasks: textual entailment and evidence retrieval, each designed to advance the state of nli systems within the clinical domain. @xcite , took part in the nli4ct-2023 challenge, proposing a model that utilizes both sentence-level and token-level encoding to address the task at hand. furthermore, they enhanced the model's overall performance by employing general (t5-based model) and domain-specific (sci-five) pre-trained llms. kanakarajan and sankarasubbu (2023), conducted an evaluation of several instruction-tuned large language models (llms) in a zero-shot setting and fine-tuned the best-performing instructiontuned model (t5 family models). their findings suggest that instruction-tuned models yield better results for datasets with limited training samples. additionally, they explored the impact of various prompts on the overall performance of the model. @xcite and @xcite , both created a model based on an ensemble approach that combines various fine-tuned iterations of biomedical llms. these models are designed to extract evidence from clinical trial report premises to support textual entailment in specific statements. @xcite , developed a system that utilizes prompts created by humans to gather information from statements, section titles, and clinical trials. they then fine-tune pretrained language models on these prompted sentences, training the models to identify the inferential connections between the statements and the clinical trials. @xcite , characterized the nli task as a form of text pair classification and utilized the gpt-3 model to classify samples within the framework of few-shot prompt-ing. this approach takes advantage of the semantic similarity between text samples and the examples provided for in-context learning. dias et al. ( 2023 ), employed supervised contrastive learning to enhance the sentence pair representations in the biomed roberta model. they then fine-tuned a linear classifier built upon these improved representations to identify evidence and execute textual entailment classification for sentence pairs. @xcite , introduced a two-tiered system to address the sub-tasks of nli4ct-2023. initially, the system employs a bert-based classifier, supplemented by contextual data augmentation, to categorize evidence-statement pairs as relevant or irrelevant. subsequently, leveraging the relevant segments of the clinical trial identified in the first stage, the system applies another bert-based classifier to ascertain whether the relationship between the elements is one of entailment or contradiction. @xcite , illustrated that a transformer model pre-trained on biomedical data for the task of entailment relation in nli4ct-2023 does not automatically outperform traditional approaches like cnns. nonetheless, their model exceeded the baseline system's performance and provided meaningful directions for future research on how the model's architecture can be developed further. ## proposed model in tackling the nli4ct task, our approach involved the construction of an ensemble model that integrates the judgments of multiple distinct decisionmakers. these decision-makers differ concerning the nature of input data they process, the foundational models they employ, and the methodologies they adopt for label determination. figure 1 provides a comprehensive illustration of the proposed solution. components of the ensemble pool were developed within the frameworks of classification or prompting, utilizing llms. for classification tasks, scifive @xcite was selected as the base model due to its exemplary performance in the nli4ct-2023 task. to enhance the models' ability to assimilate information from the input data, we employed both extractive and abstractive summarization techniques. the abstractive summarization was conducted using the t5-large model @xcite to condense the premises. for extractive summarization, the premises were initially segmented into individual sentences, after which those exhibiting lower semantic similarity to the hypothesis were excluded. the pre-trained scifive model ingests the text summarized by t5 to generate the initial component of the ensemble pool. subsequently, this model undergoes fine-tuning through two distinct methodologies utilizing the summarized data: comprehensive fine-tuning and parameter-efficient finetuning, the latter of which is facilitated by employing lora (hu et al., 2021) to produce subsequent members of the ensemble pool. the remaining decision-makers within the ensemble are derived by prompting generative llm 1 in a zero-shot inference context, utilizing both the original input data and variously summarized inputs. the specific prompt employed for the model is delineated in listing 1. ultimately, the final decision for the test samples were made using a weighted majority voting approach. the performance of models on prac-tice_test set were used for the combination process. ## experiments we have conducted our experiments utilizing the dataset provided by the task's organizers that is explained in section 4.1. for the models based on prompting, we utilized only the test and prac-tice_test datasets, whereas the training data was employed exclusively for fine-tuning the classificationbased models. beyond experimenting with models within our ensemble framework, we also explored the integration of results from fine-tuned classification models as a form of external knowledge within the context of prompting. the efficacy of all models is evaluated using three metrics: macro f1 score, faithfulness, and consistency, each of which is briefly described in section 4.2. ## results the performance result of individual models within the ensemble, as applied on both practice_test and test datasets, are illustrated in the proposed model exhibits faithfulness and consistency scores of 28% and 52%, respectively, suggesting a necessity for more robust models to effectively manage clinical trials involving diverse data types. the findings reveal that the proposed overall model performs similarly to the generative model in the prompting context. this similarity underscores the considerable potential of generative llms. these models can achieve better performance when instruction tuning is applied with domain-specific data. additionally, using classification results as external knowledge for the prompting model showed minimal impact. moreover, the use of extractive summarization yielded the lowest results, aligning with our expectations. this approach, which focuses on the similarity between individual sentences and the statement, can lead to a loss of comprehension of the entirety of the premises. ## conclusion in conclusion, our participation in nli4ct-2024 involved proposing an ensemble approach that incorporated multiple decision-makers, with two large language models (llms) serving as foundational models. we explored various data preparation techniques, including abstractive summarization and similarity-based sentence filtering, for use in both prompting and classification contexts. the comparable performance of the prompt-based model to the overall ensemble model, coupled with its significant outperformance of the classification models, underscores the substantial potential of pretrained generative foundation models in solving similar problems. we posit that the application of instruction tuning and the incorporation of domainspecific data could markedly enhance the results.