BD - NLP at S em E val-2024 Task 2: Investigating Generative and Discriminative Models for Clinical Inference with Knowledge Augmentation (2024)

Abstract: Healthcare professionals rely on evidence from clinical trial records (CTRs) to devise treatment plans. However, the increasing quantity of CTRs poses challenges in efficiently assimilating the latest evidence to provide personalized evidence-based care. In this paper, we present our solution to the SemEval- 2024 Task 2 titled “Safe Biomedical Natural Language Inference for Clinical Trials”. Given a statement and one/two CTRs as inputs, the task is to determine whether or not the statement entails or contradicts the CTRs. We explore both generative and discriminative large language models (LLM) to investigate their performance for clinical inference. Moreover, we contrast the general-purpose LLMs with the ones specifically tailored for the clinical domain to study the potential advantage in mitigating distributional shifts. Furthermore, the benefit of augmenting additional knowledge within the prompt/statement is examined in this work. Our empirical study suggests that DeBERTa-lg, a discriminative general-purpose natural language inference model, obtains the highest F1 score of 0.77 on the test set, securing the fourth rank on the leaderboard. Intriguingly, the augmentation of knowledge yields subpar results across most cases.

## introduction clinical trials are conducted on human subjects to test the safety and effectiveness of the medicine prior to designing a new treatment, especially in evidence-based treatments @xcite . medical professionals prescribe and treat their patients based on clinical trial reports (ctr) in which the methodology and results of clinical trials are outlined. however, the increasing quantity of ctrs poses a challenge for healthcare professionals to manually assess all of them since this process is both time-consuming and labor-intensive @xcite . to tackle the aforementioned issue, recent advancements in natural language processing (nlp) encourage medical professionals to employ large language models (llms) to interpret and retrieve medical evidence from large quantities of ctrs @xcite . employing natural language inference (nli) and textual entailment in the clinical domain @xcite , professionals can formulate a prompt or statement, for example, "a minimum bodyweight of 55kg is required to participate in the primary trial." and input it along with the ctrs into an llm to determine whether the statement entails or contradicts the evidence from ctrs @xcite . additionally, llms can aid in retrieving evidence from the vast amount of ctrs. this technology has the potential to ensure a higher level of precision and efficiency in delivering personalized evidence-based care. while llms have demonstrated significant performance in numerous nlp tasks in recent years @xcite , it is still challenging for them to be deployed in the clinical domain due to their limitation in semantic and quantitative reasoning in language understanding. moreover, the distribution shift in the clinical domain makes it even more intricate, which requires extensive research in this field @xcite . to address the challenges, @xcite organizes the semeval-2024 task 2, titled "safe biomedical natural language inference for clinical trials", which aims to investigate the robustness of nli models when applied to clinical trials with cancer patients. this task seeks to develop an nli system to connect the new evidence and infer the knowledge from ctrs to find an inferential relation, namely either entailment or contradiction, between a clinical trial document and a statement/claim. in this study, we investigate the performance of discriminative and generative transformer-based llms in the realm of clinical inference. in addition, we explore the potential of clinical domain-specific llms and compare them with the general-purpose ones with the hypothesis that llms pre-trained on clinical data may exhibit superior performance. furthermore, we study the impact of augmenting knowledge on the semantic reasoning abilities of these llms. our extensive experiments demonstrate that our system, leveraging the discriminative, general-purpose deberta-lg nli model, achieves an f1 score of 0.77 and consistency score of 0.76 without employing knowledge augmentation on the test set and ranks fourth on the official leaderboard. ## system overview in this section we describe about our system for this task. ## experimental setup we keep the original data split (1700: 200: 5500) provided by the task organizer for training, validation, and testing sets respectively. huggingface transformers 1 library is used for tokenization and further finetuning. data preprocessing steps are mainly adapted from @xcite . for short text, the max sequence length for the tokenizer is set to 256 and 512 for long text for bi-olinkbert, clinicalbert and bioclinicalbert. for deberta-large, scifive and flan-t5 the max sequence length for the tokenizer is set to 512 and 1024 for short and long input text respectively. we train all language models for 20 epochs and an adamw @xcite optimizer is used for optimization with a default learning rate of 5e-6 for discriminative models and 5e-5 for 1 https://huggingface.co/docs/transformers generative models with weight ratio of 0.06 and weight decay of 0.01. the models are evaluated on the validation set after each epoch by using precision, recall, and f1 scores and saved best model based on least evaluation loss. we measure the performance of the models based on faithfulness and consistency proposed by @xcite . faithfulness measures the ability to predict the output based on the correct reason. therefore, if semantic reason change in future, models will be able to change its prediction accordingly. on one hand, consistency measures the ability to make same prediction for the semantically equal statements which ensures the semantic preserving in a model. ## results and discussion the experimental results are presented in among the models, only deberta-lg and flant5 are general-purpose models while the rest are tailored for the clinical domains by pretraining the models on domain-specific data. also, deberta-lg and scifive are the only task-specific nli models studied in this work. this is intriguing to observe that although deberta-lg is not pre-trained on clinical data, it yields the highest f1-score. thus, a model tailored to the task but not initially trained on domain-specific data may outperform a domain-specific model that lacks task specificity, demonstrating the importance of taskoriented adaptation rather than relying solely on domain-specific pre-training. this outcome contradicts our initial hypothesis that domain-specific pre-trained llms are necessary for superior performance. finally, the number of trainable parameters of the discriminative models is not found to be linked with model performance since discriminative bi-olinkbert, containing 340m parameters, performs either on par with or subpar than clinical-bert and bioclinicalbert with 110m parameters each. however, the generative scifive model, consisting of a larger number of trainable parameters than flant5, exhibits better performance in certain metrics e.g. faithfulness and consistency. ## conclusion in this paper, we describe our system for the semeval-2024 task 2, dealing with nli for clinical trials. leveraging deberta-lg, a discriminative pre-trained model tailored to the nli task, we achieve a consistency score of 0.76, securing the 4th position out of 31 participants. our exploration yields intriguing insights: both discriminative and generative models exhibit promise for this clinical inference task. in addition, we find that knowledge augmentation poses challenges for the model, possibly due to the higher dimensionality of the input space. moreover, task-specific but not domain-specific models are found to be better performing than domain-specific but not task-specific models. however, it is worthwhile to mention that all the models are fine-tuned with the same clinical data. interestingly, while the performance of the discriminative model is not affected by the number of parameters, it appears to influence the performance of generative models. as part of future work, we intend to explore the applicability of parameter-efficient techniques including adapter-tuning @xcite , lora @xcite , etc. by deploying them for the clinical inference task.