SEME at S em E val-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials (2024)

Abstract: This paper describes our submission to Task 2 of SemEval-2024: Safe Biomedical Natural Language Inference for Clinical Trials. The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) consists of a Textual Entailment (TE) task focused on the evaluation of the consistency and faithfulness of Natural Language Inference (NLI) models applied to Clinical Trial Reports (CTR). We test 2 distinct approaches, one based on finetuning and ensembling Masked Language Models and the other based on prompting Large Language Models using templates, in particular, using Chain-Of-Thought and Contrastive Chain-Of-Thought. Prompting Flan-T5-large in a 2-shot setting leads to our best system that achieves 0.57 F1 score, 0.64 Faithfulness, and 0.56 Consistency.

## introduction the digitization of medical documents allows the development of tools using various nlp techniques. in the case of clinical trial reports (ctr), these tools can facilitate recruiting patients to participate in a trial or help researchers keep up to date with the literature. natural language inference (nli) is particularly useful in detecting the relationship between a ctr and a statement. for instance, it can be used for patient-trial matching. task 2 of semeval 2024 defines a textual entailment (te) task applied to english breast cancer ctrs. a submitted system must perform a binary classification based on a ctr and a given statement, using the labels entailment or contradiction. in addition to the traditional f1-measure for textual entailment, the submitted systems are evaluated on 2 strong metrics: faithfulness and consistency. in this paper, we first introduce the task and some related work in sec. 2. sec. 3 describes our proposed approaches, while sec. 4 gives further details about the experimental setup. sec. 5 presents the results and comparative analysis of methods, and sec. 6 sums up our work done and provides ideas for future work. ## system overview to address the nli4ct task, we tested 2 main approaches: the first uses pretrained masked language models (mlm), and the second uses generative large language models. we wanted to compare the ability of these two kinds of architectures to solve the same task, in particular in terms of consistency and faithfulness. ## conclusion and future work this paper describes the two systems proposed by the seme team for the semeval 2024 task 2 nli4ct. our first approach is based on the finetuning and ensembling of masked language models, using only the challenge's data. our second approach consists of a pipeline to prompt large language models, using prompt engineering techniques, such as chain-of-thought and contrastive chain-of-thought, in zero-shot, 1-shot, and 2-shot manners. our two best-reported results are 0.57 f1 score, 0.64 faithfulness, and 0.56 consistency, with prompting flan-t5-large in a 2-shot manner, ranking 27th out of 32 submissions for f1, 18th for faithfulness and 25th for consistency. we obtain the same scores for the mlm system using an ensemble composed of a finetuned nli-roberta + clinical-longformer + clinicalbert + the predictions of flan-t5-large, that is 0.57 for f1 score, 0.64 for faithfulness, and 0.56 for consistency. some future work could include the continuation of the masked language models pretraining on unlabeled clinical trials, before performing a similar finetuning as presented in the paper. we could also apply this approach to medical large language models like meditron @xcite , by performing instruction-tuning using clinically oriented instructions and then prompting the resulting model on the task data. another possible approach, similar to @xcite , would be to incorporate domain ontologies (like umls) into the finetuning of masked language models to provide definitions and supplementary knowledge.