E dinburgh Clinical NLP at S em E val-2024 Task 2: Fine-tune your model unless you have access to GPT -4 (2024)

Abstract: The NLI4CT task assesses Natural Language Inference systems in predicting whether hypotheses entail or contradict evidence from Clinical Trial Reports. In this study, we evaluate various Large Language Models (LLMs) with multiple strategies, including Chain-of-Thought, In-Context Learning, and Parameter-Efficient Fine-Tuning (PEFT). We propose a PEFT method to improve the consistency of LLMs by merging adapters that were fine-tuned separately using triplet and language modelling objectives. We found that merging the two PEFT adapters improves the F1 score (+0.0346) and consistency (+0.152) of the LLMs. However, our novel methods did not produce more accurate results than GPT-4 in terms of faithfulness and consistency. Averaging the three metrics, GPT-4 ranks joint-first in the competition with 0.8328. Finally, our contamination analysis with GPT-4 indicates that there was no test data leakage. Our code is available at https://github.com/EdinburghClinicalNLP/semeval_nli4ct.

## introduction extracting insights from clinical trial reports (ctrs) is vital for advancing personalised medicine, yet manual analysis of these vast datasets is impractical. the natural language inference for clinical trial data (nli4ct) task @xcite 2 addresses this challenge by evaluating natural language inference (nli) systems' ability to understand and reason within this domain. in this study, we evaluate various llms, such as llama2 @xcite , mistral @xcite , mistrallite @xcite , and @xcite . we employed prompting strategies like in-context learning (icl) and chain-of-thought (cot) to improve their accuracy. we also proposed a parameter-efficient fine-tuning (peft) method that merges independently fine-tuned adapters trained with distinct objectives, namely a triplet loss and a language modelling (lm) loss, to improve the consistency of the llms. our findings reveal that our novel peft method improves the f1 and consistency scores of the llms. however, gpt-4 produces more accurate results than all of the models we considered, coleading the competition leaderboard. although gpt-4 places fifth in the f1 score, its high faithfulness and consistency scores highlight its potential for a reliable prediction in the clinical domain. lastly, we conduct a contamination analysis of gpt-4 to check whether instances of the nli4ct dataset were included in gpt-4's pre-training data. ## system overview we experimented with two strategies. the first involved no fine-tuning, aiming to comprehend llms' inherent ability to solve clinical tasks. the second employed our proposed peft method to improve the consistency of the model. both systems ingest ctr-hypothesis pairs, predicting the correct label one token at a time from left to right. ## results the results shown in rq 1: can zero-shot llms perform well? in a zero-shot setting, mistrallite-7b showed zero performance across all metrics due to it outputting llm (2) triplet fine-tuning embed triplet lora 🔥 (1) language modelling fine-tuning llm lm lora 🔥 ❄ generate entailment contradiction or ❄ ✅ hypo 1 ❌ hypo 2 ⚓ evidence (3) merge lora adapters (4) inference generate entailment contradiction or llm merged lora ❄ ❄ hypothesis 1 (entailment) only 2 patients in the primary trial did not have recurrence-free survival ## conclusion this study assesses the performance of various llms, employing diverse strategies such as cot, icl, and peft. we propose a peft method, merging independent adapters fine-tuned separately using triplet and lm losses. our proposed peft method improves the f1 and consistency scores but reduces faithfulness -our best fine-tuned model, mistrallite-7b + lm lora + triplet lora, achieved an average score of 0.7862. however, it does not outperform gpt-4 in terms of faithfulness and consistency: gpt-4 ranks jointfirst in the competition with an average score of 0.8328. a contamination analysis on gpt-4 revealed no nli4ct test data leakage, indicated by a low extractable match score (0.033), and showcased its ability to identify keywords from ctrs with a relatively high partial match score (0.322).