Lisbon Computational Linguists at S em E val-2024 Task 2: Using a Mistral-7 B Model and Data Augmentation (2024)

Abstract: ABSTRACT: This paper describes our approach to the SemEval-2024 safe biomedical Natural Language Inference for Clinical Trials (NLI4CT) task, which concerns classifying statements about Clinical Trial Reports (CTRs). We explored the capabilities of Mistral-7B, a generalistic open-source Large Language Model (LLM). We developed a prompt for the NLI4CT task, and fine-tuning a quantized version of the model using a slightly augmented version of the training dataset. The experimental results show that this approach can produce notable results in terms of the macro F1-score, while having limitations in terms of faithfulness and consistency. All the developed code is publicly available on a GitHub repository.

## introduction large language models (llms) currently achieve state-of-the-art performance on different natural language processing (nlp) tasks, including in the assessment of textual entailment relations. however, these models are heavily susceptible to shortcut learning @xcite , factual inconsistency @xcite , and performance degradation when exposed to data from specialized domains, such as in the case of medical data. noting the aforementioned challenges, task 2 at semeval-2024 addressed a safe biomedical natural language inference for clinical trials (nli4ct) task @xcite , which concerns classifying statements about clinical trial reports (ctrs). nli4ct investigated the accuracy, faithfulness, and consistency of the reasoning performed by llms in this particular medical task. the goal of the task is to determine whether there is an entailment relation or a contradiction relation between ctrs and statements, making some type of claim about a single ctr or a pair of ctrs. given the 1 https://github.com/araag2/semeval2024-task2 specific focus on assessing model faithfulness and consistency (i.e., the ability to make correct predictions for the correct reasons), the dataset associated to the task involved the systematic application of controlled interventions, either preserving or inverting the entailment relations originally generated by clinical domain experts. this way, the task investigated the robustness of nli models in their representation of the semantic phenomena necessary for complex inference in clinical settings. our approach to the nli4ct task involved the use of open-source llms, with good results in general purpose benchmarks foot_0 and capable of following task instructions. we opted for mistral-7b-instruct-v0.2 foot_1 @xcite , quantizing the model to 4-bits and simultaneously using low-rank adaptation (lora) @xcite to fine-tune the model to the nli4ct task, using a slightly augmented version of the training dataset that features a mixture of manually curated and synthetic statements. our overall best submission to the task achieved a macro f1-score of 0.80 (1st place on the leaderboard), a consistency score of 0.72 (15th), and a faithfulness score of 0.83 (11th). our method excels in classification accuracy, but fails at being robust to perturbations on the statements, i.e. predicting the same label on contradictory examples and different labels on paraphrased examples. ## background the nli4ct task concerns inferring if statements can be entailed by a given textual context, with each statement referring to one or two ctrs. these ctrs belong to a corpus consisting of 1000 different trials concerning breast cancer, extracted from set # samples single -compari. entail. -contr. the united states national library of medicine foot_2 . these trial reports are exclusively written in the english language and average 817 words in length. ctrs are divided into four sections: eligibility criteria, describing a set of conditions to allow or exclude patients in the trial; interventions, detailing all information about the conducted treatments; results, outlining outcomes and experimental results gathered through the trial; and adverse events, reporting patient observations concerning symptoms and physiological signs. an instance of the nli4ct dataset contains either one or two ctrs (i.e., cases denoted as single or comparison, respectively), a statement, a section marker, and a ground-truth label (i.e., entailment or contradiction). an example is shown next. ## system overview we now describe our general approach to the semeval-2024 nli4ct task. ## experimental setup making official submissions to the task leaderboard required the participants to submit full runs of the test set, outputting a label for each of its instances. we obtained the labels for each instance by following the procedure described in subsection 3.3. the task uses the following evaluation measures: macro f1-score, i.e. the arithmetic mean of precision and recall, averaged over the two classes; faithfulness, i.e. a measure created to assess the capacity of model to arrive at the correct prediction for the correct reason, calculated by measuring the ability of model to change its prediction label after semantically altering a statement; and consistency, which completes faithfulness by measuring the ability of a model in outputting the same prediction for semantically equivalent statements foot_4 . we evaluated our runs using the official metrics obtained from the leaderboard. following the training procedure described in section 3.5, we tested different combinations of training data (as described in section 3.4). the full set of hyper-parameters associated to our best run can be found in appendix a.2. all the different runs used python libraries and packages that can be found in our github repository foot_5 . ## results and discussion we tested the llm without any training (i.e., zero-shot results), and fine-tuning with the base and augment datasets, all with our best instruction format. as expected, there is a significant difference in performance towards fine-tuned models. overall, a mixture between manually curated sam-ples and synthetically generated ones performed best (train_manual-synthetic, as described on section 4), outperforming the best run that did not use any data augmentation. if more instances could have been manually curated, specifically targeting adversarial re-writes of the same statements, we hypothesize that results could be improved further. even though train_full-synthetic corresponds to the largest training set (i.e., featuring 11011 samples), the lack of quality in the automatically generated statements potentially impaired the f1-score while also limiting consistency and faithfulness. the run trained with train_manual-synthetic corresponds to our best overall result. when compared to the top submissions, we can see that our f1-score corresponds to a tie with another system in the 1st place of the leaderboard. however, results are much worse in the other two metrics, with significant differences between the top systems (i.e., with scores of 0.95 in faithfulness and 0.81 in consistency) and our submission. in the post-task phase of the competition, groundtruth labels for all examples were released, specifying which type of interventions were made in each instance. therefore, we are now able to analyse our system's errors (see comparing all errors across the different instance types, the average error rate is much higher on intervention errors (26.7%) against base statement errors (19.8%), which is to be expected as our training sets had fewer examples of this type. specifically, we can see that label preserving interventions (88.6%) have a high percentage of errors. our system can identify instances which suffered contradictory interventions with an error rate of 19.5% for textual changes, and 8.7% for numerical changes. instances that were perturbed with paraphrasing cause an error rate of 22.9%, while numerical paraphrasing errors correspond to 25.9%. at the worst end we have the samples with text appended to the end, which causes an error rate of 40.6%. note that we did did not explicitly augment the training instances by appending text to the existing statements, and the absence of examples like this was very costly in terms of the final results. ## conclusions adapting evaluation methodologies to better inform the safe deployment of llms in critical domains is an urgent necessity. the nli4ct task at semeval-2024 addressed this specific concern, and through our participation we improved our understanding on how llms can be fine-tuned to encompass robust results on clinical natural language inference. overall, our results show that the simple fine-tuning of an open-source llm to this specific task can achieve notable results in terms of the macro-averaged f1, although with limitations in terms of faithfulness and consistency. augmenting the data with high-quality curated examples can improve result quality, although augmenting the training set with synthetic examples requires careful quality control. for future work we would like to explore the following ideas: