T ü D uo at S em E val-2024 Task 2: Flan-T5 and Data Augmentation for Biomedical NLI (2024)

Abstract: This paper explores using data augmentation with smaller language models under 3 billion parameters for the SemEval-2024 Task 2 on Biomedical Natural Language Inference for Clinical Trials. We fine-tune models from the Flan-T5 family with and without using augmented data automatically generated by GPT-3.5-Turbo and find that data augmentation through techniques like synonym replacement, syntactic changes, adding random facts, and meaning reversion improves model faithfulness (ability to change predictions for semantically different inputs) and consistency (ability to give same predictions for semantic preserving changes). However, data augmentation tends to decrease performance on the original dataset distribution, as measured by F1 score. Our best system is the Flan-T5 XL model fine-tuned on the original training data combined with over 6,000 augmented examples. The system ranks in the top 10 for all three metrics.

## introduction in the recent years, the rapid and triumphant advance of large language models (llms) has affected virtually every area of nlp, biomedical nlp included. we aim to prove that biomedical nlp can still benefit from smaller models of no more than three billion parameters. first, as the saying goes, "you must not use a steam hammer to crack a nut, if a nutcracker would do". in other words, while llms' performance is unmatched in complex applications, smaller models may be perfectly sufficient for simpler tasks, such as text classification or natural language inference. second, being pre-trained on extremely large corpora of unlabelled data, modern llms have been shown to exhibit dataset-related bias @xcite . in fields with a high error cost, pretraining and fine-tuning models on smaller, care-fully curated, high-quality datasets is safer and more predictable than using black-box giant llms in a zero-shot or few-shot setting. finally, as of now, best-performing state-of-the-art llms are either largely unavailable to the end-user due to computational constraints (for open-source models) or cost-inefficient (for proprietary models with access via api). the nli4ct-2024 shared task @xcite consists in building a system for natural language inference (nli) based on a collection of breast cancer clinical trial reports (ctrs) in english. the task's main challenge is the complex and heterogeneous nature of the data. for each datapoint, the premise comes from one of the four sections of a ctr -intervention, eligibility, results, or adverse events. naturally, the sections are different from each other in terms of the mean length, the proportion of numerical data present, and the level of world knowledge required for drawing conclusions. compared to the previous year's iteration of the task @xcite , this year's challenge calls for a system robust to alterations in the data. apart from f1 measure, two new metrics are used to evaluate the model performance: faithfulness, "measuring the ability of a model to correctly change its predictions when exposed to a semantic-altering intervention", and consistency, "measuring the ability of a system to predict the same label for original statements and contrast statements for semantic preserving interventions". according to the last year participants' reports, various augmentation techniques have not led to significant performance improvement in terms of f1 and the top-3 best-performing systems did not use data augmentation at all @xcite . however, given the new metrics that are used in this year's evaluation, it seems reasonable to continue exploring the effect that various kinds of augmentation have on f1, faithfulness, and consistency at the same time. in this paper, we fine-tune mod-1 737 els of flan-t5 family with and without the use of augmented data automatically generated using gpt-3.5-turbo. we find that using various kinds of additional data leads to an increase in model's faithfulness and consistency, but a decrease in f1. our best system is flan-t5 xl, fine-tuned on 1900 original train and development instances and 6650 automatically generated ones. the system ranks 7th for consistency, 9th for faithfulness, and 10th for f1. ## related work language models and biomedical nlp there has been a surge of llms fine-tuned on biomedical data, from relatively small -7 billion parameter chatdoctor @xcite , medalpaca @xcite , pmc-llama @xcite ; 6 billion parameter doctorglm @xcite and ophglm @xcite to extremely large ones -540b med-palm (singhal et al., 2023), 175b codex-med (liévin et al., 2022), 80b med-flamingo @xcite )which were reported to break state-of-the-art results on a number of biomedical nlp tasks. moreover, without any fine-tuning on biomedical data, gpt-4 was reported to have passed every step of the us-medical licensing exam @xcite . however, researchers argue that smaller language models, such as t5 base and t5 large, still outperform gigantic all-purpose models when fine-tuned for a specific task @xcite . model robustness in nli tasks many nli models suffer from bias related to superficial correlations between input text features and labels in the training dataset, which leads to a drop in performance on datasets where these correlations do not hold @xcite . among those are hypothesis only bias, where models rely mostly on the hypothesis without taking premise and premisehypothesis relations into account @xcite , and word-overlap bias, where models rely on the presence of shared words or phrases in premise and hypothesis @xcite . various techniques may be used to mitigate this bias, such as adversarial training @xcite and data augmentation with predicate-argument structures @xcite and syntactic transformations @xcite . ## experimental setup in our experiments, we aim to test whether language models of relatively small size, under three billion parameters, can achieve decent performance on a task with a simple objective -as the model chooses between only two options, entailment and contradiction -and complex data -as dealing with clinical test reports requires complex reasoning and understanding of numerical data. as a starting point for the experiments, we have chosen flan-t5. ## conclusion in this paper, we explore the impact of data augmentation on model performance and robustness. specifically, we focus on leveraging advanced language models like gpt-3.5-turbo to expand the training set for fine-tuning smaller models such as flan-t5 xl. our experiments involve various prompts to generate new ctr-hypothesis pairs. enriching the training set with new examples that underwent semantic-preserving changes, such as synonym replacement, change in word or clause order, and random true fact addition, improves the model's consistency. adding augmented examples that underwent semantic-altering changes, such as meaning reversion, improves the model's faithfulness as well as consistency. however, all kinds of augmentation except for synonym replacement lead to a decrease in model performance in terms of f1 on the original unaltered dataset. the model selected for the final submission is flan-t5 xl fine-tuned on augmented development and training set. it features higher robustness but lower base performance than flan-t5 xl fine-tuned on original data only, with faithfulness of 0.841 (+0.061), consistency of 0.752 (+0.085), and f1 of 0.76 (-0.019). b prompts used to obtain augmented data from gpt-3.5-turbo figure 4: prompt used to generate a synonym-based paraphrased version of hypothesis. 8