DKE -Research at S em E val-2024 Task 2: Incorporating Data Augmentation with Generative Models and Biomedical Knowledge to Enhance Inference Robustness (2024)

Abstract: Safe and reliable natural language inference is critical for extracting insights from clinical trial reports but poses challenges due to biases in large pre-trained language models. This paper presents a novel data augmentation technique to improve model robustness for biomedical natural language inference in clinical trials. By generating synthetic examples through semantic perturbations and domain-specific vocabulary replacement and adding a new task for numerical and quantitative reasoning, we introduce greater diversity and reduce shortcut learning. Our approach, combined with multi-task learning and the DeBERTa architecture, achieved significant performance gains on the NLI4CT 2024 benchmark compared to the original language models. Ablation studies validate the contribution of each augmentation method in improving robustness. Our best-performing model ranked 12th in terms of faithfulness and 8th in terms of consistency, respectively, out of the 32 participants.

## introduction in the domain of clinical trial analysis, researchers and practitioners are overwhelmed with an everexpanding corpus of clinical trial reports (ctrs). the current repository contains a vast number of documents and is rapidly growing, a trend that correlates with the increasing prevalence of crossnational, cross-ethnic, and multi-center clinical studies @xcite . this growth necessitates a scalable approach to evaluate and interpret the massive amount of data in these reports @xcite . recent advances in natural language processing (nlp) offer promising avenues for the automated analysis of ctrs. such analyses include medical evidence understanding @xcite , information retrieval @xcite , causal relationship identification @xcite , and the inference of underlying reasons for trial outcomes @xcite . integrating natural language inference (nli) with ctrs has the potential to revolutionize the large-scale, nlp-based examination of experimental medicine @xcite . despite the progress in nlp, the application of large language models to this task presents several challenges, including susceptibility to shortcut learning, hallucination, and biases stemming from word distribution patterns within the training data @xcite . to address these issues, we propose a novel method that leverages generative language models, such as gpt-3.5 foot_0 , and biomedical domain knowledge graphs to enhance data diversity. our approach introduces three types of data augmentation: numeric question-answering data generation, semantic perturbations, and domain-tailored lexical substitutions for the biomedical field. by combining these data augmentation techniques with multitask learning and the deberta @xcite ) architecture, we have achieved significant improvements in terms of faithfulness and consistency on the nli4ct 2024 dataset. this paper outlines our approach, elaborates on the design of the perturbations and the multi-task learning process, and demonstrates the efficacy of our method through rigorous evaluation. ## background in a crucial field like healthcare, where misinterpretations can have severe implications, nli models must present precise predictions and reliable interpretations. this highlights the importance of accurate and trustworthy reasoning in these nli models. semeval 2024 task 2 @xcite provides multi-sentence textual data consisting of patient case histories and medical reports. the objec-tive of this task is to predict the logical relationship between the ctr and a given statement, including entailment and contradiction. the evaluation emphasizes prediction accuracy as well as the robustness to the controlled interventions, helping increase healthcare practitioners' trust in the system's predictions. enhancing the robustness of nli models for healthcare can be strategically achieved using data augmentation techniques. synthetic data generation via techniques like conditional text generation can expand training data diversity and volume to improve model generalization capabilities @xcite @xcite . meanwhile, multi-task learning with auxiliary objectives related to logical reasoning and explanation generation can enhance faithful reasoning abilities @xcite . useful domain knowledge can be captured by training language models on domain-specific medical textual datasets @xcite . complementary data-centric methods can augment model architecture design to develop more capable, trustworthy, and clinical nli systems. ## system overview in this section, we describe the proposed system to tackle the nli problem and enhance the model's robustness against interventions spanning numerical, vocabulary, and semantic dimensions, as shown in figure 1 . ## results we conducted experiments with different-sized de-berta models, iteratively adding augmented data from three different interventions to the training set. as shown in from this iterative process, we can see that semantic perturbation with generative ai contributes mainly to the performance gain for both nli models. compared with this, vocabulary replacement in the biomedical domain has only a minor effect. this may suggest that vocabulary replacement in our work may be relatively less effective in this case because it only swaps out individual words, while semantic perturbation modifies the whole statement. hence, semantic perturbation provides more meaningful variations to augment the training data. while the augmented data improved the robustness to interventions, we noticed a slight performance drop in the control set. for example, the f1 score on the control set decreased by 3.16% for deberta-l and 0.48% for deberta-b after adding all the augmented data. this performance decline indicates there may have been a small tradeoff between improving robustness to interventions and maintaining strong performance on the original data. one of the reasons accounting for this could be that the generative ai may generate noisy or irrelevant data. for example, in numerical question answering data generation, if the original entailed statement discusses an assumption about a 50-year-old patient not mentioned in the ctr, the generative model may create an unrelated question about the patient's age that cannot be inferred from the given information. another example involves vocabulary replacement: we observed that there exist some cases where even two words having very similar embeddings in the biomedical domain please generate a contradictory statement based on the given statement, with a minor change: ## conclusion in this work, we proposed a data augmentation approach to enhance the robustness of natural language inference models for clinical trial report analysis. our method leverages generative ai and biomedical knowledge graphs to augment training data along three dimensions: numerical reasoning, semantic perturbations, and domain-tailored lexical substitutions. experiments on the nli4ct 2024 dataset demonstrate that our approach effectively improves model faithfulness and consistency against controlled interventions, with significant gains against the deberta baselines. however, we observed a slight performance drop on the unaltered test set, indicating a trade-off between robustness to perturbations and maintaining strong performance on original data. future work will focus on: 1) generating higher-quality augmented examples using numerical questionanswering data generation to minimize or avoid performance drop; 2) validating the perturbed samples to help remove noisy or irrelevant examples @xcite ; 3) incorporating external structured knowledge via pre-training on knowledge graphs and not just lexical substitution, which can provide more contextual domain information. postgraduate research scholarship (pgrs) (contract number pgrs-20-06-013) at xi'an jiaotong-liverpool university. additionally, this research has received partial funding from the jiangsu science and technology programme (contract number bk20221260) and the research development fund (contract number rdf-22-01-132) at xi'an jiaotong-liverpool university.