NYCU - NLP at S em E val-2024 Task 2: Aggregating Large Language Models in Biomedical Natural Language Inference for Clinical Trials (2024)

Abstract: This study describes the model design of the NYCU-NLP system for the SemEval-2024 Task 2 that focuses on natural language inference for clinical trials. We aggregate several large language models to determine the inference relation (i.e., entailment or contradiction) between clinical trial reports and statements that may be manipulated with designed interventions to investigate the faithfulness and consistency of the developed models. First, we use ChatGPT v3.5 to augment original statements in training data and then fine-tune the SOLAR model with all augmented data. During the testing inference phase, we fine-tune the OpenChat model to reduce the influence of interventions and fed a cleaned statement into the fine-tuned SOLAR model for label prediction. Our submission produced a faithfulness score of 0.9236, ranking second of 32 participating teams, and ranked first for consistency with a score of 0.8092.

## introduction biomedical natural language inference (nli) seeks to determine whether a proposed statement is entailment, contradiction, or neutral according to a given clinical trial. the mediqa-2019 shared task @xcite covered an nli subtask in the medical domain, including clinical sentences from the mimic-iii database @xcite . in this shared task, most systems were built on the bert model @xcite and mt-dnn @xcite . the bert-bilstm-attention model @xcite was proposed for medical text inference. the doubletransfer model @xcite was presented to use a multi-source transfer learning approach to acquire knowledge from mt-dnn and sci-bert @xcite . in addition, since the evaluation data is sourced from the clinical domain, variations of bert such as biobert @xcite and clinicalbert @xcite were used frequently. semeval-2023 task 7 @xcite ) (called nli4ct) focused on multi-evidence natural language inference for clinical trial reports (ctr) @xcite . participants should determine the inference relation (i.e., entailment or contradiction) between ctrstatements in the nli subtask. the sentence-level and token-level encodings were exploited in a multi-granularity inference network (mgnet) @xcite . the deberta-v3 model @xcite was fine-tuned on the prompted input sentences to discriminate the inference relation between the statement and clinical trials @xcite . the biolinkbert transformer @xcite was used with a soft voting ensemble mechanism to enhance the nli performance @xcite . the flan-t5 model @xcite was fine-tuned with instructions to explore its capabilities for multievidence nli @xcite . following the success of the nli4ct-2023 task, semeval-2024 task 2 @xcite regrounds this task in interventional and causal analyses of nli models @xcite , with a contrast set containing the designed interventions and expected labels to investigate the faithfulness and consistency of the developed models. this task is based on the same collection of breast cancer ctrs @xcite . the statements in the training set are identical to those in the previous task, but perform a variety of interventions to statements on the development and test sets, making claims about a single ctr or comparing nycu-nlp at semeval-2024 task 2: aggregating large language models in biomedical natural language inference for clinical trials two ctrs while either preserving or inversing the entailment relations. for the nli4ct-2024 task, given a statement with/without interventions, the participating system should determine the inference relation as either entailment or contradiction. this paper describes the nycu-nlp (national yang ming chiao tung university, natural language processing lab) system for the nli4ct-2024 task. given the promising results obtained by large language models (llm) for various nlp tasks, we aggregate several llms in biomedical nli for clinical trials. we use chatgpt (openai, 2023) to augment original statements and then fine-tune the solar model @xcite with instructions designed for the nli task. since a statement may be manipulated during testing inference phase, we first fine-tune the openchat model @xcite to reduce the influence of interventions. finally, a cleaned statement along with ctrs is fed into the fine-tuned solar model for label prediction (i.e., entailment or contradiction). evaluation results show that our proposed nycu-nlp system had a faithfulness score of 0.9236, ranking second among 32 participating teams, and ranked first for consistency with a score of 0.8092. the rest of this paper is organized as follows. section 2 describes the nycu-nlp system for the nli4ct-2024 task. section 3 presents the results and performance comparisons. conclusions are finally drawn in section 4. ## the nycu-nlp system fig. 1 shows our nycu-nlp system architecture for the nli4ct-2024 task. our system is composed of four main parts: 1) chatgpt (openai, 2023) for data augmentation; 2) instruction tuning on solar @xcite ; 3) openchat @xcite for intervention reduction; and 4) fine-tuned llm for label prediction. ## conclusions this study describes the nycu-nlp submission for the semeval-2024 nli4ct task, including system design, implementation and evaluation. we aggregated several llms to determine the inference relation between ctrs and statements that may be manipulated with designed interventions to investigate the faithfulness and consistency of the developed models. our system obtained a faithfulness score of 0.9236, ranking second among all 32 participating teams, and ranked first for consistency with a score of 0.8092.