LMU - B io NLP at S em E val-2024 Task 2: Large Diverse Ensembles for Robust Clinical NLI (2024)

Abstract: In this paper, we describe our submission for the NLI4CT 2024 shared task on robust Natural Language Inference over clinical trial reports. Our system is an ensemble of nine diverse models which we aggregate via majority voting. The models use a large spectrum of different approaches ranging from a straightforward Convolutional Neural Network over fine-tuned Large Language Models to few-shot-prompted language models using chain-of-thought reasoning.Surprisingly, we find that some individual ensemble members are not only more accurate than the final ensemble model but also more robust.

## introduction in this paper, we describe our submission to semeval-2024 task 2: safe biomedical natural language inference for clinical trials (nli4ct 2024) @xcite . in nli4ct 2024, every model receives as input one or two clinical trial reports (ctrs) describing a breast cancer study. further the model gets a hypothesis which makes a claim about the study and the section where the relevant information about the claim can be found in the ctr. following a classical nli setup @xcite , the task of the model is to decide whether the hypothesis is logically entailed by the ctr or whether it contradicts the information in the ctr. nli4ct 2024 is a continuation of a similar task that was held in 2023 @xcite and uses the same training and validation datasets. in contrast to the previous edition, nli4ct 2024 focuses on the robustness of the submitted models. specifically, it evaluates whether a model is consistent in its predictions and whether it predicts the correct label for the right reasons via targeted modifications of the test data; see section 3 and @xcite for more details. * equal contribution. the order of the first-authors was chosen randomly. we approach this task by building a large ensemble of diverse models. our hypothesis is that ensembling a large variety of strong and weak models would improve robustness. for that we build ensembles of up to 25 models derived from 9 different approaches via different ensembling strategies. these approaches were implemented as part of a master's course on biomedical natural language processing at lmu munich. teams of two to three students chose a broad initial approach such as convolutional neural networks @xcite or data-centric machine learning @xcite . then, they developed multiple models in the confines of the chosen approach while collaborating occasionally with other groups. finally, evaluated all resulting models individually and as large ensembles on the test set. we find that ensembling generally improves robustness but that some individual approaches achieved even higher performance. ## evaluation protocol nli4ct 2024 uses three metrics to evaluate approaches. f1 score measured on the test set of nli4ct 2023, consistency and faithfulness. consistency measures whether the model always produces the same label for a set of instances that share the same meaning and thus the same gold label. formally, we evaluate all approaches on the hidden test set of nli4ct 2024. we chose this approach even though frequent test set evaluation has severe downsides (van der goot, 2021) because consistency and faithfulness could not be computed on the development set. ## results dev-f1 as model selection criterion unsurprisingly, using only dev-f1 as the criterion for model selection and hyperparameter tuning is not sufficient for maximizing the average performance over test-f1 consistency, and faithfulness. in five out of nine approaches, the model that achieves the best dev-f1 score does not achieve the best average score. this also has consequences for our bestperforming ensembling approach best because it implies that in five out of nine cases we include a suboptimal model in our ensemble. this suggests that using a development set that allows for measuring consistency and faithfulness for model selection, hyperparameter tuning or ensemble construction could improve these properties at test time. ## conclusion this paper describes our contribution to the semeval-2024 nli4ct shared task on robust nli for clinical trial reports. we investigate whether a large diverse ensemble can improve robustness. our results largely confirm this hypothesis, but we find that some individual approaches perform even better and more robust than our best ensemble. in this work, we investigated only ensembling based on voting procedures and completely disregarded the confidences of the individual models. further, we did not use more sophisticated approaches such as stacking. finally, we used datacentric approaches only to augment the training data of individual models, but did not use it to evaluate the robustness of models. we believe that all of these could potenetially improve the accuracy and robustnes of nli models for clinical trial reports.