Ensemble-based Fine-Tuning Strategy for Temporal Relation Extraction from the Clinical Narrative (2022)

Abstract: In this paper, we investigate ensemble methods for fine-tuning transformer-based pretrained models for clinical natural language processing tasks, specifically temporal relation extraction from the clinical narrative. Our experimental results on the THYME data show that ensembling as a fine-tuning strategy can further boost model performance over single learners optimized for hyperparameters. Dynamic snapshot ensembling is particularly beneficial as it fine-tunes a wide array of parameters and results in a 2.8% absolute improvement in F1 over the base single learner.

## introduction the clinical narrative in electronic medical records (emrs) can provide critical information for improving quality of care, patient outcomes, and safety. extracting information from emrs has been an active area of research in recent years due to the advances in natural language processing (nlp) techniques. as transformer-based neural language models, such as bidirectional encodings representations from transformers (bert) @xcite , have achieved state-of-the-art performance for a variety of nlp tasks they have gained increased prominence in clinical nlp. however, in the clinical domain, data is often sparsely labeled and not shareable as it is guarded by patient confidentiality provisions. building large transformer-based models from scratch using such data is thus often infeasible. a common approach has been to take models pretrained on large general domain corpora, and continue pretraining them on clinical corpora to derive domain-specific language models @xcite @xcite . the weights of pretrained models are adjusted for a specific clinical nlp task through the process of fine-tuning. this process often involves searching for optimal hyperparameters while continuing to train the pretrained model on a domain-specific dataset. the search is challenging due to the high dimensionality of the search space, which includes random seed, initial learning rate, batch size, etc. given the limited computing resources available in practice, only a small number of values for each hyperparameter can be explored, and often only a subset of hyperparameters can be fine-tuned. are we able to retain the benefits from the existing search efforts and to further improve model performance for the same task or new tasks without too much extra effort? ensemble methods have been successful in boosting predictive performance of single learners @xcite @xcite and thus are promising. in this paper, we will investigate ensemble-based finetuning methods to answer this question. another downside of the limited search capability is that some hyperparameters are unexplored in past efforts. for example, learning rate schedules have rarely been explored in previous efforts of fine-tuning. one promising approach is training with cyclical learning rates (e.g., cosine annealing learning rate and slanted triangular learning rate), which have been shown to achieve improved classification accuracy in fewer iterations @xcite . we will explore the impact of cyclical learning rates in fine-tuning methods in the context of an ensemble algorithm. major contributions: in this work, (1) we use ensembles to investigate the impact of various hyperparameters for fine-tuning pretrained transformerbased models for the clinical domain by focusing on one critical task -temporal relation extraction; ## ensemble-based fine-tuning and experimental setup our intuition behind using ensembles for finetuning is to leverage models from local optima to obtain greater coverage of the feature space, and get consensus for the predictions so that the ensemble learner can reduce the overall risk of making a poor selection. in this section, we first describe our setting and implementation of a base model based on the state-of-the-art setting described by @xcite . then we discuss fine-tuning several hyperparameters during training and their potential impact on model performance. based on these discussions, we then introduce the bagging ensemble method @xcite and the dynamic snapshot ensemble method @xcite and apply them to the fine-tuning process. ## results and discussion we show model performance in however, all sources of diversity are not equal, with the diversity from different epochs of a training run being most helpful, and diversity of learning rate schedulers and diversity of pubmedbert variants helping little. a possible reason is that both lrscheduler-ens and pretrainedmodel-ens have only 3 components while the other ensemble learners have 5 components, as @xcite proved that a better precision can be achieved if more component learners are combined. however, that would not explain the superiority of epoch-ens to datashuffle-ens and seed-ens, and an improvement of the ensemble's performance is not guaranteed if many poor learners are combined. dynsnap-ens outperforms all the other ensemble learners, likely because it takes advantage of all the individual types of diversity: data, model parameters, epochs, and learning rate. figure 1 presents the training history on learning rate, training loss, and validation accuracy along epochs. we can observe that learning behavior changes a lot with respect to each source of diversity. dynsnap-ens combines those sources in a computationally effi-cient way and selects top single learners (marked in yellow squares) from a more diversified pool to guarantee an improvement in the final ensemble learner. ## conclusion we investigated ensemble methods in fine-tuning transformer-based pretrained models for clinical nlp tasks, specifically temporal relation extraction from the clinical narrative. our experimental results on the thyme++ data showed that ensembling can further boost performance, and that dynamic snapshot ensembling is especially effective. future works include: 1) investigating the impact of ensemble size in model performance; 2) exploring hyperparameters regarding the snapshot strategy and pruning algorithm; 3) testing the trained ensemble learners on an expanded set of clinical domain tasks.