Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation (2024)

Abstract: While fine-tuning of pre-trained language models generally helps to overcome the lack of labelled training samples, it also displays model performance instability. This instability mainly originates from randomness in initialisation or data shuffling. To address this, researchers either modify the training process or augment the available samples, which typically results in increased computational costs. We propose a new mitigation strategy, called **Delayed Ensemble with Noisy Interpolation (DENI)**, that leverages the strengths of ensembling, noise regularisation and model interpolation, while retaining computational efficiency. We compare DENI with 9 representative mitigation strategies across 3 models, 4 tuning strategies and 7 text classification datasets. We show that: 1) DENI outperforms the best performing mitigation strategy (Ensemble), while using only a fraction of its cost; 2) the mitigation strategies are beneficial for parameter-efficient fine-tuning (PEFT) methods, outperforming full fine-tuning in specific cases; and 3) combining DENI with data augmentation often leads to even more effective instability mitigation.

## introduction tuning of pre-trained language models such as bert or roberta using either full fine-tuning or parameter efficient fine-tuning (peft) has achieved significant success across a wide range of natural language processing tasks. they are especially useful when faced with limited labelled data for quickly adapting to the specific task. despite the success, previous works observed that fine-tuning still remains unstable @xcite @xcite , especially with limited data. fine-tuning is sensitive to the effects of randomness originating from random initialisation, data shuffling or model randomness (e.g., use of non-deterministic layers such as dropout in the model). as illustrated in figure 1 , repeating the fine-tuning process multiple times, without mitigating the randomness, leads to large performance variance in the results, both for full fine-tuning and for peft methods. to deal with the fine-tuning instability, researchers propose various strategies to mitigate the effects of randomness @xcite . in most cases, the mitigation strategies focus on modifying the training process @xcite , such as adding noise @xcite @xcite , ensembling multiple models @xcite @xcite , or improving the experimental setup that potentially leads to the instability @xcite , such as using bias correction or training for longer. as many of these strategies are designed and evaluated on high-resource datasets, almost no focus is dedicated to evaluating the mitigation benefit of data augmentation @xcite . at the same time, the instability of the whole training process is addressed only when considering full fine-tuning. when using peft methods, the focus is on the initialisation of soft prompts, even though the factors such as data shuffling still lead to variance in results @xcite . overall, the best performing mitigation strategies are ensembles and model interpola-tion methods @xcite @xcite , which significantly reduce the deviation in results, but also significantly increase the computation costs. the methods that add noise to the model parameters @xcite also perform well, improving generalisability and overall performance, but not necessarily reducing the instability. inspired by the success and mutual complementarity between ensemble methods, model interpolation and noise regularisation, we propose a novel mitigation method delayed ensemble with noisy interpolation (deni). the deni method leverages the benefits of ensembling while reducing its computation costs. to achieve this, the ensemble is created at the end of training from a single model by perturbing its parameters using random noise (i.e., fighting randomness with randomness). in addition, the method creates the ensemble by adding noise, trains it for a few steps and then aggregates it into a single model multiple times during training (see figure 2 ), which leads to more effective mitigation of the randomness. using the deni method leads to lower variance in results and higher performance (as illustrated in figure 1 ). to evaluate the benefit of the deni method, we compare it with other representative mitigation strategies that modify the optimisation process. in the comparison, we also include an augmentation strategy that uses large language models to paraphrase samples, as such paraphrasing was observed to improve robustness and stability by @xcite , especially in limited data settings. besides full fine-tuning, we also explore the benefit of the proposed mitigation strategy and other baselines for representative parameter-efficient finetuning (peft) methods, namely lora, ia3 and unipelt. our main contributions and findings are foot_0 : ## related work: mitigating instability of fine-tuning the majority of strategies for mitigating the instability of fine-tuning modify the training process or the models themselves, addressing different sources of instability. one set of strategies addresses the suboptimal setup choices, suggesting that training for longer, using an optimiser with bias correction and lower learning rate with warmup and scheduling can reduce the instability @xcite . another set of approaches modifies the initialisation, such as using meta-learning (dauphin and schoenholz, 2019), re-initialising top layers of the pretrained models @xcite , or initialising multiple models and stopping the ones that show bad performance early in training @xcite . other strategies optimise only parts of the network @xcite or run supplementary pre-training on data rich tasks @xcite . mixout @xcite randomly replaces parts of the parameters with the original weights. further strategies add noise to the input or model parameters either before @xcite or during @xcite @xcite training. the ensembling strategies provide the best mitigation effectiveness, i.e., highest increase in performance and reduction in results variance, but at the cost of significant increase in computation costs @xcite @xcite . to reduce the cost, the ensemble can be created from a single model, such using stochastic weight averaging (swa) that performs an equal average of the weights traversed by the optimiser @xcite @xcite , or accelerated en-sembling that uses snapshots from different parts of training @xcite . another possibility is to share the lower layers and ensemble only the classification heads @xcite . however, such aggregation also reduces the mitigation effectiveness of the ensemble as it reduces the diversity of the models. although many augmentation strategies exist, they are mainly used to improve the overall performance in low-resource settings, without any consideration for the instability @xcite . only a few papers consider using data augmentation for addressing the optimisation instability by utilising pretrained language models as data generators @xcite or using common augmentation methods such as mixup @xcite . for the parameter-efficient fine-tuning (peft) methods, the focus is mostly on initialising prompts and mitigating randomness in prompt-tuning @xcite @xcite , while the focus on mitigating optimisation instability for the remaining peft methods is limited @xcite . ## experimental results and findings baselines. first, we use two basic baselines without any mitigation: 1) default, where the model is fine-tuned only on the limited data (representing the lower bound of performance); and 2) all data, where the model is fine-tuned on all the samples available in the dataset (representing the upper bound of performance). second, we compare with other representative approaches that introduce a change into the optimisation process, specifically: 1) best practices as reported by @xcite and @xcite , which includes using scheduling, warm-up, optimiser with bias correction and training for longer; 2) ensemble, where we concurrently train 10 models with different initialisation and use hard voting to aggregate their predictions; 3) noise input , where we periodically add noise to the input embeddings during training; 4) noise w eights , where we periodically add noise to the parameters during training; 5) stochastic weight averaging (swa) @xcite @xcite that performs an equal average of the weights traversed by the optimiser with modified learning rate schedule; and 6) mixout @xcite ) that stochastically mixes the parameters of two models instead of dropout. in addition, we also compare with augment n @xcite , which uses a large language model to paraphrase each sample n times to extend the training data (we report results only for the best performing number of paraphrases, augment 1 and 2). finally, we compare our proposed method (our den i ), its individual components (our de and our n i ) and its extended version delayed ensemble with noisy interpolation and augmented labelled samples (our den ials ) that combines deni with augment 1. more details for each of the strategies are reported in appendix c.2. datasets. the experiments are conducted on 7 text classification datasets composed of different tasks with different number of classes. we focus on 3 binary classification datasets from the glue benchmark (wang et al., 2018): sst2 (socher et al., 2013) for sentiment classification, cola (warstadt et al., 2019) for determining the grammatical acceptability of a sentence, and mrpc @xcite for determining the semantic equivalence relationship between two sentences. in addition, we use 4 multi-class text datasets: ag news @xcite for news classification, trec (voorhees and tice, 2000) for question classification, dbpedia @xcite for topic classification and snips @xcite for intent classification. ## conclusions in this work, we have proposed a novel method for effective and efficient mitigation of the instability in fine-tuning of pre-trained language models based on ensembling, model interpolation and noise regularisation. we compare the method with prior mitigation strategies, showing it leads to stronger mitigation, with lower computation cost. in addition, we show that the mitigation strategies provide more benefit to the peft methods, such as lora, ia3 or unipelt. combining mitigation strategies with peft can often lead to higher performance and lower deviation than their combination with full fine-tuning. moreover, we show that data augmentation provides more benefit when limiting the number of trainable parameters, such as using ia3 or training smaller models.