Bi-Drop: Enhancing Fine-tuning Generalization via Synchronous sub-net Estimation and Optimization (2023)

Abstract: Pretrained language models have achieved remarkable success in natural language understanding. However, fine-tuning pretrained models on limited training data tends to overfit and thus diminish performance. This paper presents Bi-Drop, a fine-tuning strategy that selectively updates model parameters using gradients from various sub-nets dynamically generated by dropout. The sub-net estimation of Bi-Drop is performed in an in-batch manner, so it overcomes the problem of hysteresis in sub-net updating, which is possessed by previous methods that perform asynchronous sub-net estimation. Also, Bi-Drop needs only one mini-batch to estimate the sub-net so it achieves higher utility of training data. Experiments on the GLUE benchmark demonstrate that Bi-Drop consistently outperforms previous fine-tuning methods. Furthermore, empirical results also show that Bi-Drop exhibits excellent generalization ability and robustness for domain transfer, data imbalance, and low-resource scenarios.

## introduction in recent years, natural language processing (nlp) has achieved significant progress due to the emergence of large-scale pretrained language models (plms) @xcite @xcite . for downstream tasks, compared with training from scratch, fine-tuning pretrained models can usually achieve efficient adaptation and result in better performance. despite the great success, fine-tuning methods still face challenges in maintaining generalization performance on downstream tasks -they tend to run into the overfitting issue when the training data is limited @xcite @xcite . to improve the generalization ability of finetuning methods, many regularization techniques have been proposed @xcite et al., 2021; @xcite @xcite , such as sub-net optimization strategies like child-tuningd @xcite and dps @xcite . child-tuningd selects a static sub-net for updating based on parameter importance estimated by fisher information (fi). as an improved variant of child-tuning d , dps dynamically decides the sub-net to be updated by estimating fi with multiple mini-batches of data. although these fi-based methods achieve better generalization ability than vanilla fine-tuning, they still have two limitations: (1) hysteresis in sub-net updating: the sub-net preference is estimated with the model parameters in previous iterations and may be incompatible with the current update step; and in this paper, we delve deeper into adaptive subnet optimization strategies and propose bi-drop, a fi-free strategy for fine-tuning pretrained language models. unlike fisher information estimation, which requires cumulative gradients of minibatches, bi-drop only relies on information in a single mini-batch to select the parameters to update. specifically, bi-drop utilizes gradient information from different sub-nets dynamically generated by dropout in each mini-batch. as illustrated in figure 1 , within a single training step of bi-drop, a minibatch will go through the forward pass multiple times and, due to the randomness introduced by dropout, yield various distinct sub-nets. we then apply a parameter selection algorithm with perturbation and scaling factors to stabilize the gradient updates. with this synchronous parameter selection strategy, bi-drop can selectively update model parameters according to the information from only the current mini-batch, and thus mitigate overfitting with a high utility of training data. extensive experiments on the glue benchmark demonstrate that bi-drop shows remarkable superiority over the state-of-the-art fine-tuning regularization methods, with a considerable margin of 0.53 ∼ 1.50 average score. moreover, bi-drop consistently outperforms vanilla fine-tuning by 0.83 ∼ 1.58 average score across various plms. further analysis indicates that bi-drop attains superb generalization ability for domain transfer and task transfer, and is robust for data imbalance and lowresource scenarios. to sum up, our contributions are three-fold: models sub-net granularity hysteresis data depend. child-tuning d task-wise yes yes dps cycle-wise yes yes bi-drop step-wise no no regularization methods for fine-tuning largescale plms are prone to over-fitting @xcite and exhibit inadequate generalization ability when fine-tuned with limited training data @xcite , resulting in degraded performance. to tackle this issue, various regularization techniques have been suggested to enhance the generalization capacity of models, including advanced dropout alternatives @xcite , applying adversarial perturbations @xcite @xcite and constrained regularization methods @xcite . in recent years, child-tuning @xcite and dps @xcite propose to estimate parameter importance based on fisher information (fi) and selectively optimize a sub-net during fine-tuning to mitigate overfitting. fi-based methods have a strong dependence on the training data and exhibit hysteresis in sub-net updating. as shown in in-batch data (1) forward multiple times with dropout 𝜕ℒ 𝑡𝑟𝑎𝑖𝑛 𝜕θ 1 𝜕ℒ 𝑡𝑟𝑎𝑖𝑛 𝜕θ 𝑘 ⋯ ⋯ backward gradients of sub-net 1 backward gradients of sub-net k 0 1 0 1 0 1 (2) calculate the gradients mask ⨀ average gradients (3) selectively update parameters neuron units dropped units gradient mask masked average gradients (2) sub-net selection: an advanced strategy is adopted to select the sub-net to be updated based on the gradients of distinct sub-nets generated by dropout; ## limitations we propose a novel and effective fine-tuning method, bi-drop, which achieves a considerable performance improvement in downstream tasks. however, similar to some previous studies @xcite @xcite , bi-drop requires multiple forward propagations, which makes its training time efficiency not good enough compared with the vanilla fine-tuning method.