Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model (2022)

Abstract: Ensembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.

## introduction ensembling @xcite has long been an easy and effective approach to improve model performance by averaging the outputs of multiple comparable but independent models. @xcite explain that different models obtain different views for judgments, and the ensemble uses complementary views to make more robust decisions. a good ensemble requires diverse member models. however, how to encourage diversity without sacrificing the accuracy of each model is non-trivial @xcite @xcite . the pretrain-then-finetune paradigm has become another best practice for achieving state-of-the-art performance on nlp tasks @xcite . the cost of large-scale pretraining, however, is enormously high @xcite ; this often makes it difficult to independently pretrain multiple models. therefore, most researchers and practitioners only use a single pretrained model, which is distributed by resource-rich organizations. this situation brings up a novel question to ensemble learning: can we make an effective ensemble from only a single pre-trained model? although ensembles can be combined with the pretrain-then-finetune paradigm, an ensemble of figure 1 : when finetuning from a single pretrained model (left), the models are less diverse (center). if we finetune different sparse subnetworks, they become more diverse and make the ensemble effective (right). models finetuned from a single pretrained model is much less effective than that using different pretrained models from scratch in many tasks @xcite . naïve ensemble offers limited improvements, possibly due to the lack of diversity of finetuning from the same initial parameters. in this paper, we propose a simple yet effective method called multi-ticket ensemble, ensembling finetuned winning-ticket subnetworks @xcite in a single pretrained model. we empirically demonstrate that pruning a single pretrained model can make diverse models, and their ensemble can outperform the naïve dense ensemble if winning-ticket subnetworks are found. output of a model with the parameter vector θ given the input x, the output of an ensemble is f m (x) = θ∈m f (x; θ)/|m|, where @xmath0, ..., θ |m| } is the member parameters. ## subnetwork exploration we propose a simple yet effective method, multiticket ensemble, which finetunes different subnetworks instead of dense networks. because it could be a key how to find subnetworks, we explore three variants based on iterative magnitude pruning. 1 some concurrent and recent studies also investigate subnetworks for effective ensemble @xcite for training-from-scratch settings of image recognition. pruning mask pretrained model before each finetuning. we ex-088 pect that, during finetuning, each sub-network ac-089 quires different views using different sub-spaces of 090 the pretrained knowledge. this idea has two chal-091 lenges: the diversity and the accuracy of the sub-092 networks. recent studies on the lottery ticket hy-093 pothesis (frankle and carbin, 2019) suggest that a 094 dense neural network at an initialization contains a 095 sub-network, called winning ticket, whose accuracy 096 becomes comparable with that of the dense network 097 after the same training steps. a pretrained bert 098 also has sparse sub-networks (e.g., 50%), which 099 can achieve the same accuracy with the entire net-100 work when finetuning on downstream tasks (chen 101 et al., 2020). however, it is still unclear how di-102 verse winning tickets exist and how to find them 103 tasks using the bert in this paper, the same problem happens in other settings generally. in the results by raffel et al. (2020), we found that this happened on almost all the tasks of glue (wang et al., 2018), superglue (wang et al., 2019), squad (rajpurkar et al., 2016), summarization, and machine translations using the t5 models. verse winning tickets exist and how to find them 103 tasks using the bert in this paper, the same problem happens in other settings generally. in the results by @xcite , we found that this happened on almost all the tasks of glue @xcite , superglue @xcite , squad @xcite , summarization, and machine translations using the t5 models. on a random seed s. also, let ✓ s represent the parameter of the pretrained bert and the task-specific layer, whose parameter is randomly initialized by random seed s. after finetuning ✓ s to fine(✓ s , s), we identify and prune the parameters with 10% lowest magnitudes in fine(✓ s , s). we also get the corresponding binary mask of pruning, m s,10% 2 {0, 1} |✓s| , where the surviving positions have 1 otherwise 0. the pruning of parameters ✓ by a mask m can be also represented as ✓ m, where is the element-wise product. next, we replay finetuning but from ✓ s m s,10% and get fine(✓ s m s,10% , s) as well as 20%-pruning mask m s,20% . by repeating iterative magnitude pruning, we obtain the parameter fine(✓ s m s,p % , s). in our experiments, we set @xmath1, i.e., evaluate ensemble of 30%-pruning sub-networks, where @xmath2,30% , s 1 ), ..., fine(✓ s |m| 2 for diversifying models more, we propose to 086 introduce a novel randomness, (4) pruning of the 087 pretrained model before each finetuning. we ex- (2020), we found that this happened on almost all the tasks of glue @xcite , superglue @xcite , squad @xcite , summarization, and machine translations using the t5 models. let fine(✓, s) be the new parameters, which 2020 ), we found that this happened on almost all the tasks of glue @xcite , superglue @xcite , squad @xcite , summarization, and machine translations using the t5 models. the parameter of the pretrained bert and the 2020 ), we found that this happened on almost all the tasks of glue @xcite , superglue @xcite , squad @xcite , summarization, and machine translations using the t5 models. ## experiments we evaluate the performance of ensembles using four finetuning schemes: (1) finetuning without pruning (baseline), (2) finetuning of lotteryticket subnetworks found with the naïve iterative magnitude pruning (base-lt), and (3) with l 1 regularizer by the active masking (active-lt) or (4) random masking (random-lt). we also compare with (5) bagging-based ensemble, which trains dense models on different random 90% training subsets. we use the glue benchmark @xcite as tasks. the implementation and settings follow chen et al. (2020) foot_3 using the transformers library @xcite and its bert-baseuncased pretrained model. we report the average performance using twenty different random seeds. ensembles are evaluated using exhaustive combinations of five members. we also perform student's t-test for validating statistical significance foot_4 . note that, while the experiments focus on using bert, we believe that the insights would be helpful to other pretrain-then-finetune settings in general foot_5 . ## conclusion we raised a question on difficulty of ensembling large-scale pretrained models. as an efficient remedy, we explored methods to use subnetworks in a single model. we empirically demonstrated that ensembling winning-ticket subnetworks could outperform the dense ensembles via diversification and indicated a limitation too.