Original: Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Revised: Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Original: Anonymous ACL submission
Revised: Preferred Networks, Inc.

Revised: RIKENsosk@preferred.jp shun.kiyono@riken.jp jun.suzuki@tohoku.ac.jp inui@ecei.tohoku.ac.jp

Original: Abstract
Revised: Abstract

Original: Ensembling is a popular method used to improve performance as a last resort.
Revised: Ensembling is a popular method used to improve performance as a last resort.

Original: However,ensembling multiple models ﬁnetuned from asingle pretrained model has been not very effective; this could be due to the lack of diversityamong ensemble members.
Revised: However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members.

Original: This paper proposes Multi-Ticket Ensemble , which ﬁnetunesdifferent subnetworks of a single pretrainedmodel and ensembles them.
Revised: This paper proposes Multi-Ticket Ensemble , which finetunes different subnetworks of a single pretrained model and ensembles them.

Original: We empiricallydemonstrated that winning-ticket subnetworksproduced more diverse predictions than densenetworks, and their ensemble outperformed thestandard ensemble on some tasks.
Revised: We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.

Original:  Introduction
Revised:  Introduction

Original: Ensembling (Levin et al., 1989; Domingos, 1997)has long been an easy and effective approach to improve model performance by averaging the outputsof multiple comparable but independent models.
Revised: Ensembling (Levin et al., 1989; Domingos, 1997) has long been an easy and effective approach to improve model performance by averaging the outputs of multiple comparable but independent models.

Original: Allen-Zhu and Li (2020) explain that different models obtain different views for judgments, and theensemble uses complementary views to make morerobust decisions.
Revised: Allen-Zhu and Li (2020) explain that different models obtain different views for judgments, and the ensemble uses complementary views to make more robust decisions.

Original: A good ensemble requires diversemember models.
Revised: A good ensemble requires diverse member models.

Original: However, how to encourage diversity without sacriﬁcing the accuracy of each modelis non-trivial (Liu and Yao, 1999; Kirillov et al. ,2016; Rame and Cord, 2021).
Revised: However, how to encourage diversity without sacrificing the accuracy of each model is non-trivial (Liu and Yao, 1999; Kirillov et al., 2016; Rame and Cord, 2021).

Original: The pretrain-then-ﬁnetune paradigm has becomeanother best practice for achieving state-of-the-artperformance on NLP tasks (Devlin et al., 2019).
Revised: The pretrain-then-finetune paradigm has become another best practice for achieving state-of-the-art performance on NLP tasks (Devlin et al., 2019).

Original: The cost of large-scale pretraining, however, isenormously high (Sharir et al., 2020); This oftenmakes it difﬁcult to independently pretrain multiple models.
Revised: The cost of large-scale pretraining, however, is enormously high (Sharir et al., 2020); This often makes it difficult to independently pretrain multiple models.

Original: Therefore, most researchers and practitioners only use a single pretrained model, whichis distributed by resource-rich organizations.
Revised: Therefore, most researchers and practitioners only use a single pretrained model, which is distributed by resource-rich organizations.

Original: This situation brings up a novel question to ensemble learning: Can we make an effective ensemble from only a single pre-trained model ?
Revised: This situation brings up a novel question to ensemble learning: Can we make an effective ensemble from only a single pre-trained model ?

Original: Although ensembles can be combined with thepretrain-then-ﬁnetune paradigm, an ensemble of models ﬁnetuned from a single pretrained model is much less effective than that using different pretrained models from scratch in many tasks (Raffel et al., 2020).
Revised: Although ensembles can be combined with the pretrain-then-finetune paradigm, an ensemble of models finetuned from a single pretrained model is much less effective than that using different pretrained models from scratch in many tasks (Raffel et al., 2020).

Original: Naïve ensemble offers limited improvements, possibly due to the lack of diversity of ﬁnetuning from the same initial parameters.
Revised: Naïve ensemble offers limited improvements, possibly due to the lack of diversity of finetuning from the same initial parameters.

Original: In this paper, we propose a simple yet effective method called Multi-Ticket Ensemble , ensembling ﬁnetuned winning-ticket subnetworks (Frankle and
Revised: In this paper, we propose a simple yet effective method called Multi-Ticket Ensemble , ensembling finetuned winning-ticket subnetworks (Frankle and

Original: Carbin, 2019) in a single pretrained model.
Revised: Carbin, 2019) in a single pretrained model.

Original: We empirically demonstrate that pruning a single pretrained model can make diverse models, and their ensemble can outperform the naïve dense ensemble if winning-ticket subnetworks are found.
Revised: We empirically demonstrate that pruning a single pretrained model can make diverse models, and their ensemble can outperform the naïve dense ensemble if winning-ticket subnetworks are found.

Original:  Diversity in a Single Pretrained Model
Revised:  Diversity in a Single Pretrained Model

Original: In this paper, we discuss the most standard way of ensemble, which averages the outputs of multiple neural networks; each has the same architecture but different parameters.
Revised: In this paper, we discuss the most standard way of ensemble, which averages the outputs of multiple neural networks; each has the same architecture but different parameters.

Original: That is, let f ( x ; ✓ ) be the output of a model with the parameter vector ✓ giventhe input x , the output of an ensemble is f M ( x )=
Revised: That is, let f ( x ; θ ) be the output of a model with the parameter vector θ given the input x , the output of an ensemble is f M ( x )

Original: P ✓ 2 M f ( x ; ✓ ) / |M| , where M = { ✓ 1 , ..., ✓ |M| }is the member parameters.
Revised: = (cid:80) θ ∈M f ( x ; θ ) / |M| , where M = { θ 1 , ..., θ |M| } is the member parameters.

Original: 2. Diversity from Finetuning
Revised: 2. Diversity from Finetuning

Original: As discussed, when constructing an ensemble f M by ﬁnetuning from a single pretrainedmodel multiple times with different random seeds{ s 1 , ..., s |M| } , the boost in performance tends tobe only marginal.
Revised: As discussed, when constructing an ensemble f M by finetuning from a single pretrained model multiple times with different random seeds { s 1 , ..., s |M| } , the boost in performance tends to be only marginal.

Original: In the case of BERT (Devlinet al., 2019) and its variants, three sources of diversities can be considered: random initialization ofthe task-speciﬁc layer, dataset shufﬂing for stochastic gradient descent (SGD), and dropout.
Revised: In the case of BERT (Devlin et al., 2019) and its variants, three sources of diversities can be considered: random initialization of the task-specific layer, dataset shuffling for stochastic gradient descent (SGD), and dropout.

Original: However,empirically, such ﬁnetuned parameters tend not tobe largely different from the initial parameters, andthey do not lead to diverse models (Radiya-Dixitand Wang, 2020).
Revised: However, empirically, such finetuned parameters tend not to be largely different from the initial parameters, and they do not lead to diverse models (Radiya-Dixit and Wang, 2020).

Original: Of course, if one adds signiﬁcant noise to the parameters, it leads to diversity;however, it would also hurt accuracy.
Revised: Of course, if one adds significant noise to the parameters, it leads to diversity; however, it would also hurt accuracy.

Original: 2. Diversity from Pruning
Revised: 2. Diversity from Pruning

Original: To make models ensuring both accuracy and diversity, we focus on subnetworks in the pretrainedmodel.
Revised: To make models ensuring both accuracy and diversity, we focus on subnetworks in the pretrained model.

Original: Different subnetworks employ differentsubspaces of the pre-trained knowledge (RadiyaDixit and Wang, 2020; Zhao et al., 2020; Cao et al.,2021); this would help the subnetworks to acquiredifferent views, which can be a source of desired diversity 1 .
Revised: Different subnetworks employ different subspaces of the pre-trained knowledge (RadiyaDixit and Wang, 2020; Zhao et al., 2020; Cao et al., 2021); this would help the subnetworks to acquire different views, which can be a source of desired diversity 1 .

Original: Also, in terms of accuracy, recent studieson the lottery ticket hypothesis (Frankle and Carbin,2019) suggest that a dense network at initializationcontains a subnetwork, called the winning ticket ,whose accuracy becomes comparable to that of thedense one after the same training.
Revised: Also, in terms of accuracy, recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that a dense network at initialization contains a subnetwork, called the winning ticket , whose accuracy becomes comparable to that of the dense one after the same training.

Original: Interestingly,the pretrained BERT also has a winning ticket forﬁnetuning on downstream tasks (Chen et al., 2020).
Revised: Interestingly, the pretrained BERT also has a winning ticket for finetuning on downstream tasks (Chen et al., 2020).

Original: Thus, if we can ﬁnd diverse winning tickets , theycan be good ensemble members with the two desirable properties: diversity and accuracy.
Revised: Thus, if we can find diverse winning tickets , they can be good ensemble members with the two desirable properties: diversity and accuracy.

Original:  Subnetwork Exploration
Revised:  Subnetwork Exploration

Original: We propose a simple yet effective method, multiticket ensemble , which ﬁnetunes different subnetworks instead of dense networks.
Revised: We propose a simple yet effective method, multiticket ensemble , which finetunes different subnetworks instead of dense networks.

Original: Because it couldbe a key how to ﬁnd subnetworks, we explore threevariants based on iterative magnitude pruning.
Revised: Because it could be a key how to find subnetworks, we explore three variants based on iterative magnitude pruning.

Original: [Table]
Revised: [Table]

Original: Pruning mask the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.
Revised: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.

Original: Diversity of Finetuning from a Single
Revised: Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 , ..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 , ..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 , ..., s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 , ..., s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are prunedefﬁciently for better ensembles.
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 114 First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 114 First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ , s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ , s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent the parameter of the pretrained BERT and the task-speciﬁc layer, whose parameter is randomly initialized by random seed s .
Revised: Also, let ✓ s represent the parameter of the pretrained BERT and the task-speciﬁc layer, whose parameter is randomly initialized by random seed s .

Original: After ﬁnetuning ✓ s to FINE ( ✓ s , s ) , we identify and prune the parameters with 10% lowest magnitudes in
Revised: After ﬁnetuning ✓ s to FINE ( ✓ s , s ) , we identify and prune the parameters with 10% lowest magnitudes in

Original: FINE ( ✓ s , s ) .
Revised: FINE ( ✓ s , s ) .

Original: We also get the corresponding binary mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where the surviving positions have 1 otherwise 0.

Original: The pruning of parameters ✓ by a mask m can be also represented as ✓  m , where  is the element-wise product.
Revised: The pruning of parameters ✓ by a mask m can be also represented as ✓  m , where  is the element-wise product.

Original: Next, we replay ﬁnetuning but from ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% , s ) as well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% , s ) as well as 20%-pruning mask m s, 20% .

Original: By repeating iterative magnitude pruning, we obtain the parameter FINE
Revised: By repeating iterative magnitude pruning, we obtain the parameter FINE

Original: ( ✓ s  m s,P % , s ) .
Revised: ( ✓ s  m s,P % , s ) .

Original: In our experiments, we set P = 30 , i.e., evaluate ensemble of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% , s 1 ) , ..., FINE ( ✓ s |M|  2 We prune embedding , following Chen
Revised: In our experiments, we set P = 30 , i.e., evaluate ensemble of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% , s 1 ) , ..., FINE ( ✓ s [Table] 2 We also did

Original: [Table]
Revised: [Table]

Original: ; Prasanna
Revised: the

Original: [Table]
Revised: [Table]

Original: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: [Table] [Table] ) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned  for better 
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned Figure 2: Overview of iterative magnitude pruning. efﬁciently for better ensembles.

Original: In this paper, we propose methods to ﬁnd di-
Revised: In this paper, we propose methods to ﬁnd di-

Original: 105 verse sub
Revised: 105 verse sub

Original: -networks.
Revised: -networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub- 109 networks by pruning random parameters at initial- 110 ization, the accuracy becomes signiﬁcantly deterio- 111 rated
Revised: For example, 108 if we aggressively realize the diversity of the sub- 109 networks by pruning random parameters at initial- 110 ization, the accuracy becomes signiﬁcantly deterio- 111 rated

Original: (Prasanna et al., 2020).
Revised: (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub- 116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub- 116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and thetask-speciﬁc layer, whose parameter is randomlyinitialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and thetask-speciﬁc layer, whose parameter is randomlyinitialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune theparameters with 10% lowest magnitudes in
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune theparameters with 10% lowest magnitudes in

Original: FINE ( ✓ s ,s ) .
Revised: FINE ( ✓ s ,s ) .

Original: We also get the corresponding binarymask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binarymask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: Thepruning of parameters ✓ by a mask m can be alsorepresented as ✓  m , where  is the element-wise 132 product.
Revised: Thepruning of parameters ✓ by a mask m can be alsorepresented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeatingiterative magnitude pruning, we obtain theparameter FINE
Revised: By repeatingiterative magnitude pruning, we obtain theparameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper- 137 iments, we set P = 30 , i.e., evaluate ensembleof 30%-pruning sub-networks, where M = [Table] [Table] [Table]
Revised: In our exper- 137 iments, we set P = 30 , i.e., evaluate ensembleof 30%-pruning sub-networks, where M ={ FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M| 2 We also did not prune the embedding layer, following Chen et al.

Original: [Table]
Revised: (2020); Prasanna et al.

Original:  the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetworkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetworkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- pensinothersettingsgenerally.
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- pensinothersettingsgenerally.

Original: IntheresultsbyRaffeletal.
Revised: IntheresultsbyRaffeletal.

Original: (2020),wefoundthatthishappenedonalmostallthetasksofGLUE(Wangetal.
Revised: (2020),wefoundthatthishappenedonalmostallthetasksofGLUE(Wangetal.

Original: ,2018),SuperGLUE(Wangetal.
Revised: ,2018),SuperGLUE(Wangetal.

Original: ,2019),SQuAD(Rajpurkaretal.
Revised: ,2019),SQuAD(Rajpurkaretal.

Original: ,2016),summarization,andmachine translationsusingtheT5models.
Revised: ,2016),summarization,andmachine translationsusingtheT5models.

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ne d R a nd o m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) . 
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) . We also get the corresponding binary [Table] of pruning, m s,

Revised: [Table]

Original: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: [Table] } | [Table] s | , where 129 the surviving positions have 1 otherwise 0.

Original: The 130 pruning of parameters 
Revised: The [Table] pruning of parameters ✓ [Table]

Revised: a mask m can be

Original: [Table]
Revised: [Table]

Original: by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product. 
Revised: represented as ✓  m , where  is the element-wise [Table] product. Next, we replay ﬁnetuning but from

Revised: [Table]

Original: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: s [Table] m s, 10% and get FINE ( [Table]

Revised: s

Original: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE
Revised: [Table] m s, 10% ,s ) as [Table] well as 20%-pruning

Revised: [Table]

Original: ( ✓ s  m s,P % ,s ) .
Revised: m s, [Table] By repeating [Table]

Revised: iterative

Original: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble
Revised: [Table] pruning, we obtain the [Table] s,

Original: [Table]
Revised: [Table]

Original: of 30%-pruning sub-networks, where M = {
Revised: s ) . In our exper-

Original: [Table]
Revised: [Table]

Original:  , 30% ,s 1 ) ,..., FINE ( [Table] [Table] Chenetal
Revised: iments, we set P = 30 , i.e., evaluate ensemble

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: of 30%-pruning sub-networks, where

Original: [Table]
Revised: [Table]

Original: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using [Table] in [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: paper, [Table] hap- [Table]

Revised: generally.

Revised: In

Revised: [Table]

Revised: results by Raffel et al.

Revised: , we found that this happened on almost

Revised: [Table]

Revised: of GLUE

Revised: (Wang et al., 2018),

Revised: [Table]

Revised: Wang et al.,

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned Figure of iterative magnitude pruning.
Revised: [Table] [Table] [Table] [Table]

Original: efﬁciently for better ensembles.
Revised: are [Table]

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .

Original: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.
Revised: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE
Revised: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ [Equation] [Equation] [Equation]
Revised: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M|  2 We also did not prune the embedding layer, following Chen et al.

Original: ., FINE ( ✓

Original: [Equation]

Original: M|  2 We also did not prune the embedding layer, following Chen et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random ini- dataset for (SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empiri- cally, they do not necessarily lead to diverse mod- els, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetworkafter the same training steps. 
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetwork [Table] after the same training steps. A pretrained BERT

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: [Table] also has sparse sub-networks (e.g., 50%), which [Table] can achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chen

Original:  However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- pensinothersettingsgenerally.
Revised: [Table] et al., 2020). However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- [Table]

Original: IntheresultsbyRaffeletal.
Revised: IntheresultsbyRaffeletal.

Original: (2020),wefoundthatthishappenedonalmostallthetasksofGLUE(Wangetal.
Revised: [Table] ,wefoundthatthishappenedonalmostallthetasksofGLUE [Table] [Table]

Original: ,2018),SuperGLUE(Wangetal. ,2019),SQuAD(Rajpurkaretal.
Revised: [Table] [Table]

Original: ,2016),summarization,andmachine translationsusingtheT5models.
Revised: [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i n ed R a nd o m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are prunedFigure Overview iterative magnitude pruning. efﬁciently for better ensembles. In this paper, we propose methods toﬁnd di105 verse sub-networks.
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned propose methods to di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .

Original: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.
Revised: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE
Revised: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M|  2 We also did not prune the embedding layer, following Chenetal.(2020);Prasannaetal.(2020)
Revised: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , [Figure]

Original: We can also use regularizers during ﬁnetuning to diversify pruning 
Revised: We can also use regularizers during finetuning to diversify pruning (Section 3.2).

Original: 3. Iterative Magnitude Pruning
Revised: 3. Iterative Magnitude Pruning

Original: We employ iterative magnitude pruning (Frankle and Carbin, 2019) to ﬁnd winning tickets for simplicity.
Revised: We employ iterative magnitude pruning (Frankle and Carbin, 2019) to find winning tickets for simplicity.

Original: Other sophisticated options are left for future work.
Revised: Other sophisticated options are left for future work.

Original: Here, we explain the algorithm (refer to the paper for details).
Revised: Here, we explain the algorithm (refer to the paper for details).

Original: The algorithm explores a good pruning mask via rehearsals of ﬁnetuning.
Revised: The algorithm explores a good pruning mask via rehearsals of finetuning.

Original: First, it completes a ﬁnetuning procedure of an initialized dense network and identiﬁes the parameters with the 10% lowest magnitudes as the targets of pruning.
Revised: First, it completes a finetuning procedure of an initialized dense network and identifies the parameters with the 10% lowest magnitudes as the targets of pruning.

Original: Then, it makes the pruned subnetwork and resets its parameters to the originally-initialized (sub-)parameters.
Revised: Then, it makes the pruned subnetwork and resets its parameters to the originally-initialized (sub-)parameters.

Original: This ﬁnetune-prune-reset process is repeated until reaching the desired pruning ratio.
Revised: This finetune-prune-reset process is repeated until reaching the desired pruning ratio.

Original: We used 30% as pruning ratio.
Revised: We used 30% as pruning ratio.

Original: 3. Pruning with Regularizer
Revised: 3. Pruning with Regularizer

Original: We discussed that ﬁnetuning with different random seeds did not lead to diverse parameters in Section 2.1.
Revised: We discussed that finetuning with different random seeds did not lead to diverse parameters in Section 2.1.

Original: Therefore, iterative magnitude pruning with different seeds could also produce less diverse subnetworks.
Revised: Therefore, iterative magnitude pruning with different seeds could also produce less diverse subnetworks.

Original: Thus, we also explore means of diversifying pruning patterns by enforcing different parameters to have lower magnitudes.
Revised: Thus, we also explore means of diversifying pruning patterns by enforcing different parameters to have lower magnitudes.

Original: Motivated by this, we experiment with a simple approach, applying an L 1 regularizer (i.e., magnitude decay) to different parameters selectively depending on the random seeds.
Revised: Motivated by this, we experiment with a simple approach, applying an L 1 regularizer (i.e., magnitude decay) to different parameters selectively depending on the random seeds.

Original: Speciﬁcally, we explore two policies to determine which parameters are decayed and how strongly they are, i.e., the element-wise coefﬁcients of the L 1 regularizer, l s 2 R  

Revised: Specifically, we explore two policies to determine which parameters are decayed and how strongly they are, i.e., the element-wise coefficients of the L 1 regularizer, l s ∈ R ≥

Original: | ✓ | .
Revised: | θ | .

Original: During ﬁnetuning (for pruning), we add a regularization term ⌧ || ✓ s  l s || 1 with a positive scalar coefﬁcient ⌧ into the loss of the task (e.g., cross entropy for classiﬁcation), where  is element-wise product.
Revised: During finetuning (for pruning), we add a regularization term τ || θ s ⊙ l s || 1 with a positive scalar coefficient τ into the loss of the task (e.g., cross entropy for classification), where ⊙ is element-wise product.

Original: This softly enforces various parameters to have a lower magnitude among a set of random seeds and could lead various parameters to be pruned.
Revised: This softly enforces various parameters to have a lower magnitude among a set of random seeds and could lead various parameters to be pruned.

Original: Active Masking
Revised: Active Masking

Original: To maximize the diversity ofthe surviving parameters of member models, it isnecessary to prune the surviving parameters of therandom seed s 1 when building a model with thenext random seed s 2 .
Revised: To maximize the diversity of the surviving parameters of member models, it is necessary to prune the surviving parameters of the random seed s 1 when building a model with the next random seed s 2 .

Original: Thus, during ﬁnetuning withseed s 2 , we apply the L 1 regularizer on the ﬁrstsurviving parameters.
Revised: Thus, during finetuning with seed s 2 , we apply the L 1 regularizer on the first surviving parameters.

Original: Likewise, with the following seeds s 3 , s 4 , ..., s i , ..., s |M| , we cumulativelyuse the average of the surviving masks as the regularizer coefﬁcient mask.
Revised: Likewise, with the following seeds s 3 , s 4 , ..., s i , ..., s |M| , we cumulatively use the average of the surviving masks as the regularizer coefficient mask.

Original: Let m s j 2 { 0 , 1 } | ✓ | bethe pruning mask indicating surviving parametersfrom seed s j , the coefﬁcient mask with seed
Revised: Let m s j ∈ { 0 , 1 } | θ | be the pruning mask indicating surviving parameters from seed s j , the coefficient mask with seed

Original: s i isl s i=
Revised: s i is l s i

Revised: = (cid:80)

Original: P j<i m s j / ( i  1) .
Revised: j<i m s j / ( i − 1) .

Original: We call this afﬁrmativepolicy as active masking .
Revised: We call this affirmative policy as active masking .

Original: Random Masking
Revised: Random Masking

Original: In active masking, each coefﬁcient mask has a sequential dependence on thepreceding random seeds.
Revised: In active masking, each coefficient mask has a sequential dependence on the preceding random seeds.

Original: Thus, the training of ensemble members cannot be parallelized.
Revised: Thus, the training of ensemble members cannot be parallelized.

Original: Therefore, we also experiment with a simpler and parallelizable variant, random masking , where a maskis independently and randomly generated from arandom seed.
Revised: Therefore, we also experiment with a simpler and parallelizable variant, random masking , where a mask is independently and randomly generated from a random seed.

Original: With a random seed s i , we generate the seed-dependent random binary mask, i.e.,l s = m rand s i 2 { 0 , 1 } | ✓ | , where each element issampled from Bernoulli distribution and 0’s probability equals to the target pruning ratio.
Revised: With a random seed s i , we generate the seed-dependent random binary mask, i.e., l s = m rand s i ∈ { 0 , 1 } | θ | , where each element is sampled from Bernoulli distribution and 0’s probability equals to the target pruning ratio.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the performance of ensembles using four ﬁnetuning schemes: (1) ﬁnetuning without pruning ( BASELINE ), (2) ﬁnetuning of lotteryticket subnetworks found with the naïve iterativemagnitude pruning ( BASE - LT ), and (3) with L 1 regularizer by the active masking ( ACTIVE - LT ) or (4)random masking ( RANDOM - LT ).
Revised: We evaluate the performance of ensembles using four finetuning schemes: (1) finetuning without pruning ( BASELINE ), (2) finetuning of lotteryticket subnetworks found with the naïve iterative magnitude pruning ( BASE - LT ), and (3) with L 1 regularizer by the active masking ( ACTIVE - LT ) or (4) random masking ( RANDOM - LT ).

Original: We also comparewith (5) BAGGING -based ensemble, which trainsdense models on different random 90% trainingsubsets.
Revised: We also compare with (5) BAGGING -based ensemble, which trains dense models on different random 90% training subsets.

Original: We use the GLUE benchmark (Wang et al.,2018) as tasks.
Revised: We use the GLUE benchmark (Wang et al., 2018) as tasks.

Original: The implementation and settingsfollow Chen et al.
Revised: The implementation and settings follow Chen et al.

Original: (2020) 2 using the Transformers library (Wolf et al., 2020) and its bert-baseuncased pretrained model.
Revised: (2020) 2 using the Transformers library (Wolf et al., 2020) and its bert-baseuncased pretrained model.

Original: We report the averageperformance using twenty different random seeds.
Revised: We report the average performance using twenty different random seeds.

Original: Ensembles are evaluated using exhaustive combinations of ﬁve members.
Revised: Ensembles are evaluated using exhaustive combinations of five members.

Original: We also perform Student’st-test for validating statistical signiﬁcance 3 .
Revised: We also perform Student’s t-test for validating statistical significance 3 .

Original: Note
Revised: Note

Original: [Table]
Revised: [Table]

Original: that, while the experiments focus on using BERT, we believe that the insights would be helpful to other pretrain-then-ﬁnetune settings in general 4 .
Revised: that, while the experiments focus on using BERT, we believe that the insights would be helpful to other pretrain-then-finetune settings in general 4 .

Original: 4. Accuracy
Revised: 4. Accuracy

Original: We show the results on MRPC (Dolan and Brockett, 2005) and STS-B (Cer et al., 2017) in Table 1.
Revised: We show the results on MRPC (Dolan and Brockett, 2005) and STS-B (Cer et al., 2017) in Table 1.

Original: Multi-ticket ensembles (*- LT ) outperform BASELINE and BAGGING signiﬁcantly ( p < 0 .
Revised: Multi-ticket ensembles (*- LT ) outperform BASELINE and BAGGING significantly ( p < 0 .

Original: This result supports the effectiveness of multi-ticket ensemble.
Revised: This result supports the effectiveness of multi-ticket ensemble.

Original: Note that the improvements of *- LT are attributable to ensembling (diff.) rather than to any performance gains of the individual models (single).
Revised: Note that the improvements of *- LT are attributable to ensembling (diff.) rather than to any performance gains of the individual models (single).

Original: We also plot the improvements (ens. values relative to BASELINE ) as a function of the number of ensemble members on MRPC and STS-B in Figure 3.
Revised: We also plot the improvements (ens. values relative to BASELINE ) as a function of the number of ensemble members on MRPC and STS-B in Figure 3.

Original: This also clearly shows that while the single models of *- LT have accuracy similar to BASELINE , the gains appear when ensembling them.
Revised: This also clearly shows that while the single models of *- LT have accuracy similar to BASELINE , the gains appear when ensembling them.

Original: While multi-ticket ensemble works well even with the naive pruning method ( BASE - LT ), RANDOM - LT and ACTIVE - LT achieve the better ensembling effect on average; this suggests the effectiveness of regularizers.
Revised: While multi-ticket ensemble works well even with the naive pruning method ( BASE - LT ), RANDOM - LT and ACTIVE - LT achieve the better ensembling effect on average; this suggests the effectiveness of regularizers.

Original: Interestingly, RANDOM - LT is simpler but more effective than ACTIVE - LT .
Revised: Interestingly, RANDOM - LT is simpler but more effective than ACTIVE - LT .

Original: When Winning Tickets are Less Accurate
Revised: When Winning Tickets are Less Accurate

Original: Does multi-ticket ensemble work well on anytasks?
Revised: Does multi-ticket ensemble work well on any tasks?

Original: The answer is no.
Revised: The answer is no.

Original: To enjoy the beneﬁtfrom multi-ticket ensemble, we have to ﬁnd diversewinning-ticket subnetworks sufﬁciently comparable to their dense network.
Revised: To enjoy the benefit from multi-ticket ensemble, we have to find diverse winning-ticket subnetworks sufficiently comparable to their dense network.

Original: When winning ticketsare less accurate than the baseline, their ensembles often fail to outperform the baseline’s ensemble.
Revised: When winning tickets are less accurate than the baseline, their ensembles often fail to outperform the baseline’s ensemble.

Original: It happened to CoLA (Warstadt et al., 2019),
Revised: It happened to CoLA (Warstadt et al., 2019),

Original: QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al.,2013), MNLI (Williams et al., 2018); the naive iterative magnitude pruning did not ﬁnd comparablewinning-ticket subnetworks (with or sometimeseven without regularizers) 567 .
Revised: QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al., 2013), MNLI (Williams et al., 2018); the naive iterative magnitude pruning did not find comparable winning-ticket subnetworks (with or sometimes even without regularizers) 567 .

Original: Note that, even insuch a case, RANDOM - LT often yielded a highereffect of ensembling (diff.), while the degradationof single models canceled out the effect in total,and BAGGING also failed to improve.
Revised: Note that, even in such a case, RANDOM - LT often yielded a higher effect of ensembling (diff.), while the degradation of single models canceled out the effect in total, and BAGGING also failed to improve.

Original: More sophisticated pruning methods (Blalock et al., 2020;
Revised: More sophisticated pruning methods (Blalock et al., 2020;

Original: Sanh et al., 2020) or tuning will ﬁnd better winningticket subnetworks and maximize the opportunitiesfor multi-ticket ensemble in future work.
Revised: Sanh et al., 2020) or tuning will find better winningticket subnetworks and maximize the opportunities for multi-ticket ensemble in future work.

Original: 4. Diversity of Predictions
Revised: 4. Diversity of Predictions

Original: As an auxiliary analysis of behaviors, we showthat each subnetwork produces diverse predictions.
Revised: As an auxiliary analysis of behaviors, we show that each subnetwork produces diverse predictions.

Original: Because any existing diversity scores donot completely explain or justify the ensemble performance 8 , we discuss only rough trends in ﬁvepopular metrics of classiﬁcation diversity; Q statistic (Yule, 1900), ratio errors (Aksela, 2003), negative double fault (Giacinto and Roli, 2001), disagreement measure (Skalak, 1996), and correlationcoefﬁcient (Kuncheva and Whitaker, 2003).
Revised: Because any existing diversity scores do not completely explain or justify the ensemble performance 8 , we discuss only rough trends in five popular metrics of classification diversity; Q statistic (Yule, 1900), ratio errors (Aksela, 2003), negative double fault (Giacinto and Roli, 2001), disagreement measure (Skalak, 1996), and correlation coefficient (Kuncheva and Whitaker, 2003).

Original: See
Revised: See

Original: Kuncheva and Whitaker (2003); Cruz et al.
Revised: Kuncheva and Whitaker (2003); Cruz et al.

Original: (2020)for their summarized deﬁnitions.
Revised: (2020) for their summarized definitions.

Original: As shown in Table 2, in all the metrics, winning-ticket subnetworks(*- LT ) produced more diverse predictions than the
Revised: As shown in Table 2, in all the metrics, winning-ticket subnetworks (*- LT ) produced more diverse predictions than the

Revised: ., 2020; Liang et al., 2021) reported that they found winning- ticket subnetworks on these tasks, our finding did not contradict it.

Revised: Their were et

Revised: (2020) defined as mances the

Original: [Table]
Revised: [Table]

Original:  Diversity of Subnetwork Structures
Revised: 4. Diversity of Subnetwork Structures

Original: We ﬁnally revealed the diversity of the subnetwork structures on MRPC.
Revised: We finally revealed the diversity of the subnetwork structures on MRPC.

Original: We calculated the overlap ratio of two pruning masks, which is deﬁned as intersection over union, IoU = | m i \
Revised: We calculated the overlap ratio of two pruning masks, which is defined as intersection over union, IoU = | m i ∩ m j | | m i ∪ m j |

Original: m j | | m i [ m j | (Chen et al., 2020).
Revised: (Chen et al., 2020).

Original: In Figure 4, we show the overlap ratio between the pruning masks for the ﬁve random seeds, i.e., { m s 1 , ..., m s 5 } .
Revised: In Figure 4, we show the overlap ratio between the pruning masks for the five random seeds, i.e., { m s 1 , ..., m s 5 } .

Original: At ﬁrst, we can see that ACTIVE - LT and RANDOM - LT using the regularizers resulted in diverse pruning.
Revised: At first, we can see that ACTIVE - LT and RANDOM - LT using the regularizers resulted in diverse pruning.

Original: This higher diversity could lead to the best improvements by ensembling, as discussed in Section 4.1.
Revised: This higher diversity could lead to the best improvements by ensembling, as discussed in Section 4.1.

Original: Secondly,
Revised: Secondly,

Original: BASE - LT produced surprisingly similar (99%) pruning masks with different random seeds.
Revised: BASE - LT produced surprisingly similar (99%) pruning masks with different random seeds.

Original: However, recall that even BASE - LT using the naïve iterative magnitude pruning performed better than BASELINE .
Revised: However, recall that even BASE - LT using the naïve iterative magnitude pruning performed better than BASELINE .

Original: This result shows that even seemingly small changes in structure can improve the diversity of predictions and the performance of the ensemble.
Revised: This result shows that even seemingly small changes in structure can improve the diversity of predictions and the performance of the ensemble.

Original:  Conclusion
Revised:  Conclusion

Original: We raised a question on difﬁculty of ensembling large-scale pretrained models.
Revised: We raised a question on difficulty of ensembling large-scale pretrained models.

Original: As an efﬁcient remedy, we explored methods to use subnetworks in a single model.
Revised: As an efficient remedy, we explored methods to use subnetworks in a single model.

Original: We empirically demonstrated that ensembling winning-ticket subnetworks could outperform the dense ensembles via diversiﬁcation and indicated a limitation too. 
Revised: We empirically demonstrated that ensembling winning-ticket subnetworks could outperform the dense ensembles via diversification and indicated a limitation too. A The Setting of Fine-tuning

Revised: We follow the setting of Chen et al.

Revised: (2020)’s implementation; epoch: 3, initial learning rate: 2ewith linear decay, maximum sequence length: 128, batch size: 32, dropout probability: 0.1.

Revised: This is one of the most-used settings for finetuning a BERT; e.g., the example of finetuning in the Transformers library (Wolf et al., 2020) uses the setting 9 .

Revised: We did not prune the embedding layer, following

Revised: Chen et al.

Revised: (2020); Prasanna et al.

Revised: The coefficient of L 1 regularizer, τ , is decayed using the same scheduler as the learning rate.

Revised: We tuned it on MRPC and used it for other tasks.

Revised: B The Learning Rate Scheduler of Chen et al. (2020)

Revised: Our implementation used in the experiments are derived from Chen et al.

Revised: (2020)’s implementation 10 .

Revised: However, we found a bug in Chen et al.

Revised: (2020)’s implementation on GitHub.

Revised: Thus, we fixed it and experimented with the correct version.

Revised: In their implementation, the learning rate schedule did not follow the common setting and the description mentioned in the paper; ‘We use standard implementations and hyperparameters [49].

Revised: Learning rate decays linearly from initial value to zero’ .

Revised: Specifically, the learning rate with linear decay did not reach zero but was at significant levels even at the end of the finetuning.

Revised: Our implementation corrected it so that it did reach zero as specified in their paper and in the common setting.

Revised: C The Combinations of Ensembles

Revised: In the experiments, we first prepared twenty random seeds and split them into two groups, each of which trained ten models.

Revised: For stabilizing the measurement of the result, we exhaustively evaluated all the possible combinations of ensembles (i.e., depending on the number of members, 10 C 2 , 10 C 3 , 10 C 4 , 10 C 5 patterns, respectively) among the ten models for each group, and averaged the results with the two groups.

Revised: The performance of the members is also averaged over all the seeds.

Revised: [Table]

Revised: D The Results with RoBERTa

Revised: We simply conducted supplementary experiments with RoBERTa (Liu et al., 2019)

Revised: (robeta-base model), although optimal hyperparameters were not searched well.

Revised: The results were similar to the cases of base-base-uncased.

Revised: The patterns can be categorized into the three.

Revised: First, multi-ticket ensembles worked well with roberta on MRPC, as shown in Table 3.

Revised: Secondly, accurate winningticket subnetworks were not found on CoLA and

Revised: QNLI.

Revised: Although the effect of ensembleing was improved after pruning, each single model got worse and the final ensemble accuracy did not outperform the dense baseline.

Revised: Thirdly, although accurate winning-ticket subnetworks were found on STS-B and SST-2, regularizations worsened single-model performances.

Revised: While this case also improved the effect of ensembling, the final accuracy did not outperform the baseline.

Revised: These experiments further emphasized the importance of development of more sophisticated pruning methods without sacrifice of model performances in the context of the lottery ticket hypothesis.

Revised: E Related Work

Revised: Some concurrent studies also investigate the usage of subnetworks for ensembles.

Revised: Gal and Ghahramani (2016) is a pioneer to use subnetwork ensemble.

Revised: A trained neural network with dropout can infer with many different subnetworks, and their ensemble can be used for uncertainty estimation, which is called MC-dropout.

Revised: Durasov et al.

Revised: (2021) improved the efficiency of MC-dropout by exploring subnetworks.

Revised: Zhang et al.

Revised: (2021) (unpublished) experimented with an ensemble of subnetworks of different structures and initialization when trained from scratch, while the improvements possibly could be due to regularization of each single model.

Revised: Havasi et al.

Revised: (2021) is a similar but more elegant approach, which does not explicitly identify subnetworks.

Revised: Instead, it trains a single dense model with training using multi-input multi-output inference; the optimization can implicitly find multiple disentangled subnetworks in the dense model during optimization from random initialization.

Revised: These studies support our assumption that different subnetworks can improve ensemble by diversity.

Revised: Some other directions for introducing diversity exist, while most are unstable.

Revised: Promising directions are to use entropy (Pang et al., 2019) or adversarial training (Rame and Cord, 2021).

Revised: Although they required complex optimization processes, they improved the robustness or ensemble performance on small image recognition datasets.

Revised: Recently, concurrent work (Sellam et al., 2022;

Revised: Tay et al., 2022) provide multiple BERT or T5 models pretrained from different seeds for investigation of seed dependency using large-scale computational resources.

Revised: Further research with the models and computational resources will be helpful for more solid comparison and analysis.

Revised: Note that no prior work tackled the problem of ensembles from a pre-trained model.

Revised: Framing the problem is one of the contributions of this paper.

Revised: Secondly, our multi-ticket ensemble based on random masking enables an independently parallelizable training while existing methods require a sequential processing or a grouped training procedure.

Revised: Finally, multi-ticket ensemble can be combined with other methods, which can improve the total performance together.