Original: Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Revised: Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model

Original: Preferred Networks, Inc.
Revised: Preferred Networks, Inc.

Original: RIKENsosk@preferred.jp shun.kiyono@riken.jp jun.suzuki@tohoku.ac.jp inui@ecei.tohoku.ac.jp
Revised: RIKENsosk@preferred.jp shun.kiyono@riken.jp jun.suzuki@tohoku.ac.jp inui@ecei.tohoku.ac.jp

Original: Abstract
Revised: Abstract

Original: Ensembling is a popular method used to improve performance as a last resort.
Revised: Ensembling is a popular method used to improve performance as a last resort.

Original: However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members.
Revised: However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members.

Original: This paper proposes Multi-Ticket Ensemble , which finetunes different subnetworks of a single pretrained model and ensembles them.
Revised: This paper proposes Multi-Ticket Ensemble , which finetunes different subnetworks of a single pretrained model and ensembles them.

Original: We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.
Revised: We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks, and their ensemble outperformed the standard ensemble on some tasks.

Original:  Introduction
Revised:  Introduction

Original: Ensembling (Levin et al., 1989; Domingos, 1997) has long been an easy and effective approach to improve model performance by averaging the outputs of multiple comparable but independent models.
Revised: Ensembling (Levin et al., 1989; Domingos, 1997) has long been an easy and effective approach to improve model performance by averaging the outputs of multiple comparable but independent models.

Original: Allen-Zhu and Li (2020) explain that different models obtain different views for judgments, and the ensemble uses complementary views to make more robust decisions.
Revised: Allen-Zhu and Li (2020) explain that different models obtain different views for judgments, and the ensemble uses complementary views to make more robust decisions.

Original: A good ensemble requires diverse member models.
Revised: A good ensemble requires diverse member models.

Original: However, how to encourage diversity without sacrificing the accuracy of each model is non-trivial (Liu and Yao, 1999; Kirillov et al., 2016; Rame and Cord, 2021).
Revised: However, how to encourage diversity without sacrificing the accuracy of each model is non-trivial (Liu and Yao, 1999; Kirillov et al., 2016; Rame and Cord, 2021).

Original: The pretrain-then-finetune paradigm has become another best practice for achieving state-of-the-art performance on NLP tasks (Devlin et al., 2019).
Revised: The pretrain-then-finetune paradigm has become another best practice for achieving state-of-the-art performance on NLP tasks (Devlin et al., 2019).

Original: The cost of large-scale pretraining, however, is enormously high (Sharir et al., 2020); This often makes it difficult to independently pretrain multiple models.
Revised: The cost of large-scale pretraining, however, is enormously high (Sharir et al., 2020); This often makes it difficult to independently pretrain multiple models.

Original: Therefore, most researchers and practitioners only use a single pretrained model, which is distributed by resource-rich organizations.
Revised: Therefore, most researchers and practitioners only use a single pretrained model, which is distributed by resource-rich organizations.

Original: This situation brings up a novel question to ensemble learning: Can we make an effective ensemble from only a single pre-trained model ?
Revised: This situation brings up a novel question to ensemble learning: Can we make an effective ensemble from only a single pre-trained model ?

Original: Although ensembles can be combined with the pretrain-then-finetune paradigm, an ensemble of models finetuned from a single pretrained model is much less effective than that using different pretrained models from scratch in many tasks (Raffel et al., 2020).
Revised: Although ensembles can be combined with the pretrain-then-finetune paradigm, an ensemble of models finetuned from a single pretrained model is much less effective than that using different pretrained models from scratch in many tasks (Raffel et al., 2020).

Original: Naïve ensemble offers limited improvements, possibly due to the lack of diversity of finetuning from the same initial parameters.
Revised: Naïve ensemble offers limited improvements, possibly due to the lack of diversity of finetuning from the same initial parameters.

Original: In this paper, we propose a simple yet effective method called Multi-Ticket Ensemble , ensembling finetuned winning-ticket subnetworks (Frankle and
Revised: In this paper, we propose a simple yet effective method called Multi-Ticket Ensemble , ensembling finetuned winning-ticket subnetworks (Frankle and

Original: Carbin, 2019) in a single pretrained model.
Revised: Carbin, 2019) in a single pretrained model.

Original: We empirically demonstrate that pruning a single pretrained model can make diverse models, and their ensemble can outperform the naïve dense ensemble if winning-ticket subnetworks are found.
Revised: We empirically demonstrate that pruning a single pretrained model can make diverse models, and their ensemble can outperform the naïve dense ensemble if winning-ticket subnetworks are found.

Original:  Diversity in a Single Pretrained Model
Revised:  Diversity in a Single Pretrained Model

Original: In this paper, we discuss the most standard way of ensemble, which averages the outputs of multiple neural networks; each has the same architecture but different parameters.
Revised: In this paper, we discuss the most standard way of ensemble, which averages the outputs of multiple neural networks; each has the same architecture but different parameters.

Original: That is, let f ( x ; θ ) be the output of a model with the parameter vector
Revised: That is, let f ( x ; θ ) be the output of a model with the parameter vector

Original: θ given the input x , the output of an ensemble is f M ( x )
Revised: θ given the input x , the output of an ensemble is f M ( x )

Original: = (cid:80) θ ∈M f
Revised: = (cid:80) θ ∈M f

Original: ( x ; θ ) / |M| , where M = { θ 1 , ..., θ |M| } is the member parameters.
Revised: ( x ; θ ) / |M| , where M = { θ 1 , ..., θ |M| } is the member parameters.

Original: 2. Diversity from Finetuning
Revised: 2. Diversity from Finetuning

Original: As discussed, when constructing an ensemble f M by finetuning from a single pretrained model multiple times with different random seeds { s 1 , ..., s |M| } , the boost in performance tends to be only marginal.
Revised: As discussed, when constructing an ensemble f M by finetuning from a single pretrained model multiple times with different random seeds { s 1 , ..., s |M| } , the boost in performance tends to be only marginal.

Original: In the case of BERT (Devlin et al., 2019) and its variants, three sources of diversities can be considered: random initialization of the task-specific layer, dataset shuffling for stochastic gradient descent (SGD), and dropout.
Revised: In the case of BERT (Devlin et al., 2019) and its variants, three sources of diversities can be considered: random initialization of the task-specific layer, dataset shuffling for stochastic gradient descent (SGD), and dropout.

Original: However, empirically, such finetuned parameters tend not to be largely different from the initial parameters, and they do not lead to diverse models (Radiya-Dixit and Wang, 2020).
Revised: However, empirically, such finetuned parameters tend not to be largely different from the initial parameters, and they do not lead to diverse models (Radiya-Dixit and Wang, 2020).

Original: Of course, if one adds significant noise to the parameters, it leads to diversity; however, it would also hurt accuracy.
Revised: Of course, if one adds significant noise to the parameters, it leads to diversity; however, it would also hurt accuracy.

Original: 2. Diversity from Pruning
Revised: 2. Diversity from Pruning

Original: To make models ensuring both accuracy and diversity, we focus on subnetworks in the pretrained model.
Revised: To make models ensuring both accuracy and diversity, we focus on subnetworks in the pretrained model.

Original: Different subnetworks employ different subspaces of the pre-trained knowledge (RadiyaDixit and Wang, 2020; Zhao et al., 2020; Cao et al., 2021); this would help the subnetworks to acquire different views, which can be a source of desired diversity 1 .
Revised: Different subnetworks employ different subspaces of the pre-trained knowledge (RadiyaDixit and Wang, 2020; Zhao et al., 2020; Cao et al., 2021); this would help the subnetworks to acquire different views, which can be a source of desired diversity 1 .

Original: Also, in terms of accuracy, recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that a dense network at initialization contains a subnetwork, called the winning ticket , whose accuracy becomes comparable to that of the dense one after the same training.
Revised: Also, in terms of accuracy, recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that a dense network at initialization contains a subnetwork, called the winning ticket , whose accuracy becomes comparable to that of the dense one after the same training.

Original: Interestingly, the pretrained BERT also has a winning ticket for finetuning on downstream tasks (Chen et al., 2020).
Revised: Interestingly, the pretrained BERT also has a winning ticket for finetuning on downstream tasks (Chen et al., 2020).

Original: Thus, if we can find diverse winning tickets , they can be good ensemble members with the two desirable properties: diversity and accuracy.
Revised: Thus, if we can find diverse winning tickets , they can be good ensemble members with the two desirable properties: diversity and accuracy.

Original:  Subnetwork Exploration
Revised:  Subnetwork Exploration

Original: We propose a simple yet effective method, multiticket ensemble , which finetunes different subnetworks instead of dense networks.
Revised: We propose a simple yet effective method, multiticket ensemble , which finetunes different subnetworks instead of dense networks.

Original: Because it could be a key how to find subnetworks, we explore three variants based on iterative magnitude pruning.
Revised: Because it could be a key how to find subnetworks, we explore three variants based on iterative magnitude pruning.

Original: [Table]
Revised: [Table]

Original: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.
Revised: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.

Original: Diversity of Finetuning from a Single
Revised: Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 , ..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 , ..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 , ..., s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 , ..., s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 114 First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 114 First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ , s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ , s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent the parameter of the pretrained BERT and the task-speciﬁc layer, whose parameter is randomly initialized by random seed s .
Revised: Also, let ✓ s represent the parameter of the pretrained BERT and the task-speciﬁc layer, whose parameter is randomly initialized by random seed s .

Original: After ﬁnetuning ✓ s to FINE ( ✓ s , s ) , we identify and prune the parameters with 10% lowest magnitudes in
Revised: After ﬁnetuning ✓ s to FINE ( ✓ s , s ) , we identify and prune the parameters with 10% lowest magnitudes in

Original: FINE ( ✓ s , s ) .
Revised: FINE ( ✓ s , s ) .

Original: We also get the corresponding binary mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where the surviving positions have 1 otherwise 0.

Original: The pruning of parameters ✓ by a mask m can be also represented as ✓  m , where  is the element-wise product.
Revised: The pruning of parameters ✓ by a mask m can be also represented as ✓  m , where  is the element-wise product.

Original: Next, we replay ﬁnetuning but from ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% , s ) as well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% , s ) as well as 20%-pruning mask m s, 20% .

Original: By repeating iterative magnitude pruning, we obtain the parameter FINE
Revised: By repeating iterative magnitude pruning, we obtain the parameter FINE

Original: ( ✓ s  m s,P % , s ) .
Revised: ( ✓ s  m s,P % , s ) .

Original: In our experiments, we set P = 30 , i.e., evaluate ensemble of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% , s 1 ) , ..., FINE ( ✓ s
Revised: In our experiments, we set P = 30 , i.e., evaluate ensemble of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% , s 1 ) , ..., FINE ( ✓ s

Original: [Table]
Revised: [Table]

Original: 2 We also did
Revised: 2 We also did

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: ) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd them

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned Figure 2: Overview of iterative magnitude pruning.
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned Figure 2: Overview of iterative magnitude pruning.

Original: efﬁciently for better ensembles.
Revised: efﬁciently for better ensembles.

Original: In this paper, we propose methods to ﬁnd di-
Revised: In this paper, we propose methods to ﬁnd di-

Original: 105 verse sub
Revised: 105 verse sub

Original: -networks.
Revised: -networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub- 109 networks by pruning random parameters at initial- 110 ization, the accuracy becomes signiﬁcantly deterio- 111 rated
Revised: For example, 108 if we aggressively realize the diversity of the sub- 109 networks by pruning random parameters at initial- 110 ization, the accuracy becomes signiﬁcantly deterio- 111 rated

Original: (Prasanna et al., 2020).
Revised: (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub- 116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub- 116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and thetask-speciﬁc layer, whose parameter is randomlyinitialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and thetask-speciﬁc layer, whose parameter is randomlyinitialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune theparameters with 10% lowest magnitudes in
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune theparameters with 10% lowest magnitudes in

Original: FINE ( ✓ s ,s ) .
Revised: FINE ( ✓ s ,s ) .

Original: We also get the corresponding binarymask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binarymask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: Thepruning of parameters ✓ by a mask m can be alsorepresented as ✓  m , where  is the element-wise 132 product.
Revised: Thepruning of parameters ✓ by a mask m can be alsorepresented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeatingiterative magnitude pruning, we obtain theparameter FINE
Revised: By repeatingiterative magnitude pruning, we obtain theparameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper- 137 iments, we set P = 30 , i.e., evaluate ensembleof 30%-pruning sub-networks, where M ={ FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M| 2 We also did not prune the embedding layer, following Chen et al.
Revised: In our exper- 137 iments, we set P = 30 , i.e., evaluate ensembleof 30%-pruning sub-networks, where M ={ FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M| 2 We also did not prune the embedding layer, following Chen et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetworkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetworkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- pensinothersettingsgenerally.
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap- pensinothersettingsgenerally.

Original: IntheresultsbyRaffeletal.
Revised: IntheresultsbyRaffeletal.

Original: (2020),wefoundthatthishappenedonalmostallthetasksofGLUE(Wangetal.
Revised: (2020),wefoundthatthishappenedonalmostallthetasksofGLUE(Wangetal.

Original: ,2018),SuperGLUE(Wangetal.
Revised: ,2018),SuperGLUE(Wangetal.

Original: ,2019),SQuAD(Rajpurkaretal.
Revised: ,2019),SQuAD(Rajpurkaretal.

Original: ,2016),summarization,andmachine translationsusingtheT5models.
Revised: ,2016),summarization,andmachine translationsusingtheT5models.

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .

Original: We also get the corresponding binary
Revised: We also get the corresponding binary

Original: [Table]
Revised: [Table]

Original: of pruning, m s,
Revised: of pruning, m s,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: } |
Revised: } |

Original: [Table]
Revised: [Table]

Original: s | , where 129 the surviving positions have 1 otherwise 0.
Revised: s | , where 129 the surviving positions have 1 otherwise 0.

Original: The
Revised: The

Original: [Table]
Revised: [Table]

Original: pruning of parameters ✓
Revised: pruning of parameters ✓

Original: [Table]
Revised: [Table]

Original: a mask m can be
Revised: a mask m can be

Original: [Table]
Revised: [Table]

Original: represented as ✓  m , where  is the element-wise
Revised: represented as ✓  m , where  is the element-wise

Original: [Table]
Revised: [Table]

Original: product.
Revised: product.

Original: Next, we replay ﬁnetuning but from
Revised: Next, we replay ﬁnetuning but from

Original: [Table]
Revised: [Table]

Original: s
Revised: s

Original: [Table]
Revised: [Table]

Original: m s, 10% and get FINE (
Revised: m s, 10% and get FINE (

Original: [Table]
Revised: [Table]

Original: s
Revised: s

Original: [Table]
Revised: [Table]

Original: m s, 10% ,s ) as
Revised: m s, 10% ,s ) as

Original: [Table]
Revised: [Table]

Original: well as 20%-pruning
Revised: well as 20%-pruning

Original: [Table]
Revised: [Table]

Original: m s,
Revised: m s,

Original: [Table]
Revised: [Table]

Original: By repeating
Revised: By repeating

Original: [Table]
Revised: [Table]

Original: iterative
Revised: iterative

Original: [Table]
Revised: [Table]

Original: pruning, we obtain the
Revised: pruning, we obtain the

Original: [Table]
Revised: [Table]

Original: s,
Revised: s,

Original: [Table]
Revised: [Table]

Original: s ) .
Revised: s ) .

Original: In our exper-
Revised: In our exper-

Original: [Table]
Revised: [Table]

Original: iments, we set P = 30 , i.e., evaluate ensemble
Revised: iments, we set P = 30 , i.e., evaluate ensemble

Original: [Table]
Revised: [Table]

Original: of 30%-pruning sub-networks, where
Revised: of 30%-pruning sub-networks, where

Original: [Table]
Revised: [Table]

Original: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random initialization of the task-speciﬁc output layer, (2)dataset shufﬂing for stochastic gradient descent(SGD), and (3) dropout noise.

Original: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empirically, they do not necessarily lead to diverse models, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomes comparable with that of the dense networkafter the same training steps.

Original: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).
Revised: A pretrained BERTalso has sparse sub-networks (e.g., 50%), whichcan achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chenet al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using

Original: [Table]
Revised: [Table]

Original: in
Revised: in

Original: [Table]
Revised: [Table]

Original: paper,
Revised: paper,

Original: [Table]
Revised: [Table]

Original: hap-
Revised: hap-

Original: [Table]
Revised: [Table]

Original: generally.
Revised: generally.

Original: In
Revised: In

Original: [Table]
Revised: [Table]

Original: results by Raffel et al.
Revised: results by Raffel et al.

Original: , we found that this happened on almost
Revised: , we found that this happened on almost

Original: [Table]
Revised: [Table]

Original: of GLUE
Revised: of GLUE

Original: (Wang et al., 2018),
Revised: (Wang et al., 2018),

Original: [Table]
Revised: [Table]

Original: Wang et al.,
Revised: Wang et al.,

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: are
Revised: are

Original: [Table]
Revised: [Table]

Original: In this paper, we propose methods to ﬁnd di105 verse sub-networks.
Revised: In this paper, we propose methods to ﬁnd di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .

Original: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.
Revised: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE
Revised: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M|  2 We also did not prune the embedding layer, following Chen et al.
Revised: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 , 30% ,s 1 ) ,..., FINE ( ✓ s |M|  2 We also did not prune the embedding layer, following Chen et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single
Revised: (2020) the iterative magnitude pruning, and demonstratethat the structurally diversiﬁed sub-networks leadto stronger ensembles.2 Diversity of Finetuning from a Single

Original: Pretrained Model
Revised: Pretrained Model

Original: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.
Revised: In this paper, we discuss the most standard wayof ensembling, which averages the outputs of multiple neural networks.

Original: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .
Revised: If we use models with thesame architecture, e.g., BERT, but with differentparameters, the parameters of the member modelsare represented as M = { ✓ 1 ,..., ✓ |M| } .

Original: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:
Revised: Let x andf ( x ; ✓ ) be an input and the output by f with parameter ✓ respectively, the output of their ensembleis as follows:

Original: f M ( x ) =
Revised: f M ( x ) =

Original: P ✓ 2 M f ( x ; ✓ ) / |M| .
Revised: P ✓ 2 M f ( x ; ✓ ) / |M| .

Original: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.
Revised: As discussed in the introduction, even if we construct ensemble f M by ﬁnetuning multiple modelsfrom a single pretrained model with different random seeds { s 1 ,...,s |M| } , the performance boosttends to be marginal.

Original: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random ini-
Revised: The source of diversitiesin the ﬁnetuned BERTs are the following randomnesses in the ﬁnetuning process: (1) random ini-

Original: dataset for
Revised: dataset for

Original: (SGD), and (3) dropout noise.
Revised: (SGD), and (3) dropout noise.

Original: However, empiri- cally, they do not necessarily lead to diverse mod- els, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.
Revised: However, empiri- cally, they do not necessarily lead to diverse mod- els, as Radiya-Dixit and Wang (2020) reported theﬁnetuned parameters could not be far away.

Original: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.
Revised: For diversifying models more, we propose tointroduce a novel randomness, (4) pruning of thepretrained model before each ﬁnetuning.

Original: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.
Revised: We expect that, during ﬁnetuning, each sub-network acquires different views using different sub-spaces ofthe pretrained knowledge.

Original: This idea has two challenges: the diversity and the accuracy of the subnetworks.
Revised: This idea has two challenges: the diversity and the accuracy of the subnetworks.

Original: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetwork
Revised: Recent studies on the lottery ticket hypothesis (Frankle and Carbin, 2019) suggest that adense neural network at an initialization contains asub-network, called winning ticket , whose accuracybecomescomparablewiththatofthedensenetwork

Original: [Table]
Revised: [Table]

Original: after the same training steps.
Revised: after the same training steps.

Original: A pretrained BERT
Revised: A pretrained BERT

Original: [Table]
Revised: [Table]

Original: also has sparse sub-networks (e.g., 50%), which
Revised: also has sparse sub-networks (e.g., 50%), which

Original: [Table]
Revised: [Table]

Original: can achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chen
Revised: can achieve the same accuracy with the entire network when ﬁnetuning on downstream tasks (Chen

Original: [Table]
Revised: [Table]

Original: et al., 2020).
Revised: et al., 2020).

Original: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap-
Revised: However, it is still unclear how diverse winning tickets exist and how to ﬁnd themtasks using the BERT in this paper, the same problem hap-

Original: [Table]
Revised: [Table]

Original: IntheresultsbyRaffeletal.
Revised: IntheresultsbyRaffeletal.

Original: [Table]
Revised: [Table]

Original: ,wefoundthatthishappenedonalmostallthetasksofGLUE
Revised: ,wefoundthatthishappenedonalmostallthetasksofGLUE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R
Revised: 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 P r e - t r a i ned R

Original: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.
Revised: ando m 0.2 -0.3 0.7 1.1 0.5 -0.3 -0.1 0.4 0.9 0.4 2.9 0.3 0.3 -1.3 -0.6 0.5 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 -0.1 0.2 1.3 0.1 -0.

Original: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t
Revised: 0.1 0.8 0.3 2.8 0.2 1.4 -1.2 -0.5 0.4 W e i gh t m a t

Original: r
Revised: r

Original: i x -0.1 0.2 1.3 0.6 -0.
Revised: i x -0.1 0.2 1.3 0.6 -0.

Original: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned propose methods to di105 verse sub-networks.
Revised: 0.2 0.9 0.4 2.2 0.1 0.1 -1.1 -0.6 0.6 0 1 1 1 1 1 0 1 1 1 1 0 0 1 1 1 Repeated until P% are pruned propose methods to di105 verse sub-networks.

Original: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.
Revised: Again, the key difﬁculty is the 106 tradeoff between the accuracy of a single model 107 and the diversity of the members.

Original: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).
Revised: For example, 108 if we aggressively realize the diversity of the sub109 networks by pruning random parameters at initial110 ization, the accuracy becomes signiﬁcantly deterio111 rated (Prasanna et al., 2020).

Original: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning 

Revised: 112 3 Methods for Finding Sub-networks 113 3.1 Iterative Magnitude Pruning

Original: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .
Revised: First, we describe iterative magnitude pruning, the 115 common method of ﬁnding a winning ticket sub116 networks, exactly following Frankle and Carbin 117 (2019)’s procedure 2 .

Original: We experiment with several 118 variants on the basis of this method.
Revised: We experiment with several 118 variants on the basis of this method.

Original: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .
Revised: 119 Let FINE ( ✓ ,s ) be the new parameters, which 120 are ﬁnetuned from ✓ using SGD depending 121 on a random seed s .

Original: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .
Revised: Also, let ✓ s represent 122 the parameter of the pretrained BERT and the 123 task-speciﬁc layer, whose parameter is randomly 124 initialized by random seed s .

Original: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .
Revised: After ﬁnetuning 125 ✓ s to FINE ( ✓ s ,s ) , we identify and prune the 126 parameters with 10% lowest magnitudes in 127 FINE ( ✓ s ,s ) .

Original: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.
Revised: We also get the corresponding binary 128 mask of pruning, m s, 10% 2 { 0 , 1 } | ✓ s | , where 129 the surviving positions have 1 otherwise 0.

Original: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.
Revised: The 130 pruning of parameters ✓ by a mask m can be also 131 represented as ✓  m , where  is the element-wise 132 product.

Original: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .
Revised: Next, we replay ﬁnetuning but from 133 ✓ s  m s, 10% and get FINE ( ✓ s  m s, 10% ,s ) as 134 well as 20%-pruning mask m s, 20% .

Original: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE
Revised: By repeating 135 iterative magnitude pruning, we obtain the 136 parameter FINE

Original: ( ✓ s  m s,P % ,s ) .
Revised: ( ✓ s  m s,P % ,s ) .

Original: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 ,
Revised: In our exper137 iments, we set P = 30 , i.e., evaluate ensemble 138 of 30%-pruning sub-networks, where M = { FINE ( ✓ s 1  m s 1 ,

Original: [Figure]
Revised: [Figure]

Original: We can also use regularizers during finetuning to diversify pruning (Section 3.2).
Revised: We can also use regularizers during finetuning to diversify pruning (Section 3.2).

Original: 3. Iterative Magnitude Pruning
Revised: 3. Iterative Magnitude Pruning

Original: We employ iterative magnitude pruning (Frankle and Carbin, 2019) to find winning tickets for simplicity.
Revised: We employ iterative magnitude pruning (Frankle and Carbin, 2019) to find winning tickets for simplicity.

Original: Other sophisticated options are left for future work.
Revised: Other sophisticated options are left for future work.

Original: Here, we explain the algorithm (refer to the paper for details).
Revised: Here, we explain the algorithm (refer to the paper for details).

Original: The algorithm explores a good pruning mask via rehearsals of finetuning.
Revised: The algorithm explores a good pruning mask via rehearsals of finetuning.

Original: First, it completes a finetuning procedure of an initialized dense network and identifies the parameters with the 10% lowest magnitudes as the targets of pruning.
Revised: First, it completes a finetuning procedure of an initialized dense network and identifies the parameters with the 10% lowest magnitudes as the targets of pruning.

Original: Then, it makes the pruned subnetwork and resets its parameters to the originally-initialized (sub-)parameters.
Revised: Then, it makes the pruned subnetwork and resets its parameters to the originally-initialized (sub-)parameters.

Original: This finetune-prune-reset process is repeated until reaching the desired pruning ratio.
Revised: This finetune-prune-reset process is repeated until reaching the desired pruning ratio.

Original: We used 30% as pruning ratio.
Revised: We used 30% as pruning ratio.

Original: 3. Pruning with Regularizer
Revised: 3. Pruning with Regularizer

Original: We discussed that finetuning with different random seeds did not lead to diverse parameters in Section 2.1.
Revised: We discussed that finetuning with different random seeds did not lead to diverse parameters in Section 2.1.

Original: Therefore, iterative magnitude pruning with different seeds could also produce less diverse subnetworks.
Revised: Therefore, iterative magnitude pruning with different seeds could also produce less diverse subnetworks.

Original: Thus, we also explore means of diversifying pruning patterns by enforcing different parameters to have lower magnitudes.
Revised: Thus, we also explore means of diversifying pruning patterns by enforcing different parameters to have lower magnitudes.

Original: Motivated by this, we experiment with a simple approach, applying an L 1 regularizer (i.e., magnitude decay) to different parameters selectively depending on the random seeds.
Revised: Motivated by this, we experiment with a simple approach, applying an L 1 regularizer (i.e., magnitude decay) to different parameters selectively depending on the random seeds.

Original: Specifically, we explore two policies to determine which parameters are decayed and how strongly they are, i.e., the element-wise coefficients of the L 1 regularizer, l s ∈ R ≥ 

Revised: Specifically, we explore two policies to determine which parameters are decayed and how strongly they are, i.e., the element-wise coefficients of the L 1 regularizer, l s ∈ R ≥

Original: | θ | .
Revised: | θ | .

Original: During finetuning (for pruning), we add a regularization term τ
Revised: During finetuning (for pruning), we add a regularization term τ

Original: || θ s ⊙ l s || 1 with a positive scalar coefficient τ into the loss of the task (e.g., cross entropy for classification), where ⊙ is element-wise product.
Revised: || θ s ⊙ l s || 1 with a positive scalar coefficient τ into the loss of the task (e.g., cross entropy for classification), where ⊙ is element-wise product.

Original: This softly enforces various parameters to have a lower magnitude among a set of random seeds and could lead various parameters to be pruned.
Revised: This softly enforces various parameters to have a lower magnitude among a set of random seeds and could lead various parameters to be pruned.

Original: Active Masking
Revised: Active Masking

Original: To maximize the diversity of the surviving parameters of member models, it is necessary to prune the surviving parameters of the random seed s 1 when building a model with the next random seed s 2 .
Revised: To maximize the diversity of the surviving parameters of member models, it is necessary to prune the surviving parameters of the random seed s 1 when building a model with the next random seed s 2 .

Original: Thus, during finetuning with seed s 2 , we apply the L 1 regularizer on the first surviving parameters.
Revised: Thus, during finetuning with seed s 2 , we apply the L 1 regularizer on the first surviving parameters.

Original: Likewise, with the following seeds s 3 , s 4 , ..., s i , ..., s |M| , we cumulatively use the average of the surviving masks as the regularizer coefficient mask.
Revised: Likewise, with the following seeds s 3 , s 4 , ..., s i , ..., s |M| , we cumulatively use the average of the surviving masks as the regularizer coefficient mask.

Original: Let m s j ∈ { 0 , 1 } | θ | be the pruning mask indicating surviving parameters from seed s j , the coefficient mask with seed
Revised: Let m s j ∈ { 0 , 1 } | θ | be the pruning mask indicating surviving parameters from seed s j , the coefficient mask with seed

Original: s i is l s i
Revised: s i is l s i

Original: = (cid:80)
Revised: = (cid:80)

Original: j<i m s j / ( i − 1) .
Revised: j<i m s j / ( i − 1) .

Original: We call this affirmative policy as active masking .
Revised: We call this affirmative policy as active masking .

Original: Random Masking
Revised: Random Masking

Original: In active masking, each coefficient mask has a sequential dependence on the preceding random seeds.
Revised: In active masking, each coefficient mask has a sequential dependence on the preceding random seeds.

Original: Thus, the training of ensemble members cannot be parallelized.
Revised: Thus, the training of ensemble members cannot be parallelized.

Original: Therefore, we also experiment with a simpler and parallelizable variant, random masking , where a mask is independently and randomly generated from a random seed.
Revised: Therefore, we also experiment with a simpler and parallelizable variant, random masking , where a mask is independently and randomly generated from a random seed.

Original: With a random seed s i , we generate the seed-dependent random binary mask, i.e., l s = m rand s i ∈ { 0 , 1 } | θ | , where each element is sampled from Bernoulli distribution and 0’s probability equals to the target pruning ratio.
Revised: With a random seed s i , we generate the seed-dependent random binary mask, i.e., l s = m rand s i ∈ { 0 , 1 } | θ | , where each element is sampled from Bernoulli distribution and 0’s probability equals to the target pruning ratio.

Original:  Experiments
Revised:  Experiments

Original: We evaluate the performance of ensembles using four finetuning schemes: (1) finetuning without pruning ( BASELINE ), (2) finetuning of lotteryticket subnetworks found with the naïve iterative magnitude pruning ( BASE - LT ), and (3) with L 1 regularizer by the active masking ( ACTIVE - LT ) or (4) random masking ( RANDOM - LT ).
Revised: We evaluate the performance of ensembles using four finetuning schemes: (1) finetuning without pruning ( BASELINE ), (2) finetuning of lotteryticket subnetworks found with the naïve iterative magnitude pruning ( BASE - LT ), and (3) with L 1 regularizer by the active masking ( ACTIVE - LT ) or (4) random masking ( RANDOM - LT ).

Original: We also compare with (5) BAGGING -based ensemble, which trains dense models on different random 90% training subsets.
Revised: We also compare with (5) BAGGING -based ensemble, which trains dense models on different random 90% training subsets.

Original: We use the GLUE benchmark (Wang et al., 2018) as tasks.
Revised: We use the GLUE benchmark (Wang et al., 2018) as tasks.

Original: The implementation and settings follow Chen et al.
Revised: The implementation and settings follow Chen et al.

Original: (2020) 2 using the Transformers library (Wolf et al., 2020) and its bert-baseuncased pretrained model.
Revised: (2020) 2 using the Transformers library (Wolf et al., 2020) and its bert-baseuncased pretrained model.

Original: We report the average performance using twenty different random seeds.
Revised: We report the average performance using twenty different random seeds.

Original: Ensembles are evaluated using exhaustive combinations of five members.
Revised: Ensembles are evaluated using exhaustive combinations of five members.

Original: We also perform Student’s t-test for validating statistical significance 3 .
Revised: We also perform Student’s t-test for validating statistical significance 3 .

Original: Note
Revised: Note

Original: [Table]
Revised: [Table]

Original: that, while the experiments focus on using BERT, we believe that the insights would be helpful to other pretrain-then-finetune settings in general 4 .
Revised: that, while the experiments focus on using BERT, we believe that the insights would be helpful to other pretrain-then-finetune settings in general 4 .

Original: 4. Accuracy
Revised: 4. Accuracy

Original: We show the results on MRPC (Dolan and Brockett, 2005) and STS-B (Cer et al., 2017) in Table 1.
Revised: We show the results on MRPC (Dolan and Brockett, 2005) and STS-B (Cer et al., 2017) in Table 1.

Original: Multi-ticket ensembles (*- LT ) outperform BASELINE and BAGGING significantly ( p < 0 .
Revised: Multi-ticket ensembles (*- LT ) outperform BASELINE and BAGGING significantly ( p < 0 .

Original: This result supports the effectiveness of multi-ticket ensemble.
Revised: This result supports the effectiveness of multi-ticket ensemble.

Original: Note that the improvements of *- LT are attributable to ensembling (diff.) rather than to any performance gains of the individual models (single).
Revised: Note that the improvements of *- LT are attributable to ensembling (diff.) rather than to any performance gains of the individual models (single).

Original: We also plot the improvements (ens. values relative to BASELINE ) as a function of the number of ensemble members on MRPC and STS-B in Figure 3.
Revised: We also plot the improvements (ens. values relative to BASELINE ) as a function of the number of ensemble members on MRPC and STS-B in Figure 3.

Original: This also clearly shows that while the single models of *- LT have accuracy similar to BASELINE , the gains appear when ensembling them.
Revised: This also clearly shows that while the single models of *- LT have accuracy similar to BASELINE , the gains appear when ensembling them.

Original: While multi-ticket ensemble works well even with the naive pruning method ( BASE - LT ), RANDOM - LT and ACTIVE - LT achieve the better ensembling effect on average; this suggests the effectiveness of regularizers.
Revised: While multi-ticket ensemble works well even with the naive pruning method ( BASE - LT ), RANDOM - LT and ACTIVE - LT achieve the better ensembling effect on average; this suggests the effectiveness of regularizers.

Original: Interestingly, RANDOM - LT is simpler but more effective than ACTIVE - LT .
Revised: Interestingly, RANDOM - LT is simpler but more effective than ACTIVE - LT .

Original: When Winning Tickets are Less Accurate
Revised: When Winning Tickets are Less Accurate

Original: Does multi-ticket ensemble work well on any tasks?
Revised: Does multi-ticket ensemble work well on any tasks?

Original: The answer is no.
Revised: The answer is no.

Original: To enjoy the benefit from multi-ticket ensemble, we have to find diverse winning-ticket subnetworks sufficiently comparable to their dense network.
Revised: To enjoy the benefit from multi-ticket ensemble, we have to find diverse winning-ticket subnetworks sufficiently comparable to their dense network.

Original: When winning tickets are less accurate than the baseline, their ensembles often fail to outperform the baseline’s ensemble.
Revised: When winning tickets are less accurate than the baseline, their ensembles often fail to outperform the baseline’s ensemble.

Original: It happened to CoLA (Warstadt et al., 2019),
Revised: It happened to CoLA (Warstadt et al., 2019),

Original: QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al., 2013), MNLI (Williams et al., 2018); the naive iterative magnitude pruning did not find comparable winning-ticket subnetworks (with or sometimes even without regularizers) 567 .
Revised: QNLI (Rajpurkar et al., 2016), SST-2 (Socher et al., 2013), MNLI (Williams et al., 2018); the naive iterative magnitude pruning did not find comparable winning-ticket subnetworks (with or sometimes even without regularizers) 567 .

Original: Note that, even in such a case, RANDOM - LT often yielded a higher effect of ensembling (diff.), while the degradation of single models canceled out the effect in total, and BAGGING also failed to improve.
Revised: Note that, even in such a case, RANDOM - LT often yielded a higher effect of ensembling (diff.), while the degradation of single models canceled out the effect in total, and BAGGING also failed to improve.

Original: More sophisticated pruning methods (Blalock et al., 2020;
Revised: More sophisticated pruning methods (Blalock et al., 2020;

Original: Sanh et al., 2020) or tuning will find better winningticket subnetworks and maximize the opportunities for multi-ticket ensemble in future work.
Revised: Sanh et al., 2020) or tuning will find better winningticket subnetworks and maximize the opportunities for multi-ticket ensemble in future work.

Original: 4. Diversity of Predictions
Revised: 4. Diversity of Predictions

Original: As an auxiliary analysis of behaviors, we show that each subnetwork produces diverse predictions.
Revised: As an auxiliary analysis of behaviors, we show that each subnetwork produces diverse predictions.

Original: Because any existing diversity scores do not completely explain or justify the ensemble performance 8 , we discuss only rough trends in five popular metrics of classification diversity; Q statistic (Yule, 1900), ratio errors (Aksela, 2003), negative double fault (Giacinto and Roli, 2001), disagreement measure (Skalak, 1996), and correlation coefficient (Kuncheva and Whitaker, 2003).
Revised: Because any existing diversity scores do not completely explain or justify the ensemble performance 8 , we discuss only rough trends in five popular metrics of classification diversity; Q statistic (Yule, 1900), ratio errors (Aksela, 2003), negative double fault (Giacinto and Roli, 2001), disagreement measure (Skalak, 1996), and correlation coefficient (Kuncheva and Whitaker, 2003).

Original: See
Revised: See

Original: Kuncheva and Whitaker (2003); Cruz et al.
Revised: Kuncheva and Whitaker (2003); Cruz et al.

Original: (2020) for their summarized definitions.
Revised: (2020) for their summarized definitions.

Original: As shown in Table 2, in all the metrics, winning-ticket subnetworks (*- LT ) produced more diverse predictions than the
Revised: As shown in Table 2, in all the metrics, winning-ticket subnetworks (*- LT ) produced more diverse predictions than the

Original: ., 2020; Liang et al., 2021) reported that they found winning- ticket subnetworks on these tasks, our finding did not contradict it.
Revised: ., 2020; Liang et al., 2021) reported that they found winning- ticket subnetworks on these tasks, our finding did not contradict it.

Original: Their were et
Revised: Their were et

Original: (2020) defined as mances the
Revised: (2020) defined as mances the

Original: [Table]
Revised: [Table]

Original: 4. Diversity of Subnetwork Structures
Revised: 4. Diversity of Subnetwork Structures

Original: We finally revealed the diversity of the subnetwork structures on MRPC.
Revised: We finally revealed the diversity of the subnetwork structures on MRPC.

Original: We calculated the overlap ratio of two pruning masks, which is defined as intersection over union, IoU = | m i ∩ m j | | m i ∪ m j |
Revised: We calculated the overlap ratio of two pruning masks, which is defined as intersection over union, IoU = | m i ∩ m j | | m i ∪ m j |

Original: (Chen et al., 2020).
Revised: (Chen et al., 2020).

Original: In Figure 4, we show the overlap ratio between the pruning masks for the five random seeds, i.e., { m s 1 , ..., m s 5 } .
Revised: In Figure 4, we show the overlap ratio between the pruning masks for the five random seeds, i.e., { m s 1 , ..., m s 5 } .

Original: At first, we can see that ACTIVE - LT and RANDOM - LT using the regularizers resulted in diverse pruning.
Revised: At first, we can see that ACTIVE - LT and RANDOM - LT using the regularizers resulted in diverse pruning.

Original: This higher diversity could lead to the best improvements by ensembling, as discussed in Section 4.1.
Revised: This higher diversity could lead to the best improvements by ensembling, as discussed in Section 4.1.

Original: Secondly,
Revised: Secondly,

Original: BASE - LT produced surprisingly similar (99%) pruning masks with different random seeds.
Revised: BASE - LT produced surprisingly similar (99%) pruning masks with different random seeds.

Original: However, recall that even BASE - LT using the naïve iterative magnitude pruning performed better than BASELINE .
Revised: However, recall that even BASE - LT using the naïve iterative magnitude pruning performed better than BASELINE .

Original: This result shows that even seemingly small changes in structure can improve the diversity of predictions and the performance of the ensemble.
Revised: This result shows that even seemingly small changes in structure can improve the diversity of predictions and the performance of the ensemble.

Original:  Conclusion
Revised:  Conclusion

Original: We raised a question on difficulty of ensembling large-scale pretrained models.
Revised: We raised a question on difficulty of ensembling large-scale pretrained models.

Original: As an efficient remedy, we explored methods to use subnetworks in a single model.
Revised: As an efficient remedy, we explored methods to use subnetworks in a single model.

Original: We empirically demonstrated that ensembling winning-ticket subnetworks could outperform the dense ensembles via diversification and indicated a limitation too.
Revised: We empirically demonstrated that ensembling winning-ticket subnetworks could outperform the dense ensembles via diversification and indicated a limitation too.

Revised: Acknowledgments

Revised: We appreciate the helpful comments from the anonymous reviewers.

Revised: This work was supported by JSPS KAKENHI Grant Number JP19H

Original: A The Setting of Fine-tuning
Revised: A The Setting of Fine-tuning

Original: We follow the setting of Chen et al.
Revised: We follow the setting of Chen et al.

Original: (2020)’s implementation; epoch: 3, initial learning rate: 2ewith linear decay, maximum sequence length: 128, batch size: 32, dropout probability: 0.1.
Revised: (2020)’s implementation; epoch: 3, initial learning rate: 2ewith linear decay, maximum sequence length: 128, batch size: 32, dropout probability: 0.1.

Original: This is one of the most-used settings for finetuning a BERT; e.g., the example of finetuning in the Transformers library (Wolf et al., 2020) uses the setting 9 .
Revised: This is one of the most-used settings for finetuning a BERT; e.g., the example of finetuning in the Transformers library (Wolf et al., 2020) uses the setting 9 .

Original: We did not prune the embedding layer, following
Revised: We did not prune the embedding layer, following

Original: Chen et al.
Revised: Chen et al.

Original: (2020); Prasanna et al.
Revised: (2020); Prasanna et al.

Original: The coefficient of L 1 regularizer, τ , is decayed using the same scheduler as the learning rate.
Revised: The coefficient of L 1 regularizer, τ , is decayed using the same scheduler as the learning rate.

Original: We tuned it on MRPC and used it for other tasks.
Revised: We tuned it on MRPC and used it for other tasks.

Original: B The Learning Rate Scheduler of Chen et al. (2020)
Revised: B The Learning Rate Scheduler of Chen et al. (2020)

Original: Our implementation used in the experiments are derived from Chen et al.
Revised: Our implementation used in the experiments are derived from Chen et al.

Original: (2020)’s implementation 10 .
Revised: (2020)’s implementation 10 .

Original: However, we found a bug in Chen et al.
Revised: However, we found a bug in Chen et al.

Original: (2020)’s implementation on GitHub.
Revised: (2020)’s implementation on GitHub.

Original: Thus, we fixed it and experimented with the correct version.
Revised: Thus, we fixed it and experimented with the correct version.

Original: In their implementation, the learning rate schedule did not follow the common setting and the description mentioned in the paper; ‘We use standard implementations and hyperparameters [49].
Revised: In their implementation, the learning rate schedule did not follow the common setting and the description mentioned in the paper; ‘We use standard implementations and hyperparameters [49].

Original: Learning rate decays linearly from initial value to zero’ .
Revised: Learning rate decays linearly from initial value to zero’ .

Original: Specifically, the learning rate with linear decay did not reach zero but was at significant levels even at the end of the finetuning.
Revised: Specifically, the learning rate with linear decay did not reach zero but was at significant levels even at the end of the finetuning.

Original: Our implementation corrected it so that it did reach zero as specified in their paper and in the common setting.
Revised: Our implementation corrected it so that it did reach zero as specified in their paper and in the common setting.

Original: C The Combinations of Ensembles
Revised: C The Combinations of Ensembles

Original: In the experiments, we first prepared twenty random seeds and split them into two groups, each of which trained ten models.
Revised: In the experiments, we first prepared twenty random seeds and split them into two groups, each of which trained ten models.

Original: For stabilizing the measurement of the result, we exhaustively evaluated all the possible combinations of ensembles (i.e., depending on the number of members, 10 C 2 , 10 C 3 , 10 C 4 , 10 C 5 patterns, respectively) among the ten models for each group, and averaged the results with the two groups.
Revised: For stabilizing the measurement of the result, we exhaustively evaluated all the possible combinations of ensembles (i.e., depending on the number of members, 10 C 2 , 10 C 3 , 10 C 4 , 10 C 5 patterns, respectively) among the ten models for each group, and averaged the results with the two groups.

Original: The performance of the members is also averaged over all the seeds.
Revised: The performance of the members is also averaged over all the seeds.

Original: [Table]
Revised: [Table]

Original: D The Results with RoBERTa
Revised: D The Results with RoBERTa

Original: We simply conducted supplementary experiments with RoBERTa (Liu et al., 2019)
Revised: We simply conducted supplementary experiments with RoBERTa (Liu et al., 2019)

Original: (robeta-base model), although optimal hyperparameters were not searched well.
Revised: (robeta-base model), although optimal hyperparameters were not searched well.

Original: The results were similar to the cases of base-base-uncased.
Revised: The results were similar to the cases of base-base-uncased.

Original: The patterns can be categorized into the three.
Revised: The patterns can be categorized into the three.

Original: First, multi-ticket ensembles worked well with roberta on MRPC, as shown in Table 3.
Revised: First, multi-ticket ensembles worked well with roberta on MRPC, as shown in Table 3.

Original: Secondly, accurate winningticket subnetworks were not found on CoLA and
Revised: Secondly, accurate winningticket subnetworks were not found on CoLA and

Original: QNLI.
Revised: QNLI.

Original: Although the effect of ensembleing was improved after pruning, each single model got worse and the final ensemble accuracy did not outperform the dense baseline.
Revised: Although the effect of ensembleing was improved after pruning, each single model got worse and the final ensemble accuracy did not outperform the dense baseline.

Original: Thirdly, although accurate winning-ticket subnetworks were found on STS-B and SST-2, regularizations worsened single-model performances.
Revised: Thirdly, although accurate winning-ticket subnetworks were found on STS-B and SST-2, regularizations worsened single-model performances.

Original: While this case also improved the effect of ensembling, the final accuracy did not outperform the baseline.
Revised: While this case also improved the effect of ensembling, the final accuracy did not outperform the baseline.

Original: These experiments further emphasized the importance of development of more sophisticated pruning methods without sacrifice of model performances in the context of the lottery ticket hypothesis.
Revised: These experiments further emphasized the importance of development of more sophisticated pruning methods without sacrifice of model performances in the context of the lottery ticket hypothesis.

Original: E Related Work
Revised: E Related Work

Original: Some concurrent studies also investigate the usage of subnetworks for ensembles.
Revised: Some concurrent studies also investigate the usage of subnetworks for ensembles.

Original: Gal and Ghahramani (2016) is a pioneer to use subnetwork ensemble.
Revised: Gal and Ghahramani (2016) is a pioneer to use subnetwork ensemble.

Original: A trained neural network with dropout can infer with many different subnetworks, and their ensemble can be used for uncertainty estimation, which is called MC-dropout.
Revised: A trained neural network with dropout can infer with many different subnetworks, and their ensemble can be used for uncertainty estimation, which is called MC-dropout.

Original: Durasov et al.
Revised: Durasov et al.

Original: (2021) improved the efficiency of MC-dropout by exploring subnetworks.
Revised: (2021) improved the efficiency of MC-dropout by exploring subnetworks.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2021) (unpublished) experimented with an ensemble of subnetworks of different structures and initialization when trained from scratch, while the improvements possibly could be due to regularization of each single model.
Revised: (2021) (unpublished) experimented with an ensemble of subnetworks of different structures and initialization when trained from scratch, while the improvements possibly could be due to regularization of each single model.

Original: Havasi et al.
Revised: Havasi et al.

Original: (2021) is a similar but more elegant approach, which does not explicitly identify subnetworks.
Revised: (2021) is a similar but more elegant approach, which does not explicitly identify subnetworks.

Original: Instead, it trains a single dense model with training using multi-input multi-output inference; the optimization can implicitly find multiple disentangled subnetworks in the dense model during optimization from random initialization.
Revised: Instead, it trains a single dense model with training using multi-input multi-output inference; the optimization can implicitly find multiple disentangled subnetworks in the dense model during optimization from random initialization.

Original: These studies support our assumption that different subnetworks can improve ensemble by diversity.
Revised: These studies support our assumption that different subnetworks can improve ensemble by diversity.

Original: Some other directions for introducing diversity exist, while most are unstable.
Revised: Some other directions for introducing diversity exist, while most are unstable.

Original: Promising directions are to use entropy (Pang et al., 2019) or adversarial training (Rame and Cord, 2021).
Revised: Promising directions are to use entropy (Pang et al., 2019) or adversarial training (Rame and Cord, 2021).

Original: Although they required complex optimization processes, they improved the robustness or ensemble performance on small image recognition datasets.
Revised: Although they required complex optimization processes, they improved the robustness or ensemble performance on small image recognition datasets.

Original: Recently, concurrent work (Sellam et al., 2022;
Revised: Recently, concurrent work (Sellam et al., 2022;

Original: Tay et al., 2022) provide multiple BERT or T5 models pretrained from different seeds  for investigation of seed  dependency using large-scale computational resources.
Revised: Tay et al., 2022) provide multiple BERT or T5 models pretrained from different seeds or configurations for investigation of seed or configuration dependency using large-scale computational resources.

Original: Further research with the models and  computational resources will be helpful for more solid comparison and analysis.
Revised: Further research with the models and such computational resources will be helpful for more solid comparison and analysis.

Original: Note that no prior work tackled the problem of ensembles from a pre-trained model.
Revised: Note that no prior work tackled the problem of ensembles from a pre-trained model.

Original: Framing the problem is one of the contributions of this paper.
Revised: Framing the problem is one of the contributions of this paper.

Original: Secondly, our multi-ticket ensemble based on random masking enables an independently parallelizable training while existing methods require a sequential processing or a grouped training procedure.
Revised: Secondly, our multi-ticket ensemble based on random masking enables an independently parallelizable training while existing methods require a sequential processing or a grouped training procedure.

Original: Finally, multi-ticket ensemble can be combined with other methods, which can improve the total performance together.
Revised: Finally, multi-ticket ensemble can be combined with other methods, which can improve the total performance together.