Original: V OTING FROM N EAREST T ASKS : M ETA -V OTE P RUNING OF P RE - TRAINED M ODELS FOR D OWNSTREAM T ASKS
Revised: V OTING FROM N EAREST T ASKS : M ETA -V OTE P RUNING OF P RE - TRAINED M ODELS FOR D OWNSTREAM T ASKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task?
Revised: As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task?

Original: How to reuse the pruning results of previous tasks to accelerate the pruning for a new task?
Revised: How to reuse the pruning results of previous tasks to accelerate the pruning for a new task?

Original: To address these challenges, we create a small model for a new task from the pruned models of similar tasks.
Revised: To address these challenges, we create a small model for a new task from the pruned models of similar tasks.

Original: We show that a few ﬁne-tuning steps on this model sufﬁce to produce a promising pruned-model for the new task.
Revised: We show that a few ﬁne-tuning steps on this model sufﬁce to produce a promising pruned-model for the new task.

Original: We study this “meta-pruning” from nearest tasks on two major classes of pre-trained models, convolutional neural network (CNN) and vision transformer (ViT), under a limited budget of pruning iterations.
Revised: We study this “meta-pruning” from nearest tasks on two major classes of pre-trained models, convolutional neural network (CNN) and vision transformer (ViT), under a limited budget of pruning iterations.

Original: Our study begins by investigating the overlap of pruned models for similar tasks and how the overlap changes over different layers and blocks.
Revised: Our study begins by investigating the overlap of pruned models for similar tasks and how the overlap changes over different layers and blocks.

Original: Inspired by these discoveries, we develop a simple but effective “Meta-Vote Pruning (MVP)” method that signiﬁcantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models of its nearest tasks.
Revised: Inspired by these discoveries, we develop a simple but effective “Meta-Vote Pruning (MVP)” method that signiﬁcantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models of its nearest tasks.

Original: In experiments, we demonstrate MVP’s advantages in accuracy, efﬁciency, and generalization through extensive empirical studies and comparisons with popular pruning methods over several datasets.
Revised: In experiments, we demonstrate MVP’s advantages in accuracy, efﬁciency, and generalization through extensive empirical studies and comparisons with popular pruning methods over several datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large-scale pre-trained models usually contain tens of millions or even billions of parameters for promising generalization performance.
Revised: Large-scale pre-trained models usually contain tens of millions or even billions of parameters for promising generalization performance.

Original: The computation and memory of modern GPUs or clusters can support to train such models, but directly deploying them to edge devices can easily violate the hardware limits on memory and computation.
Revised: The computation and memory of modern GPUs or clusters can support to train such models, but directly deploying them to edge devices can easily violate the hardware limits on memory and computation.

Original: Network pruning (Han et al., 2016; Tian et al., 2020; Li et al., 2020; Chin et al., 2020) has been widely studied to compress neural nets by removing redundant connections and nodes.
Revised: Network pruning (Han et al., 2016; Tian et al., 2020; Li et al., 2020; Chin et al., 2020) has been widely studied to compress neural nets by removing redundant connections and nodes.

Original: Numerous empirical results have veriﬁed that pruning can compress the original network into much smaller sub-networks that still enjoy the comparable performance.
Revised: Numerous empirical results have veriﬁed that pruning can compress the original network into much smaller sub-networks that still enjoy the comparable performance.

Original: Instead of reducing the network to the target size by one-time pruning, iterative pruning that alternates between pruning and ﬁne-tuning for iterations usually achieves better performance (Han et al., 2015; Li et al., 2017).
Revised: Instead of reducing the network to the target size by one-time pruning, iterative pruning that alternates between pruning and ﬁne-tuning for iterations usually achieves better performance (Han et al., 2015; Li et al., 2017).

Original: Theoretically, a line of recent works (Frankle & Carbin, 2018;
Revised: Theoretically, a line of recent works (Frankle & Carbin, 2018;

Original: Ye et al., 2020; Savarese et al., 2020; Malach et al., 2020) attempts to prove the lottery ticket hypothesis, i.e., the existence of such sub-networks, for different pruning settings.
Revised: Ye et al., 2020; Savarese et al., 2020; Malach et al., 2020) attempts to prove the lottery ticket hypothesis, i.e., the existence of such sub-networks, for different pruning settings.

Original: In a variety of practical applications, a pre-trained network usually needs to be pruned for a wide variety of devices and adapted to different downstream tasks.
Revised: In a variety of practical applications, a pre-trained network usually needs to be pruned for a wide variety of devices and adapted to different downstream tasks.

Original: Running an iterative pruning algorithm for every device or task from the same pre-trained network can create enormous carbon footprint overload in our biosphere and waste a lot of computational power.
Revised: Running an iterative pruning algorithm for every device or task from the same pre-trained network can create enormous carbon footprint overload in our biosphere and waste a lot of computational power.

Original: On the other hand, the wide applications of a few pre-trained models have already created thousands of pruned models for different downstream tasks.
Revised: On the other hand, the wide applications of a few pre-trained models have already created thousands of pruned models for different downstream tasks.

Original: Can we reuse these pruned models as prior knowledge to save the pruning computation on new tasks?
Revised: Can we reuse these pruned models as prior knowledge to save the pruning computation on new tasks?

Original: We call this problem “meta-pruning”.
Revised: We call this problem “meta-pruning”.

Original: In this paper, we mainly focus on a special case of it, which initializes a sub-network for a given new task based on the pruned models of similar tasks.
Revised: In this paper, we mainly focus on a special case of it, which initializes a sub-network for a given new task based on the pruned models of similar tasks.

Original: Meta-pruning is non-parametric if no parametric model is trained to produce the initialization.
Revised: Meta-pruning is non-parametric if no parametric model is trained to produce the initialization.

Original: It is analogous to MAML (Finn et al., 2017) in that the meta-objective optimizes the initialization of a network.
Revised: It is analogous to MAML (Finn et al., 2017) in that the meta-objective optimizes the initialization of a network.

Original: It differs from MAML in that (1) both the sub-network’s architecture and weights are initialized; and (2) the initialization is not universal but task-speciﬁc.
Revised: It differs from MAML in that (1) both the sub-network’s architecture and weights are initialized; and (2) the initialization is not universal but task-speciﬁc.

Original: Since meta-pruning aims to ﬁnd better sub-network initialization for new tasks, we limit the iterations during meta-pruning to strengthen the impact of initialization on the ﬁnal pruned model.
Revised: Since meta-pruning aims to ﬁnd better sub-network initialization for new tasks, we limit the iterations during meta-pruning to strengthen the impact of initialization on the ﬁnal pruned model.

Original: This also controls the computational cost and carbon footprint of meta-pruning much less than conventional pruning that requires many iterations.
Revised: This also controls the computational cost and carbon footprint of meta-pruning much less than conventional pruning that requires many iterations.

Original: Under this constraint, a well-performed pretrained model is critical to the meta-pruning performance because (1) it needs to provide initialized sub-networks for different tasks; and (2) a few iterations of ﬁne-tuning to the sub-networks should sufﬁce to produce high-quality pruned models for targeted tasks.
Revised: Under this constraint, a well-performed pretrained model is critical to the meta-pruning performance because (1) it needs to provide initialized sub-networks for different tasks; and (2) a few iterations of ﬁne-tuning to the sub-networks should sufﬁce to produce high-quality pruned models for targeted tasks.

Original: Meta-pruning follows a practical setting where one single pre-trained model is tailored for different tasks using limited iterations.
Revised: Meta-pruning follows a practical setting where one single pre-trained model is tailored for different tasks using limited iterations.

Original: We study two classes of the most widely used pre-trained models, i.e., convolutional neural networks (CNN)
Revised: We study two classes of the most widely used pre-trained models, i.e., convolutional neural networks (CNN)

Original: (He et al., 2016) and vision transformer (ViT) (Dosovitskiy et al., 2021).
Revised: (He et al., 2016) and vision transformer (ViT) (Dosovitskiy et al., 2021).

Original: The primary contribution of this paper is two folds.
Revised: The primary contribution of this paper is two folds.

Original: In the ﬁrst part, we conduct a thorough empirical study that applies different pruning methods to CNN and ViT and compare their produced sub-networks for hundreds of downstream tasks.
Revised: In the ﬁrst part, we conduct a thorough empirical study that applies different pruning methods to CNN and ViT and compare their produced sub-networks for hundreds of downstream tasks.

Original: No meta-pruning is studied in this part and its primary purpose is to (1) ﬁnd the nearest tasks for a new task using different similarity metrics; and (2) compare the pruned models for different but similar tasks.
Revised: No meta-pruning is studied in this part and its primary purpose is to (1) ﬁnd the nearest tasks for a new task using different similarity metrics; and (2) compare the pruned models for different but similar tasks.

Original: To this end, we build a dataset of tasks and their sub-networks pruned from the same pre-trained models.
Revised: To this end, we build a dataset of tasks and their sub-networks pruned from the same pre-trained models.

Original: Statistics and evaluations on this dataset indicate similar tasks with high similarity tend to share more nodes/ﬁlters/heads preserved in their pruned models, especially in deeper layers that notably capture high-level task-speciﬁc features.
Revised: Statistics and evaluations on this dataset indicate similar tasks with high similarity tend to share more nodes/ﬁlters/heads preserved in their pruned models, especially in deeper layers that notably capture high-level task-speciﬁc features.

Original: Motivated by the empirical study, the second part of this paper proposes a simple yet strong meta-pruning method called “meta-vote pruning (MVP)”.
Revised: Motivated by the empirical study, the second part of this paper proposes a simple yet strong meta-pruning method called “meta-vote pruning (MVP)”.

Original: It can signiﬁcantly reduce the pruning cost and memory required by previous pruning approaches yet still produce pruned models with promising performance.
Revised: It can signiﬁcantly reduce the pruning cost and memory required by previous pruning approaches yet still produce pruned models with promising performance.

Original: Given a pre-trained model, MVP ﬁnds a sub-network for a new task by selecting nodes/ﬁlters/heads through majority voting among its nearest tasks, e.g., a ﬁlter will be sampled with a higher chance if it is selected into more sub-networks of similar tasks.
Revised: Given a pre-trained model, MVP ﬁnds a sub-network for a new task by selecting nodes/ﬁlters/heads through majority voting among its nearest tasks, e.g., a ﬁlter will be sampled with a higher chance if it is selected into more sub-networks of similar tasks.

Original: To keep the method simple, we sample the same proportion of nodes/ﬁlters/heads as the targeted pruning ratio.
Revised: To keep the method simple, we sample the same proportion of nodes/ﬁlters/heads as the targeted pruning ratio.

Original: Then we apply a few iterations of ﬁne-tuning to the initialized sub-network using training data of the new task.
Revised: Then we apply a few iterations of ﬁne-tuning to the initialized sub-network using training data of the new task.

Original: Although a more sophisticated procedure can be developed, the proposed method already saves a substantial amount of computation and memory while maintaining a high test accuracy of pruned models.
Revised: Although a more sophisticated procedure can be developed, the proposed method already saves a substantial amount of computation and memory while maintaining a high test accuracy of pruned models.

Original: We demonstrate these via experiments over tasks from CIFAR-100 (Krizhevsky & Hinton, 2009), ImageNet (Deng et al., 2009) and Caltech-256 (Grifﬁn et al., 2007).
Revised: We demonstrate these via experiments over tasks from CIFAR-100 (Krizhevsky & Hinton, 2009), ImageNet (Deng et al., 2009) and Caltech-256 (Grifﬁn et al., 2007).

Original: The pruned models extracted from an ImageNet pre-trained model can also vote for tasks drawn from the unseen dataset Caltech-256 with great performance, which shows the generalization of MVP.
Revised: The pruned models extracted from an ImageNet pre-trained model can also vote for tasks drawn from the unseen dataset Caltech-256 with great performance, which shows the generalization of MVP.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Network pruning Network pruning has been widely studied to compress network and accelerate its inference for a single task.
Revised: Network pruning Network pruning has been widely studied to compress network and accelerate its inference for a single task.

Original: We mainly summarize structure pruning below.
Revised: We mainly summarize structure pruning below.

Original: In CNN, to encourage the sparsity of the pruned network, L 0 (Louizos et al., 2018), L 1 (Liu et al., 2017) or L 2 (Han et al., 2015) regularization have been used, and recent polarization regularization (Zhuang et al., 2020) shrinks some nodes towards 0 and meanwhile strengthen the others to keep important nodes intact.
Revised: In CNN, to encourage the sparsity of the pruned network, L 0 (Louizos et al., 2018), L 1 (Liu et al., 2017) or L 2 (Han et al., 2015) regularization have been used, and recent polarization regularization (Zhuang et al., 2020) shrinks some nodes towards 0 and meanwhile strengthen the others to keep important nodes intact.

Original: Different criteria have been proposed to evaluate the importance of nodes/ﬁlters.
Revised: Different criteria have been proposed to evaluate the importance of nodes/ﬁlters.

Original: Li et al.
Revised: Li et al.

Original: (2017) prunes ﬁlters with the smallest sum of parameters’ absolute values.
Revised: (2017) prunes ﬁlters with the smallest sum of parameters’ absolute values.

Original: Lin et al.
Revised: Lin et al.

Original: (2018) prune ﬁlters according to the second-order Taylor expansion of the loss.
Revised: (2018) prune ﬁlters according to the second-order Taylor expansion of the loss.

Original: Methods (Bai et al., 2022; Frankle &
Revised: Methods (Bai et al., 2022; Frankle &

Original: Carbin, 2018) based on lottery ticket hypothesis try to ﬁnd a well-performed sparse initialization for each task.
Revised: Carbin, 2018) based on lottery ticket hypothesis try to ﬁnd a well-performed sparse initialization for each task.

Original: ViT has been widely used in computer vision and achieved SOTA performance in many tasks.
Revised: ViT has been widely used in computer vision and achieved SOTA performance in many tasks.

Original: The input patches for each block can be pruned to save computation for ViT. Goyal et al.
Revised: The input patches for each block can be pruned to save computation for ViT. Goyal et al.

Original: (2020) propose a metric for the importance of each patch and dynamically prune patches in each layer.
Revised: (2020) propose a metric for the importance of each patch and dynamically prune patches in each layer.

Original: PatchSlimming (Tang et al., 2022) retains patches critical to preserve the original ﬁnal output.
Revised: PatchSlimming (Tang et al., 2022) retains patches critical to preserve the original ﬁnal output.

Original: HVT (Pan et al., 2021) is a CNN-like method which shortens the patch sequence by max-pooling.
Revised: HVT (Pan et al., 2021) is a CNN-like method which shortens the patch sequence by max-pooling.

Original: Another line of work (Zhu et al., 2021; Yu et al., 2022b;a) automatically prunes the unimportant heads, nodes and blocks in ViT. These methods excel on single-task pruning but their cost linearly increases for multiple tasks (and thus more expensive than meta-pruning) because: (1) a large model needs to be trained for every task; (2) every task requires to prune its own large pre-trained model from scratch. 
Revised: Another line of work (Zhu et al., 2021; Yu et al., 2022b;a) automatically prunes the unimportant heads, nodes and blocks in ViT. These methods excel on single-task pruning but their cost linearly increases for multiple tasks (and thus more expensive than meta-pruning) because: (1) a large model needs to be trained for every task; (2) every task requires to prune its own large pre-trained model from scratch. For both CNN and ViT, it is time-consuming for these pruning methods to build a pruned model for each unseen target task from a large pre-trained model.

Revised: Our proposed method can borrow the knowledge of the existing pruned models extracted by these pruning methods and use them to generate a well-performed pruned model for the unseen task with a few ﬁne-tuning iterations.

Original: Meta-pruning To the best of our knowledge, the non-parametric meta-pruning problem, i.e., how to prune a model for a target task using the pruned models of other tasks , has not been speciﬁcally studied in previous work.
Revised: Meta-pruning To the best of our knowledge, the non-parametric meta-pruning problem, i.e., how to prune a model for a target task using the pruned models of other tasks , has not been speciﬁcally studied in previous work.

Original: However, several recent researches aim at learning meta(prior) knowledge that can improve pruning in other scenarios.
Revised: However, several recent researches aim at learning meta(prior) knowledge that can improve pruning in other scenarios.

Original: MetaPruning (Liu et al., 2019) trains a weight-generation meta-network to prune the same network for the same task under different constraints, e.g., user/hardware deﬁned pruning ratios.
Revised: MetaPruning (Liu et al., 2019) trains a weight-generation meta-network to prune the same network for the same task under different constraints, e.g., user/hardware deﬁned pruning ratios.

Original: DHP (Li et al., 2020) addresses the same problem but does not rely on any reinforcement learning or evolutionary algorithm since it makes the pruning procedure differentiable.
Revised: DHP (Li et al., 2020) addresses the same problem but does not rely on any reinforcement learning or evolutionary algorithm since it makes the pruning procedure differentiable.

Original: Meta-learning has been studied to ﬁnd better weight-initialization for pruning on different tasks, e.g., Tian et al.
Revised: Meta-learning has been studied to ﬁnd better weight-initialization for pruning on different tasks, e.g., Tian et al.

Original: (2020) applies Reptile (Nichol et al., 2018) for overﬁtting reduction.
Revised: (2020) applies Reptile (Nichol et al., 2018) for overﬁtting reduction.

Original: Meta-learning has also been studied to select the best pruning criterion for different tasks (He et al., 2019).
Revised: Meta-learning has also been studied to select the best pruning criterion for different tasks (He et al., 2019).

Original: In (Sun et al., 2020), a shared sparse backbone network is trained for multi-task learning but it cannot be adapted to new tasks.
Revised: In (Sun et al., 2020), a shared sparse backbone network is trained for multi-task learning but it cannot be adapted to new tasks.

Revised: Our method is the ﬁrst one to use meta-learning to extract a pruned model for a new task.

Original: The main differences of our approach to them are: (1) we do not train a parametric meta-learner but instead use majority voting from similar tasks; and (2) our meta-voting generates a pruned small sub-network to initialize the target task training, which signiﬁcantly reduces the ﬁne-tuning cost.
Revised: The main differences of our approach to them are: (1) we do not train a parametric meta-learner but instead use majority voting from similar tasks; and (2) our meta-voting generates a pruned small sub-network to initialize the target task training, which signiﬁcantly reduces the pruning cost.

Original:  E MPIRICAL S TUDY : P RUNING A P RE - TRAINED MODEL FOR D IFFERENT T ASKS
Revised:  E MPIRICAL S TUDY : P RUNING A P RE - TRAINED MODEL FOR D IFFERENT T ASKS

Original: In this section, we conduct an empirical study that applies different methods to prune a CNN or ViT pre-trained model for over hundreds of tasks.
Revised: In this section, we conduct an empirical study that applies different methods to prune a CNN or ViT pre-trained model for over hundreds of tasks.

Original: Our study focuses on the overlap between the pruned models for different tasks and whether/how it relates to their similarity.
Revised: Our study focuses on the overlap between the pruned models for different tasks and whether/how it relates to their similarity.

Original: To this end, we introduce different task similarities and compare the overlap associated with different similarity groups.
Revised: To this end, we introduce different task similarities and compare the overlap associated with different similarity groups.

Original: The results show that more similar tasks tend to share more nodes/ﬁlters/heads in their pruned models.
Revised: The results show that more similar tasks tend to share more nodes/ﬁlters/heads in their pruned models.

Original: And this holds across different pruning methods, datasets and pre-trained models.
Revised: And this holds across different pruning methods, datasets and pre-trained models.

Original: No meta-pruning is used in the study.
Revised: No meta-pruning is used in the study.

Original: 3. A D ATASET OF P RUNED M ODELS
Revised: 3. A D ATASET OF P RUNED M ODELS

Original: While the number of possible downstream tasks and users can be huge in practice, the current progress on foundation models show that one or a few large-scale pre-trained models with light ﬁne-tuning usually achieve the SOTA performance on most of them.
Revised: While the number of possible downstream tasks and users can be huge in practice, the current progress on foundation models show that one or a few large-scale pre-trained models with light ﬁne-tuning usually achieve the SOTA performance on most of them.

Original: To simulate this scenario on a standard dataset, our empirical study creates a dataset of pruned models for hundreds of tasks from the same pre-trained model.
Revised: To simulate this scenario on a standard dataset, our empirical study creates a dataset of pruned models for hundreds of tasks from the same pre-trained model.

Original: We choose CIFAR-100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) for the study due to many classes in them.
Revised: We choose CIFAR-100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) for the study due to many classes in them.

Original: Each class in
Revised: Each class in

Original: CIFAR-100 and ImageNet has 500 and 1300 samples, respectively.
Revised: CIFAR-100 and ImageNet has 500 and 1300 samples, respectively.

Original: For each dataset, we randomly draw 1000 classiﬁcation tasks, each deﬁned on 5 classes sampled without replacement. 
Revised: For each dataset, we randomly draw 1000 classiﬁcation tasks, each deﬁned on 5 classes sampled without replacement. We adopt

Original: We adopt ResNet-18 (He et al., 2016) pre-trained on CIFAR-100, ResNet-50 (He et al., 2016) and  ViT (Dosovitskiy et al., 2021) pre-trained on ImageNet.
Revised: ResNet-18 (He et al., 2016) pre-trained on CIFAR-100, ResNet-50 (He et al., 2016) and a small version of ViT (Touvron et al., 2021b) pre-trained on ImageNet.

Original: For ResNet-18 and ResNet-50, we prune two types of pre-trained models, i.e., the supervised training following Devries & Taylor (2017) and the self-supervised training following SimSiam Chen & He (2020) (only the encoder is used).
Revised: For ResNet-18 and ResNet-50, we prune two types of pre-trained models, i.e., the supervised training following Devries & Taylor (2017) and the self-supervised training following SimSiam Chen & He (2020) (only the encoder is used).

Original: For ViT, the training of its pre-trained model follows (Dosovitskiy et al., 2021).
Revised: For ViT, the training of its pre-trained model follows (Dosovitskiy et al., 2021).

Original: Iterative Pruning We apply iterative ﬁlter-pruning (IFP) to ResNet.
Revised: Iterative Pruning We apply iterative ﬁlter-pruning (IFP) to ResNet.

Original: Unlike magnitude-based pruning (Li et al., 2017) with one-time selection of nodes/weights, iterative pruning alternates between network pruning and ﬁne-tuning of model weights for multiple iterations, each of which prunes p % of the remaining nodes/weights so it progressively prunes a large network to the targeted size.
Revised: Unlike magnitude-based pruning (Li et al., 2017) with one-time selection of nodes/weights, iterative pruning alternates between network pruning and ﬁne-tuning of model weights for multiple iterations, each of which prunes p % of the remaining nodes/weights so it progressively prunes a large network to the targeted size.

Original: It usually performs better than other pruning methods and has also been mainly studied in theoretical works about Lottery Ticket Hypothesis (Frankle & Carbin, 2018).
Revised: It usually performs better than other pruning methods and has also been mainly studied in theoretical works about Lottery Ticket Hypothesis (Frankle & Carbin, 2018).

Original: We take the activation values of ﬁlters averaged over all training samples to measure the importance of ﬁlters (Molchanov et al., 2016), referred as Activation Pruning, in which ﬁlters with smaller activation values contain less information of input data.
Revised: We take the activation values of ﬁlters averaged over all training samples to measure the importance of ﬁlters (Molchanov et al., 2016), referred as Activation Pruning, in which ﬁlters with smaller activation values contain less information of input data.

Original: The detailed procedure of IFP is described in Alg. 2 in Appendix.
Revised: The detailed procedure of IFP is described in Alg. 2 in Appendix.

Original: Automatic Pruning Inspired by the SOTA ViT structured pruning method (Yu et al., 2022b), we prune ViT by automatic head&node pruning (AHNP) for a given task, which parameterizes the subnetwork as the pre-trained model with a learnable score multiplied to each prunable head and node.
Revised: Automatic Pruning Inspired by the SOTA ViT structured pruning method (Yu et al., 2022b), we prune ViT by automatic head&node pruning (AHNP) for a given task, which parameterizes the sub- network as the pre-trained model with a learnable score multiplied to each prunable head and node.

Original: To encourage sparsity, the differentiable scores of all prunable heads and nodes are optimized with an additional L 1 regularization loss.
Revised: To encourage sparsity, the differentiable scores of all prunable heads and nodes are optimized with an additional L 1 regularization loss.

Original: After each optimization step, we apply a simple thresholding to these scores to remove heads and nodes with small scores.
Revised: After each optimization step, we apply a simple thresholding to these scores to remove heads and nodes with small scores.

Original: The optimization stops if the pruned model reaches the targeted size and the model will be ﬁne-tuned for a few iterations.
Revised: The optimization stops if the pruned model reaches the targeted size and the model will be ﬁne-tuned for a few iterations.

Original: The detailed procedure of AHNP can be found in Alg. 3 in Appendix.
Revised: The detailed procedure of AHNP can be found in Alg. 3 in Appendix.

Original: For tasks of CIFAR-100, we run IFP for all 1000 tasks on ResNet-18.
Revised: For tasks of CIFAR-100, we run IFP for all 1000 tasks on ResNet-18.

Original: And we apply IFP and AHNP to tasks of ImageNet on ResNet-50 and
Revised: And we apply IFP and AHNP to tasks of ImageNet on ResNet-50 and

Original: ViT respectively.
Revised: ViT respectively.

Original: Finally, we create a dataset of pruned models for thousands of tasks over different pre-trained models.
Revised: Finally, we create a dataset of pruned models for thousands of tasks over different pre-trained models.

Original: For each task i , we record its labels C i , the set of preserved nodes/ﬁlters/heads { Ω (cid:96) } (cid:96) =1: L − 1 and the pruned model θ T .
Revised: For each task i , we record its labels C i , the set of preserved nodes/ﬁlters/heads { Ω (cid:96) } (cid:96) =1: L − 1 and the pruned model θ T .

Original: We use the same hyper-parameters for different tasks.
Revised: We use the same hyper-parameters for different tasks.

Original: For
Revised: For

Original: IFP on ResNet, we use a learning rate of 0 .
Revised: IFP on ResNet, we use a learning rate of 0 .

Original: 005 , pruning iterations of 1000 and batch-size offor both the tasks of CIFAR-100 and ImageNet.
Revised: 005 , pruning iterations of 1000 and batch-size offor both the tasks of CIFAR-100 and ImageNet.

Original: When applying AHNP to ViT, we follow the ViT training in (Touvron et al., 2021).
Revised: When applying AHNP to ViT, we follow the ViT training in (Touvron et al., 2021a).

Original: We reduce the pruning iterations to 1000 and use a small learning rate of 0 . 00005 for parameters inherited from the pre-trained ViT (to preserve its knowledge) and a large learning rate of 0 .
Revised: We reduce the pruning iterations to 1000 and use a small learning rate of 0 . 00005 for parameters inherited from the pre-trained ViT (to preserve its knowledge) and a large learning rate of 0 .

Original: for the learnable scores.
Revised: for the learnable scores.

Original: The pruning ratio is 0 .
Revised: The pruning ratio is 0 .

Original: 9 for all pruned models.
Revised: 9 for all pruned models.

Original: 3. D O SIMILAR TASKS SHARE MORE NODES / FILTERS / HEADS ON EACH LAYER OF THEIR PRUNED MODELS ?
Revised: 3. D O SIMILAR TASKS SHARE MORE NODES / FILTERS / HEADS ON EACH LAYER OF THEIR PRUNED MODELS ?

Original: The representations learned for a task can still be helpful to its similar tasks.
Revised: The representations learned for a task can still be helpful to its similar tasks.

Original: This motivates transfer/multi-task/meta learning methods.
Revised: This motivates transfer/multi-task/meta learning methods.

Original: But do similar tasks also share more structures in their pruned sub-networks?
Revised: But do similar tasks also share more structures in their pruned sub-networks?

Original: We apply two metrics to measure the similarity between classiﬁcation tasks in our dataset and study whether/how the similarity relate to their shared nodes/ﬁlters/heads in different layers of their pruned models.
Revised: We apply two metrics to measure the similarity between classiﬁcation tasks in our dataset and study whether/how the similarity relate to their shared nodes/ﬁlters/heads in different layers of their pruned models.

Original: Similarity Metrics We apply two metrics to compute the similarity between tasks and ﬁnd the nearest tasks, i.e., the Log Expected Empirical Prediction (LEEP)
Revised: Similarity Metrics We apply two metrics to compute the similarity between tasks and ﬁnd the nearest tasks, i.e., the Log Expected Empirical Prediction (LEEP)

Original: (Nguyen et al., 2020) and the
Revised: (Nguyen et al., 2020) and the

Original: Wordnet wup similarity (Pedersen et al., 2004; Wu & Palmer, 1994).
Revised: Wordnet wup similarity (Pedersen et al., 2004; Wu & Palmer, 1994).

Original: LEEP score is widely used in transfer learning to estimate the knowledge transferability from a source task to a target task.
Revised: LEEP score is widely used in transfer learning to estimate the knowledge transferability from a source task to a target task.

Original: In our study, for each target task, we can rank the other tasks by their LEEP similarity score from each of them to the target one.
Revised: In our study, for each target task, we can rank the other tasks by their LEEP similarity score from each of them to the target one.

Original: Computing the LEEP score only requires a single forward pass of the pruned model on the target task’s data.
Revised: Computing the LEEP score only requires a single forward pass of the pruned model on the target task’s data.

Original: Wordnet wup similarity only requires the semantic labels of classes in each task and it is based on the depths of the their corresponding synsets in the Wordnet (Miller, 1995) taxonomies.
Revised: Wordnet wup similarity only requires the semantic labels of classes in each task and it is based on the depths of the their corresponding synsets in the Wordnet (Miller, 1995) taxonomies.

Original: It does not depend on the pruned model so it is more efﬁcient to compute.
Revised: It does not depend on the pruned model so it is more efﬁcient to compute.

Original: Overlap Between Tasks Let Ω i(cid:96) and Ω j(cid:96) denote the sets of ﬁlters/nodes/heads remained in layer- (cid:96) after running IFP or AHNP for task i and j (when using the same pre-trained model), we measure the overlap of the two sets by intersection over union (IoU) ratio (Jaccard, 1901), i.e.,
Revised: Overlap Between Tasks Let Ω i(cid:96) and Ω j(cid:96) denote the sets of ﬁlters/nodes/heads remained in layer- (cid:96) after running IFP or AHNP for task i and j (when using the same pre-trained model), we measure the overlap of the two sets by intersection over union (IoU) ratio (Jaccard, 1901), i.e.,

Original: [Equation]
Revised: IoU = | Ω i(cid:96)

Original: [Figure]
Revised: ∩ Ω j(cid:96) | / | Ω i(cid:96) ∪ Ω j(cid:96) | .

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1 (ResNet) and Fig.
Revised: 1 (ResNet) and Fig.

Original: 2 (ViT) report the IoU of each layer/block for pairs of tasks with different similarities.
Revised: 2 (ViT) report the IoU of each layer/block for pairs of tasks with different similarities.

Original: For each target task, the tasks in the dataset are partitioned into 5 similarity groups according to their LEEP scores or Wordnet similarities to the target task.
Revised: For each target task, the tasks in the dataset are partitioned into 5 similarity groups according to their LEEP scores or Wordnet similarities to the target task.

Original: The similarity decreases from group 1 to group 5.
Revised: The similarity decreases from group 1 to group 5.

Original: [Figure] [Figure] For all the datasets and architectures, more similar tasks tend to share more ﬁlters/nodes/heads (larger IoU) between their pruned models.
Revised: For all the datasets and architectures, more similar tasks tend to share more ﬁlters/nodes/heads (larger IoU) between their pruned models.

Original: Therefore, for a new task, the pruned models of its nearest tasks preserve many important ﬁlters for it and combining them might result in a better and much smaller sub-network to initialize the new task.
Revised: Therefore, for a new task, the pruned models of its nearest tasks preserve many important ﬁlters for it and combining them might result in a better and much smaller sub-network to initialize the new task.

Original: Moreover, for deeper layers/blocks in both ResNet and ViT, the gap between different similarity groups on the IoU increases because the features are more task-speciﬁc in deeper layers.
Revised: Moreover, for deeper layers/blocks in both ResNet and ViT, the gap between different similarity groups on the IoU increases because the features are more task-speciﬁc in deeper layers.

Original: Due to the same reason, for every similarity group,
Revised: Due to the same reason, for every similarity group,

Original: IoU decreases with depth in the overall trend (though ﬂuctuating locally).
Revised: IoU decreases with depth in the overall trend (though ﬂuctuating locally).

Original: Furthermore, Fig.shows that the IoU gap between similarity groups deﬁned by the LEEP score is larger than that obtained by Wordnet similarity.
Revised: Furthermore, Fig. 2 shows that the IoU gap between similarity groups deﬁned by the LEEP score is larger than that obtained by [Figure] I

Revised: [Figure]

Revised: U o f

Original: This indicates that the semantic similarity between class labels might not be as accurate as the LEEP score that takes the pruned model and its learned representations into account.
Revised: [Figure] a d [Figure] [Figure]

Revised: f

Revised: [Figure]

Revised: Wordnet similarity.

Revised: This indicates that the semantic similarity between class labels might not be as accurate as the LEEP score that takes the pruned model and its learned representations into account.

Original: M ETA -V OTE P RUNING (MVP)
Revised: Input : i set D pruning ratio J different tasks Output : A pruned model for target task- i

Revised: Initialize: Ω (cid:96) ← ∅ , the set of ﬁlters in layer- (cid:96) 1 Sample/ﬁnd N similar tasks N i to task i according to LEEP score or Wordnet similarity; 2 for (cid:96) ← 1 to L − 1 do

Revised: Sample (1 − r ) n (cid:96) ﬁlters with probability p ( k ) (Eq.

Revised: and add them to Ω (cid:96) ; for k ∈ Ω (cid:96) do

Revised: Initialize ﬁlter-

Revised: k by averaging its parameters of tasks in { j ∈ N i : k ∈ Ω j(cid:96) } ; end 7 end 8 Fine-tune the pruned for J iterations on D i .

Original: Inspired by the empirical study above, we propose a simple yet strong baseline “meta-vote pruning (MVP)”
Revised: Inspired by the empirical study above, we propose a simple yet strong baseline “meta-vote pruning (MVP)”

Original: (Alg. 1) for non-parametric meta-pruning.
Revised: (Alg. 1) for non-parametric meta-pruning.

Original: The procedure of MVP majority voting is shown in Fig.
Revised: The procedure of MVP majority voting is shown in Fig.

Original: Given a target task i , MVP draws a sub-network of a pre-trained network by sampling ﬁlters/nodes/heads in each layer using majority voting from its nearest tasks N i and their pruned models.
Revised: Given a target task i , MVP draws a sub-network of a pre-trained network by sampling ﬁlters/nodes/heads in each layer using majority voting from its nearest tasks N i and their pruned models.

Original: In particular, for each ﬁlter- k ∈ [ n (cid:96) ] from layer- (cid:96) of the pre-trained model, we apply softmax (with temperature τ ) to the times of each ﬁlter being selected by tasks in N i , which yields a probability distribution over all the ﬁlters [ n (cid:96) ] , i.e., ∀ k ∈ [ n (cid:96) ] , 
Revised: In particular, for each ﬁlter- k ∈ [ n (cid:96) ] from layer- (cid:96) of the pre-trained model, we apply softmax (with temperature τ ) to the times of each ﬁlter being selected by tasks in N i , which yields a probability distribution over all the ﬁlters [ n (cid:96) ] , i.e., ∀ k ∈ [ n (cid:96) ] , [Equation]

Original: [Equation]
Revised: [Equation]

Original: To initialize layer- (cid:96) of the sub-network, MVP samples ﬁlters from this distribution (without replacement) according to the targeted pruning ratio r .
Revised: To initialize layer- (cid:96) of the sub-network, MVP samples ﬁlters from this distribution (without replacement) according to the targeted pruning ratio r .

Original: We further initialize the parameters of each ﬁlter- k by averaging its parameters in the pruned models of the similar tasks which preserve ﬁlter- k .
Revised: We further initialize the parameters of each ﬁlter- k by averaging its parameters in the pruned models of the similar tasks which preserve ﬁlter- k .

Original: MVP then ﬁne-tunes the initialized sub-network for a few iterations on the training set of the target task since MVP targets to keep the computational cost low.
Revised: MVP then ﬁne-tunes the initialized sub-network for a few iterations on the training set of the target task since MVP targets to keep the computational cost low.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct extensive experiments on CIFAR-100 (Krizhevsky, 2009) and ImageNet (Ren et al., 2018) over different pre-trained models, which evaluate MVP (Alg. 1) and compare it with SOTA pruning methods under different settings.
Revised: In this section, we conduct extensive experiments on CIFAR-100 (Krizhevsky, 2009) and ImageNet (Ren et al., 2018) over different pre-trained models, which evaluate MVP (Alg. 1) and compare it with SOTA pruning methods under different settings.

Original: We study the effect of different meta-pruning iterations, neighbour numbers, and similarity metrics for MVP.
Revised: We study the effect of different meta-pruning iterations, neighbour numbers, and similarity metrics for MVP.

Original: All the results show that MVP can outperform other methods with better performance and higher efﬁciency.
Revised: All the results show that MVP can outperform other methods with better performance and higher efﬁciency.

Original: We further validate the strong generalization of MVP by applying it to unseen tasks from Caltech-256.
Revised: We further validate the strong generalization of MVP by applying it to unseen tasks from Caltech-256.

Original: 5. I MPLEMENTATION D ETAILS
Revised: 5. I MPLEMENTATION D ETAILS

Original:  The experiments of MVP are mainly based on the tasks from the dataset introduced in Sec. 3.1.
Revised: [Figure] The experiments of MVP are mainly based on the tasks from the dataset introduced in Sec. 3.1.

Original: For each setting of experiments, we randomly draw 100 test tasks (i.e., the target task in Alg. 1) from the dataset and treat the rest tasks as training tasks.
Revised: For each setting of experiments, we randomly draw 100 test tasks (i.e., the target task in Alg. 1) from the dataset and treat the rest tasks as training tasks.

Original: To evaluate MVP on CNN, we run MVP on the pruned models of ResNet-18 and ResNet-50 for CIFAR-100 and ImageNet respectively.
Revised: To evaluate MVP on CNN, we run MVP on the pruned models of ResNetand ResNet-50 for CIFARand ImageNet respectively.

Original: For both these two experiments, we use the meta-pruning iterations of 100 , batch size of 128 , learning rate of 0 . 01 and optimizer of SGD with the cosine-annealing learning rate schedule.
Revised: For both these two experiments, we use the meta-pruning iterations of 100 , batch size of 128 , learning rate of 0 . 01 and optimizer of SGD with the cosine-annealing learning rate schedule.

Original: For experiments of ViT, MVP is applied to the pruned models of ViT for ImageNet.
Revised: For experiments of ViT, MVP is applied to the pruned models of ViT for ImageNet.

Original: The meta-pruning iterations and batch size are also set as 100 and 128 respectively.
Revised: The meta-pruning iterations and batch size are also set asand 128 respectively.

Original: Following the setting of training ViT in (Touvron et al., 2021), we apply a small learning rate of 0 . 0002 and optimizer of AdamW with the cosine-annealing learning rate schedule.
Revised: Following the setting of training ViT in (Touvron et al., 2021a), we apply a small learning rate of 0 . 0002 and optimizer of AdamW with the cosine-annealing learning rate schedule.

Original: The small number of meta-pruning iterations demonstrates the efﬁciency of MVP.
Revised: The small number of meta-pruning iterations demonstrates the efﬁciency of MVP.

Original: The target pruning ratio of MVP for all tasks is 0 .
Revised: The target pruning ratio of MVP for all tasks is 0 .

Original: All the results of accuracy shown in this section are averaged over the 100 test tasks.
Revised: All the results of accuracy shown in this section are averaged over the 100 test tasks.

Original: 5. B ASELINE M ETHODS
Revised: 5. B ASELINE M ETHODS

Original: We compare MVP with several baselines and SOTA pruning methods.
Revised: We compare MVP with several baselines and SOTA pruning methods.

Original: We ﬁrst implement two baselines to show the advantages of MVP.
Revised: We ﬁrst implement two baselines to show the advantages of MVP.

Original: (1) Conventional pruning.
Revised: (1) Conventional pruning.

Original: We apply a larger number of pruning iterations to extract pruned models for each target task by IFP or AHNP introduced in Sec. 3.1.
Revised: We apply a larger number of pruning iterations to extract pruned models for each target task by IFP or AHNP introduced in Sec. 3.1.

Original: This baseline can be regarded to provide the upper bound performance.
Revised: This baseline can be regarded to provide the upper bound performance.

Original: (2) Random pruning.
Revised: (2) Random pruning.

Original: To validate whether the initialization of MVP makes sense, for each target task, we initialize its sub-network by randomly sampling the same number of nodes/ﬁlters/heads as MVP from the pre-trained model.
Revised: To validate whether the initialization of MVP makes sense, for each target task, we initialize its sub-network by randomly sampling the same number of nodes/ﬁlters/heads as MVP from the pre-trained model.

Original: We take this baseline as the one with the lower bound performance.
Revised: We take this baseline as the one with the lower bound performance.

Original: We also include other SOTA pruning methods as baselines.
Revised: We also include other SOTA pruning methods as baselines.

Original: For MVP on CNN models, we compare
Revised: For MVP on CNN models, we compare

Original: MVP with IHT-based Reptile (Tian et al., 2020), a meta-pruning method that uses Reptile (Nichol et al., 2018) and iterative pruning to ﬁnd better weight-initialization for a pruned meta-model.
Revised: MVP with IHT-based Reptile (Tian et al., 2020), a meta-pruning method that uses Reptile (Nichol et al., 2018) and iterative pruning to ﬁnd better weight-initialization for a pruned meta-model.

Original: Given a new task, it ﬁne-tunes the pruned meta-model for a limited number of iterations to obtain the ﬁnal pruned model.
Revised: Given a new task, it ﬁne-tunes the pruned meta-model for a limited number of iterations to obtain the ﬁnal pruned model.

Original: MEST (Yuan et al., 2021) is the SOTA method in sparse training community, which trains a model from a sparse sub-network so that less computation is required.
Revised: MEST (Yuan et al., 2021) is the SOTA method in sparse training community, which trains a model from a sparse sub-network so that less computation is required.

Original: DLTH (Bai et al., 2022) is based on a variant of the Lottery Ticket Hypothesis.
Revised: DLTH (Bai et al., 2022) is based on a variant of the Lottery Ticket Hypothesis.

Original: It transforms random tickets into winning tickets.
Revised: It transforms random tickets into winning tickets.

Original: We compare MVP with UVC (Yu et al., 2022b) and PoWER (Goyal et al., 2020) on ViT pruning.
Revised: We compare MVP with UVC (Yu et al., 2022b) and PoWER (Goyal et al., 2020) on ViT pruning.

Original: Unlike AHNP, which prunes heads and nodes, UVC also skips the unimportant layers and blocks in ViT. Unlike parameter pruning, PoWER adopts a dynamic method pruning the input patches of each block for each input sample.
Revised: Unlike AHNP, which prunes heads and nodes, UVC also skips the unimportant layers and blocks in ViT. Unlike parameter pruning, PoWER adopts a dynamic method pruning the input patches of each block for each input sample.

Original: For a fair comparison, except for the upper bound baseline, the pruning iterations of all other baselines and MVP are set to 100 .
Revised: For a fair comparison, except for the upper bound baseline, the pruning iterations of all other baselines and MVP are set to 100 .

Original: And the pruning ratios of all methods are set to 0 .
Revised: And the pruning ratios of all methods are set to 0 .

Revised: [Table]

Original: 5. M AIN R ESULTS
Revised: 5. M AIN R ESULTS

Original: The results of applying MVP to tasks from CIFAR-100(ImageNet) on ResNet-18(ResNet-50) supervised and self-supervised pre-trained model, and the baseline methods are reported in Tab.[Table]
Revised: The results of applying MVP to tasks from CIFAR-100(ImageNet) on ResNet-18(ResNet-50) supervised and self-supervised pre-trained model, and the baseline methods are reported in Tab.

Original: On both datasets and pre-trained models [Table] MVP outperforms IFP which spends 10 × iterations of
Revised: On both datasets and pre-trained models, MVP outperforms IFP which spends 10 × iterations of

Original: MVP.
Revised: MVP.

Original: Hence, MVP can produce a higher-quality pruned model when using fewer iterations.
Revised: Hence, MVP can produce a higher-quality pruned model when using fewer iterations.

Original: The results demonstrate that MVP can work well on tasks from both supervised and self-supervised pre-trained models.
Revised: The results demonstrate that MVP can work well on tasks from both supervised and self-supervised pre-trained models.

Original: The random pruning performs much poorer than MVP, which indicates the importance of majority voting from nearest tasks in selecting ﬁlters.
Revised: The random pruning performs much poorer than MVP, which indicates the importance of majority voting from nearest tasks in selecting ﬁlters.

Original: We also compare MVP with SOTA pruning methods for CNN.
Revised: We also compare MVP with SOTA pruning methods for CNN.

Original: IHT-based Reptile Tian et al.
Revised: IHT-based Reptile Tian et al.

Original: (2020) trains a universal sparse sub-network for all target tasks by applying meta-learning on training tasks.
Revised: (2020) trains a universal sparse sub-network for all target tasks by applying meta-learning on training tasks.

Original: MVP achieves higher accuracy than IHT-based Reptile under the same training iterations, implying that MVP can ﬁnd an accurate sub-network for each target task as its initialization and improve its performance.
Revised: MVP achieves higher accuracy than IHT-based Reptile under the same training iterations, implying that MVP can ﬁnd an accurate sub-network for each target task as its initialization and improve its performance.

Original: MEST Yuan et al.
Revised: MEST Yuan et al.

Original: (2021) can speed up pruning by starting training from a well-designed sub-network.
Revised: (2021) can speed up pruning by starting training from a well-designed sub-network.

Original: As a variant of Lottery Ticket Hypothesis, DLTH Bai et al.
Revised: As a variant of Lottery Ticket Hypothesis, DLTH Bai et al.

Original: (2022) proposes a method to transform any random ticket into the winning ticket.
Revised: (2022) proposes a method to transform any random ticket into the winning ticket.

Original: MVP outperforms
Revised: MVP outperforms

Original: MEST and DLTH by a large margin because MVP is trained on a sub-network selected using meta knowledge from similar tasks.
Revised: MEST and DLTH by a large margin because MVP is trained on a sub-network selected using meta knowledge from similar tasks.

Original: In contrast, the initial sub-network for MEST or the winning ticket of DLTH does not leverage any prior knowledge about the target task.
Revised: In contrast, the initial sub-network for MEST or the winning ticket of DLTH does not leverage any prior knowledge about the target task.

Original: [Table]
Revised: [Table]

Original: Tab. 2 shows the comparison between MVP and baseline methods on ViT. Similar to the results on pruning CNN, the performance of MVP on ViT is comparable to AHNP that applies much more pruning iterations.
Revised: Tab. 2 shows the comparison between MVP and baseline methods on ViT. Similar to the results on pruning CNN, the performance of MVP on ViT is comparable to AHNP that applies much more pruning iterations.

Original: The accuracy of random pruning is still much worse.
Revised: The accuracy of random pruning is still much worse.

Original: MVP also outperforms
Revised: MVP also outperforms

Original: SOTA pruning methods developed for ViT.
Revised: SOTA pruning methods developed for ViT.

Original: Hence, on ViT, MVP can efﬁciently produce a small yet high-quality sub-network for each new task by exploiting the nearest tasks’ models.
Revised: Hence, on ViT, MVP can efﬁciently produce a small yet high-quality sub-network for each new task by exploiting the nearest tasks’ models.

Original: The baselines are slower and require more iterations than MVP because they need to re-train the model to achieve a small loss when some parameters or patches are removed.
Revised: The baselines are slower and require more iterations than MVP because they need to re-train the model to achieve a small loss when some parameters or patches are removed.

Original: Both UVC (Yu et al., 2022b) and
Revised: Both UVC (Yu et al., 2022b) and

Original: PoWER (Goyal et al., 2020) cannot recover the accuracy under this strong constraint.
Revised: PoWER (Goyal et al., 2020) cannot recover the accuracy under this strong constraint.

Original: In contrast, the majority voting in MVP directly produces a small sub-network from similar tasks’ models so only a few iterations sufﬁce to reach a downstream task performance comparable to AHNP with 10x iterations. 
Revised: In contrast, the majority voting in MVP directly produces a small sub-network from similar tasks’ models so only a few iterations sufﬁce to reach a downstream task performance comparable to AHNP with 10x iterations. [Figure]

Original: 5. A BLATION S TUDY
Revised: 5. A BLATION S TUDY

Original: Effect of Iteration Numbers Given a new target task and a pre-trained model, MVP can build a well-performed small model in a few iterations, demonstrating its capability in reducing adaptation cost.
Revised: Effect of Iteration Numbers Given a new target task and a pre-trained model, MVP can build a well-performed small model in a few iterations, demonstrating its capability in reducing adaptation cost.

Original: In the left plot of Fig.
Revised: In the left plot of Fig.

Original: 4, we compare MVP with conventional pruning methods using different[Figure] numbers of iterations.
Revised: 4, we compare MVP with conventional pruning methods using different numbers of iterations.

Original: On different architectures of pre-trained models, MVP converges to a high accuracy after nearly 100 iteration.
Revised: On different architectures of pre-trained models, MVP converges to a high accuracy after nearly 100 iteration.

Original: On the contrary, the conventional pruning methods need much more iterations( > 500) to be comparable to MVP.
Revised: On the contrary, the conventional pruning methods need much more iterations( > 500) to be comparable to MVP.

Original: With only ≤ 50 pruning iterations, MVP can reach a reasonable accuracy, while conventional pruning methods perform poorly.
Revised: With only ≤ 50 pruning iterations, MVP can reach a reasonable accuracy, while conventional pruning methods perform poorly.

Original: These imply that the initialized sub-network obtained by majority voting already contains helpful knowledge from its similar tasks to speed up the training of the pruned model.
Revised: These imply that the initialized sub-network obtained by majority voting already contains helpful knowledge from its similar tasks to speed up the training of the pruned model.

Original: Effect of Similarities between Tasks MVP consistently achieves better performance when applied to nearest tasks with the highest similarities.
Revised: Effect of Similarities between Tasks MVP consistently achieves better performance when applied to nearest tasks with the highest similarities.

Original: In the right plot of Fig.
Revised: In the right plot of Fig.

Original: 4, we compare the LEEP score with the Wordnet similarity and study the effect of applying MVP to neighbour tasks with different similarities.
Revised: 4, we compare the LEEP score with the Wordnet similarity and study the effect of applying MVP to neighbour tasks with different similarities.

Original: From similarity group 1 to group 5 , the similarities between tasks decrease.
Revised: From similarity group 1 to group 5 , the similarities between tasks decrease.

Original: We ﬁnd that for both the two similarity metrics, the accuracy of MVP improves signiﬁcantly when the similarities between tasks increase.
Revised: We ﬁnd that for both the two similarity metrics, the accuracy of MVP improves signiﬁcantly when the similarities between tasks increase.

Original: When the pruning iterations are small( = 20 ), where the initialization of the sub-network is more important, the accuracy of tasks from similarity groupleads to similarity group 5 by 15% .
Revised: When the pruning iterations are small( = 20 ), where the initialization of the sub-network is more important, the accuracy of tasks from similarity groupleads to similarity group 5 by 15% .

Original: Despite the accuracy of similarity group 5 improving when the pruning iterations increase to 100 , there is still a gap of 7% .
Revised: Despite the accuracy of similarity group 5 improving when the pruning iterations increase to 100 , there is still a gap of 7% .

Original: This result indicates that neighbour tasks with high similarities share more knowledge with the target task.
Revised: This result indicates that neighbour tasks with high similarities share more knowledge with the target task.

Original: In this plot, we also ﬁnd that tasks in different similarity groups classiﬁed by LEEP score show larger differences than Wordnet similarity, implying that LEEP score can better evaluate similarities between tasks.
Revised: In this plot, we also ﬁnd that tasks in different similarity groups classiﬁed by LEEP score show larger differences than Wordnet similarity, implying that LEEP score can better evaluate similarities between tasks.

Original: This result is consistent with our observation in the empirical study.
Revised: This result is consistent with our observation in the empirical study.

Original: The performance of Wordnet similarity is also good and can still be an alternative when the time and computational resources are limited.
Revised: The performance of Wordnet similarity is also good and can still be an alternative when the time and computational resources are limited.

Original: Comparison between Pruned Models Extracted by Different Pruning Method In this part, we apply MVP to pruned models extracted by Taylor Pruning (Molchanov et al., 2019) on ResNet18 for CIFAR-100 tasks, to prove that MVP works well on pruned models extracted by various pruning methods.
Revised: Comparison between Pruned Models Extracted by Different Pruning Method In this part, we apply MVP to pruned models extracted by Taylor Pruning (Molchanov et al., 2019) on ResNet18 for CIFAR-100 tasks, to prove that MVP works well on pruned models extracted by various pruning methods.

Original: Taylor Pruning measures the importance of each ﬁlter by the effect of removing this ﬁlter on ﬁnal loss.
Revised: Taylor Pruning measures the importance of each ﬁlter by the effect of removing this ﬁlter on ﬁnal loss.

Original: In the left plot of Figure 5, we show the IoU of each layer for pairs of tasks with different task similarities, of which the pruned models are extracted by Taylor Pruning.
Revised: In the left plot of Figure 5, we show the IoU of each layer for pairs of tasks with different task similarities, of which the pruned models are extracted by Taylor Pruning.

Original: Consistent with our observation in the empirical study, pruned models with higher similarities share more ﬁlters.
Revised: Consistent with our observation in the empirical study, pruned models with higher similarities share more ﬁlters.

Original: In the right plot of Figure 5, we investigate the effect of the number of neighbours for MVP.
Revised: In the right plot of Figure 5, we investigate the effect of the number of neighbours for MVP.

Original: When the number = 1 , MVP reduces to transfer learning which learns from the pruned model of a single selected similar task.
Revised: When the number = 1 , MVP reduces to transfer learning which learns from the pruned model of a single selected similar task.

Original: In the plot, when the number of neighbours increase from 1 to 2 , the performance improves sharply.
Revised: In the plot, when the number of neighbours increase from 1 to 2 , the performance improves sharply.

Original: This result implies the effectiveness of meta knowledge from different neighbours.
Revised: This result implies the effectiveness of meta knowledge from different neighbours.

Original: When the number of neighbours ≥ 3 , for both Activation Pruning and
Revised: When the number of neighbours ≥ 3 , for both Activation Pruning and

Original: Taylor Pruning, the accuracy improves little, which indicates that 3 neighbours are enough for MVP to produce a high-quality initialization.[Figure] ee [Figure]
Revised: Taylor Pruning, the accuracy improves little, which indicates that 3 neighbours are enough for MVP to produce a high-quality initialization.

Original: d i ff

Original: [Figure]

Original: r e n t t a

Original: [Figure]
Revised: [Figure]

Original: 5. P ERFORMANCE ON U NSEEN D ATASET
Revised: 5. P ERFORMANCE ON U NSEEN D ATASET

Original: [Table]
Revised: [Table]

Original: In this section, to validate the generalization of MVP, we apply MVP to produce pruned models for target tasks from
Revised: In this section, to validate the generalization of MVP, we apply MVP to produce pruned models for target tasks from

Original: Caltech-256 (Grifﬁn et al., 2007) using the pruned models of tasks from ResNettraining on ImageNet.
Revised: Caltech-256 (Grifﬁn et al., 2007) using the pruned models of tasks from ResNettraining on ImageNet.

Original: The data of Caltech256 is never seen by the pre-trained model and training tasks in the pruned model dataset.
Revised: The data of Caltech256 is never seen by the pre-trained model and training tasks in the pruned model dataset.

Original: Each target task is deﬁned onclasses sampled without replacement from
Revised: Each target task is deﬁned onclasses sampled without replacement from

Original: Caltech-256, and each class has aboutsamples.
Revised: Caltech-256, and each class has aboutsamples.

Original: The performance of MVP on
Revised: The performance of MVP on

Original: Caltech-256 are shown in Tab. 3, which is still comparable to the conventional pruning method using 10 x pruning iterations.
Revised: Caltech-256 are shown in Tab. 3, which is still comparable to the conventional pruning method using 10 x pruning iterations.

Original: When the number of pruning iterations of the conventional pruning method decreases, its performance becomes much worse.
Revised: When the number of pruning iterations of the conventional pruning method decreases, its performance becomes much worse.

Original: The results show that MVP can still produce a high-quality initialization for the task from Caltech-256 by majority voting of similar tasks, so that the pruned model can converge quickly with high accuracy.
Revised: The results show that MVP can still produce a high-quality initialization for the task from Caltech-256 by majority voting of similar tasks, so that the pruned model can converge quickly with high accuracy.

Original: This experiment demonstrates that MVP can be applied to various datasets and generalizes well.
Revised: This experiment demonstrates that MVP can be applied to various datasets and generalizes well.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we study “non-parametric meta-pruning” problem that aims to reduce the memory and computational costs of single-task pruning, via reusing a pre-trained model and similar tasks’ pruned models to ﬁnd an initialization sub-network for a new task.
Revised: In this paper, we study “non-parametric meta-pruning” problem that aims to reduce the memory and computational costs of single-task pruning, via reusing a pre-trained model and similar tasks’ pruned models to ﬁnd an initialization sub-network for a new task.

Original: We conduct an empirical study to investigate the relationship between task similarity and the pruned models of two tasks for different datasets and deep neural networks.
Revised: We conduct an empirical study to investigate the relationship between task similarity and the pruned models of two tasks for different datasets and deep neural networks.

Original: The empirical study motivates a simple yet strong baseline for metapruning, called “meta-vote pruning (MVP)”
Revised: The empirical study motivates a simple yet strong baseline for metapruning, called “meta-vote pruning (MVP)”

Original: (Alg. 1).
Revised: (Alg. 1).

Original: By extensive experiments on multiple tasks drawn from several datasets under different training settings, we demonstrate the advantages of MVP over other SOTA pruning methods in the region of limited computation and show its potential on reducing carbon footprint of pruning/ﬁne-tuning large networks for billions of edge devices and tasks.A PPENDIX
Revised: By extensive experiments on multiple tasks drawn from several datasets under different training settings, we demonstrate the advantages of MVP over other SOTA pruning methods in the region of limited computation and show its potential on reducing carbon footprint of pruning/ﬁne-tuning large networks for billions of edge devices and tasks.

Original: Input : Pre-trained network F ( · ; θ ) , Task T and training set D T , Hyperparameters J, h, r, p

Original: Initialize: Ω (cid:96) ← [ n (cid:96) ] , the set of ﬁlters preserved in layer- (cid:96) 9 for j ← 1 to J do if j % h = 0 and | Ω (cid:96) | > (1 − r ) n (cid:96) then for (cid:96) ← 1 to L − 1 do

Original: Prune p % of ﬁlters in Ω (cid:96) with the smallest importance score over D T ; end end

Original: Apply one SGD step on a mini-batch of D T to ﬁne-tune the remained ﬁlters { θ (cid:96),i : (cid:96)

Original: ∈

Original: [ L − 1] , i ∈ Ω (cid:96) } and θ L ; 16 end

Original: A. I TERATIVE F ILTER P RUNING
Revised: A. I TERATIVE F ILTER P RUNING

Original: The detailed procedure of IFP is described in Algorithm 2.
Revised: The detailed procedure of IFP is described in Algorithm 2.

Original: Given a pre-trained network F ( · ; θ ) of L layers (layer- L is fully-connected) with parameter θ = { θ (cid:96) } (cid:96) =1: L and a training set D T of a target task T , let θ (cid:96) =
Revised: Given a pre-trained network F ( · ; θ ) of L layers (layer- L is fully-connected) with parameter θ = { θ (cid:96) } (cid:96) =1: L and a training set D T of a target task T , let θ (cid:96) =

Original: { θ (cid:96),i } i =1: n (cid:96) denote all parameters in layer- (cid:96) composed of θ (cid:96),i for every ﬁlter- i .
Revised: { θ (cid:96),i } i =1: n (cid:96) denote all parameters in layer- (cid:96) composed of θ (cid:96),i for every ﬁlter- i .

Original: IFP ﬁne-tunes the model for total J iterations.
Revised: IFP ﬁne-tunes the model for total J iterations.

Original: It prunes p % of the ﬁlters remained in each layer every h iterations according to their activation values f (cid:96),i ( x ) .
Revised: It prunes p % of the ﬁlters remained in each layer every h iterations according to their activation values f (cid:96),i ( x ) .

Original: It stops to prune layer- (cid:96) if reaching the targeted pruning ratio r .
Revised: It stops to prune layer- (cid:96) if reaching the targeted pruning ratio r .

Original: A. A UTOMATIC H EAD &N ODE P RUNING
Revised: A. A UTOMATIC H EAD &N ODE P RUNING

Original: The detailed procedure of AHNP is described in Algorithm 3.
Revised: The detailed procedure of AHNP is described in Algorithm 3.

Original: Given a pre-trained network F ( · ; θ ) of L layers (layer- L is fully-connected) with parameter θ = { θ (cid:96) } (cid:96) =1: L and a training set D T of a target task T , let θ (cid:96) =
Revised: Given a pre-trained network F ( · ; θ ) of L layers (layer- L is fully-connected) with parameter θ = { θ (cid:96) } (cid:96) =1: L and a training set D T of a target task T , let θ (cid:96) =

Original: { θ (cid:96),i } i =1: n (cid:96) denote all parameters in layer- (cid:96) composed of θ (cid:96),i for every head/node- i .
Revised: { θ (cid:96),i } i =1: n (cid:96) denote all parameters in layer- (cid:96) composed of θ (cid:96),i for every head/node- i .

Original: S (cid:96),i denote the score for each prunable head/node- i in layer- (cid:96) .
Revised: S (cid:96),i denote the score for each prunable head/node- i in layer- (cid:96) .

Original: AHNP ﬁne-tunes the model and scores for total J iterations.
Revised: AHNP ﬁne-tunes the model and scores for total J iterations.

Original: It prunes the heads/nodes if their scores are smaller than the threshold τ .
Revised: It prunes the heads/nodes if their scores are smaller than the threshold τ .

Original: It stops to prune layer- (cid:96) if reaching the targeted pruning ratio r .
Revised: It stops to prune layer- (cid:96) if reaching the targeted pruning ratio r .

Original: Then, AHNP ﬁne-tunes the pruned model for K iterations.
Revised: Then, AHNP ﬁne-tunes the pruned model for K iterations.

Original: Algorithm
Revised: Algorithm

Original: Input : Pre-trained network F ( · ; θ ) , Task T and training set D T , Hyperparameters J, K, r, τ
Revised: Input : Pre-trained network F ( · ; θ ) , Task T and training set D T , Hyperparameters J, K, r, τ

Original: Initialize: Ω (cid:96) ← [ n (cid:96) ] , the set of heads/nodes preserved in layer- (cid:96) .
Revised: Initialize: Ω (cid:96) ← [ n (cid:96) ] , the set of heads/nodes preserved in layer- (cid:96) .

Original: S (cid:96),i ← 1 , the score for each prunable head/node in layer- (cid:96) .
Revised: S (cid:96),i ← 1 , the score for each prunable head/node in layer- (cid:96) .

Original: 17 for j ← 1 to J do for (cid:96) ← 1 to L − 1 do for i ∈ Ω (cid:96) do
Revised: 17 for j ← 1 to J do for (cid:96) ← 1 to L − 1 do for i ∈ Ω (cid:96) do

Original: Prune the head/node if its score S (cid:96),i < τ ; end end
Revised: Prune the head/node if its score S (cid:96),i < τ ; end end

Original: Stop pruning if reaching the target pruning ratio r .
Revised: Stop pruning if reaching the target pruning ratio r .

Original: Apply one optimization step on a mini-batch of D T to ﬁne-tune the remained heads/nodes and scores { θ (cid:96),i , S (cid:96),i :
Revised: Apply one optimization step on a mini-batch of D T to ﬁne-tune the remained heads/nodes and scores { θ (cid:96),i , S (cid:96),i :

Original: (cid:96) ∈
Revised: (cid:96) ∈

Original: [ L − 1] , i ∈ Ω (cid:96) } and θ L ; 25 end 26 Remove S , ﬁne-tune 
Revised: [ L − 1] , i ∈ Ω (cid:96) } and θ L ; 25 end 26 Remove S , ﬁne-tune the pruned model for K iterations on Di. A. R ESULTS OF APPLYING MVP TO SUB - TASKS OF DIFFERENT SIZES In the experiments of applying MVP to 10 -classiﬁcation and 3 -classiﬁcation sub-tasks, the pruning ratio is set to 85% and 95% respectively. The results are shown in Tab.4.

Revised: From the results we can ﬁnd that on sub-tasks of different sizes, MVP can always achieve comparable or better performance

Revised: [Table]

Revised: than IFP which needs much more computation.

Revised: MVP is applicable to a variety of tasks of different sizes.

Revised: A. C OMPARISON OF I O U BETWEEN MVP AND RANDOM PRUNING

Revised: In Fig.6, besides MVP, we also show the IoU of pruned models extracted by random pruning.

Revised: When the similarity between tasks is large, the IoU of MVP is much larger than random pruning, implying that these tasks contain lots of relevant information.

Revised: When the similarity between tasks is small, in the last few blocks, the IoU of nodes is similar to that of random pruning, which indicates that tasks of low similarity share little high-level information with the target task.

Revised: [Figure]

Revised: A

Revised: T HE DIFFERENCE OF I O U BETWEEN DIFFERENT SIMILARITY GROUPS

Revised: In Fig.7, we draw the difference of IoU between tasks of similarity group 1 and similarity groupfor ResNet-50.

Revised: As the layer gets deeper, the difference increases.

Revised: In Fig.2 of the paper, the average difference of IoU between similarity group 1 and similarity group 5 over all layers is 0 .

Revised: 195 and 0 .

Revised: 150 respectively for LEEP and Wordnet similarity, which has a large gap.

Revised: In Fig.4 of the paper, the LEEP score performs a little better than Wordnet similarity in MVP which indicates that models with the larger IoU share more relevant parameters and LEEP has a good ability to ﬁnd the nearest neighbours.

Revised: [Figure]