Original: One Layer is All You Need Anonymous Author(s)
Revised: Parameter-Efficient Masking Networks io

Original: AffiliationAddress email
Revised: PEMN

Original: Abstract
Revised: Abstract

Original: A deeper network structure generally handles more complicated non-linearity andperforms more competitively.
Revised: A deeper network structure generally handles more complicated non-linearity and performs more competitively.

Original: Nowadays, advanced network designs often containa large number of repetitive structures (e.g., Transformer).
Revised: Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer).

Original: They empower thenetwork capacity to a new level but also increase the model size inevitably, whichis unfriendly to either model restoring or transferring.
Revised: They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring.

Original: In this study, we are thefirst to investigate the representative potential of fixed random weights with limitedunique values byiteratively learning different masks, leading to a new paradigm formodel compression to diminish the model size.
Revised: In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning diverse masks and introduce the Parameter-Efficient Masking Networks (PEMN). It also naturally leads to a new paradigm for model compression to diminish the model size.

Original: Concretely,  we utilize one randominitialized layer, accompanied with different masks, to convey different featuremappings and represent repetitive modules in a deep network.
Revised: Concretely, motivated by the repetitive structures in modern neural networks, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive network modules.

Original: As a result, themodel can be expressed as one-layer with a bunch of masks, which significantlyreduce the model storage cost.
Revised: Therefore, the model can be expressed as one-layer with a bunch of masks, which significantly reduce the model storage cost.

Original: Furthermore, we enhance our strategy by learningmasks for a model filled by padding a given random weights sequence.
Revised: Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights vector.

Original: In thisway, our method can further lower the space complexity, especially for modelswithout many repetitive architectures.
Revised: In this way, our method can further lower the space complexity, especially for models without many repetitive architectures.

Original: We validate the potential of leveragingrandom weights  and test our compression paradigm based on different networkarchitectures.
Revised: We validate the potential of PEMN learning masks on random weights with limited unique values and test its effectiveness for a new compression paradigm based on different network architectures. Code is available at https://github.com/yueb17/PEMN.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks have emerged in several application fields and achieved state-of-the-art performances [5, 13, 23].
Revised: Deep neural networks have emerged in several application fields and achieved state-of-the-art performances [9, 18, 34].

Original: Along with the data explosion in this era, huge amount of data gathered to buildnetwork models with higher capacity [2, 4, 19]
Revised: Along with the data explosion in this era, huge amount of data gathered to build network models with higher capacity [4, 8, 30].

Original: In addition, researchers also pursue a unified networkframework to deal with multi-modal and multi-task problems as a powerful intelligent model [19, 25].
Revised: In addition, researchers also pursue a unified network framework to deal with multi-modal and multi-task problems as a powerful intelligent model [30, 42].

Original: All these trending topics inevitably require even larger and deeper network models to tackle diversedata flows, arising new challenges to compress and transmit models, especially for mobile systems.
Revised: All these trending topics inevitably require even larger and deeper network models to tackle diverse data flows, arising new challenges to compress and transmit models, especially for mobile systems.

Original: Despite the success of recent years with promising task performances, advanced neural networkssuffer from their growing size, which causes inconvenience for both model storage and transferring.
Revised: Despite the success of recent years with promising task performances, advanced neural networks suffer from their growing size, which causes inconvenience for both model storage and transferring.

Original: To reduce the model size of a given network architecture, neural network pruning is a typicaltechnique [17, 15, 9].
Revised: To reduce the model size of a given network architecture, neural network pruning is a typical technique [26, 24, 13].

Original: Pruning approaches remove redundant weights using designed criteria and thepruning operation can be conducted for both pretrained model (conventional pruning: [10, 9]) andrandomly initialized model (pruning at initialization: [16]).
Revised: Pruning approaches remove redundant weights using designed criteria and the pruning operation can be conducted for both pretrained model (conventional pruning: [14, 13]) and randomly initialized model (pruning at initialization: [25, 37]).

Original: Another promising direction is to obtainsparse network by dynamic sparse training [6, 18].
Revised: Another promising direction is to obtain sparse network by dynamic sparse training [10, 29].

Original: They jointly optimize network architecturesand weights to find good sparse networks.
Revised: They jointly optimize network architectures and weights to find good sparse networks.

Original: However, the methods mentioned above demand regulartraining, and the final weights are updated by optimization algorithms like SGD automatically.
Revised: Basically, these methods commonly demand regular training, and the final weights are updated by optimization algorithms like SGD automatically.

Original: Now that the trained weights have such a great representative capacity, one may wonder what isthe potential of random and fixed weights or is it possible to achieve the same thing on randomweights?
Revised: Now that the trained weights have such a great representative capacity, one may wonder what is the potential of random and fixed weights or is it possible to achieve the same performance on random weights?

Original: If we consider a whole network, the answer is obviously negative as a random networkcannot provide informative and distinguishable outputs.
Revised: If we consider a whole network, the answer is obviously negative as a random network cannot provide informative and distinguishable outputs.

Original: However, picking a subnetwork from arandom dense network make it possible as feature mapping varies with changes of subnetwork[Figure] structures.
Revised: However, picking a subnetwork from a random dense network make it possible as feature mapping varies with changes of subnetwork structures.

Original: Then, the question has been updated as what is the  potential of random and fixedweights with selecting subnetwork structures?
Revised: Then, the question has been updated as what is the representative potential of random and fixed weights with selecting subnetwork structures?

Original: Pioneer work LTH [7] shows the winning ticketexists in random network with good trainability but cannot be used directly without further training.
Revised: Pioneer work LTH [11] shows the winning ticket exists in random network with good trainability but cannot be used directly without further training.

Original: Supermasks [26] enhances the winning ticket and enable it being usable directly.
Revised: Supermasks [44] enhances the winning ticket and enable it being usable directly.

Original: Recent work Popup [20] significantly improves subnetwork capacity from its dense counterpart by learning themasks using backpropagation.
Revised: Recent work Popup [31] significantly improves subnetwork capacity from its dense counterpart by learning the masks using backpropagation.

Original: Following this insightful perspective, we further ask a question – whatis the maximum representative potential of a set of random weights?
Revised: Following this insightful perspective, we further ask a question – what is the maximum representative potential of a set of random weights?

Original: In our work, we first make athorough exploration of the question.
Revised: In our work, we first make a thorough exploration of this scientific question to propose our Parameter-Efficient

Original:  Then, back to our practical starting point, we propose a newnetwork compression paradigm by combining a set of fixed random weights with different learnedmasks to represent different subnetworks.
Revised: Masking Networks (PEMN). Then, leveraging on the PEMN, we naturally introduce a new network compression paradigm by combining a set of fixed random weights with a corresponding learned mask to represent the whole network.

Original: We focus on the recent popular network architectures which adopt a similar design style, i.e. , buildinga small-scale encoding module and stacking it to obtain a deep neural network [5, 22, 21].
Revised: We start with network architectures which recent popular design style, i.e. , building a small-scale encoding module and stacking it to obtain a deep neural network [9, 33, 32].

Original: Basedon this point, we naturally propose the One-layer strategy by using one module as a prototype andcopy its parameter into other repetitive structures.
Revised: Based on this point, we naturally propose the One-layer strategy by using one module as a prototype and copy its parameter into other repetitive structures.

Original: More generally, we further provide two versions:max-layer padding (MP) and random weight padding (RP) to handle diverse network structures.
Revised: More generally, we further provide two versions: max-layer padding (MP) and random weight padding (RP) to handle diverse network structures.

Original: Specifically, MP chooses the layer with the most number of parameters as the prototype and usesfirst certain parameters of prototype to fill in other layers.
Revised: Specifically, MP chooses the layer with the most number of parameters as the prototype and uses first certain parameters of prototype to fill in other layers.

Original: RP even breaks the constraint of networkarchitecture.
Revised: RP even breaks the constraint of network architecture.

Original: It samples afixed length random vector  as the prototype which is copied several times tofill in allother layers based on their different lengths.
Revised: It samples a random vector with certain length as the prototype which is copied several times to fill in all layers based on their different lengths.

Original: RP is architecture-agnostic and can be seen asa most general strategy in our work.
Revised: RP is architecture-agnostic and can be seen as a most general strategy in our work.

Original: Three strategies are from specific to general manner and reducethe number of unique parameters gradually.
Revised: Three strategies are from specific to general manner and reduce the number of unique parameters gradually.

Original: We first employ these strategies to randomly initializenetwork.
Revised: We first employ these strategies to randomly initialize network.

Original: Then, we learn different masks to explore the random weights potential and positivelyanswer the  question above.
Revised: Then, we learn different masks to explore the random weights potential and positively answer the scientific question above.

Original: Leveraging on it, we propose a new network compression paradigmby using a set of random weights with a bunch of masks to represent a network model instead ofrestoring sparse weights for all layers (see Fig. 1).
Revised: Leveraging on it, we propose a new network compression paradigm by using a set of random weights with a bunch of masks to represent a network model instead of restoring sparse weights for all layers (see Fig. 1).

Original: We conduct comprehensive experiments to explorethe random weights  potential and test the model compression performance to validate our paradigm.
Revised: We conduct comprehensive experiments to explore the random weights representative potential and test the model compression performance to validate our paradigm.

Original: We summarize our contributions as below:
Revised: We summarize our contributions as below:

Original: • We  explore themaximum representative potential of  random weights with limited unique 
Revised: • We scientifically explore the representative potential of fixed random weights with limited unique values and introduce our Parameter-Efficient Masking Networks (PEMN).

Original: values, leveraging on learning different masks to represent different feature mappings.
Revised: It leverages on learning different masks to represent different feature mappings.

Original: • A novel network compression paradigm is  proposed by fully utilizing the representative capacity of random weights.
Revised: • A novel network compression paradigm is naturally proposed by fully utilizing the representative capacity of random weights.

Original: We  restore a network based on a given random vector withdifferent masks instead of retaining all the sparse weights.•
Revised: We represent and restore a network based on a given random vector with a bunch of masks instead of retaining all the sparse weights.

Original:  Extensive experimental results explore the random weights potential  and test the compression performance of our new paradigm.
Revised: • Extensive experimental results explore the random weights potential by using our PEMN and test the compression performance of our new paradigm.

Original: We expect our work can inspire more interestingexplorations in this direction.
Revised: We expect our work can inspire more interesting explorations in this direction.

Original:  Related Works2.1 Sparse Network Training
Revised:  Related Works 2.1 Sparse Network Training

Original: Our work is related to sparse network training.
Revised: Our work is related to sparse network training.

Original: Conventional pruning techniques finetune the prunednetwork from pretrained models [9, 10] with various pruning criteria [17, 11, 12].
Revised: Conventional pruning techniques finetune the pruned network from pretrained models [13, 14] with various pruning criteria for different applications [26, 15, 16, 38, 36, 40, 41].

Original: Instead of pruninga pretrained model, pruning at initialization  approaches attempt to find winning ticket from therandom weight.
Revised: Instead of pruning a pretrained model, pruning at initialization [37] approaches attempt to find winning ticket from the random weight.

Original: Gradient information is considered to build pruning criteria in [16, 24].
Revised: Gradient information is considered to build pruning criteria in [25, 35].

Original: Differentfrom pruning methods above, sparse network training also can be conducted in a dynamic fashion.
Revised: Different from pruning methods above, sparse network training also can be conducted in a dynamic fashion.

Original: Toname a few, Rigging the Lottery [6] edits the network connections and jointly updates the learnableweights.
Revised: To name a few, Rigging the Lottery [10] edits the network connections and jointly updates the learnable weights.

Original: Dynamic Sparse Reparameterization [18] modifies the parameter budget among the wholenetwork dynamically.
Revised: Dynamic Sparse Reparameterization [29] modifies the parameter budget among the whole network dynamically.

Original: Sparse Networks from Scratch [3] proposes a momentum based approach toadaptively grow weights and empirically  verifies its effectiveness.
Revised: Sparse Networks from Scratch [7] proposes a momentum based approach to adaptively grow weights and empirically [Figure] verifies its effectiveness.

Original: Most of the sparse network trainingachieve the network sparsity by keeping necessary weights and removing others, which reduces thecost of model storage and transferring.
Revised: Most of the sparse network training achieve the network sparsity by keeping necessary weights and removing others, which reduces the cost of model storage and transferring.

Original: In our work, we propose a novel model compression paradigmby leveraging the  potential of random weights accompanied with subnetwork selection.
Revised: In our work, we propose a novel model compression paradigm by leveraging the representative potential of random weights accompanied with subnetwork selection.

Original: 2.2 Random Network Selection
Revised: 2.2 Random Network Selection

Original: Our work inherits the research line of exploring the representative capacity of random network.
Revised: Our work inherits the research line of exploring the representative capacity of random network.

Original: Thepotential of randomly initialized network is pioneeringly explored by the Lottery Ticket Hypothesis [7].
Revised: The potential of randomly initialized network is pioneeringly explored by the Lottery Ticket Hypothesis [11], and further investigated by [28, 2, 39].

Original: It articulates that there exists a winning ticket subnetwork in a random dense network.
Revised: It articulates that there exists a winning ticket subnetwork in a random dense network.

Original: Thissubnetwork can be trained in isolation and achieves comparable results with its dense counterpart.
Revised: This subnetwork can be trained in isolation and achieves comparable results with its dense counterpart.

Original: Moreover, the potential of the winning ticket is further explored in Supermasks [26].
Revised: Moreover, the potential of the winning ticket is further explored in Supermasks [44].

Original: It surprisinglydiscovers the subnetwork can be identified from dense network to obtain reasonable performancewithout training.
Revised: It surprisingly discovers the subnetwork can be identified from dense network to obtain reasonable performance without training.

Original: It extends and proves the potential of subnetwork from good trainability to beingused directly.
Revised: It extends and proves the potential of subnetwork from good trainability to being used directly.

Original: More recently, the representative capacity of subnetworks is enhanced by Popupalgorithm proposed by [20].
Revised: More recently, the representative capacity of subnetworks is enhanced by Popup algorithm proposed by [31].

Original: Based on random dense initialization, the learnable mask is optimized toobtain subnetwork with promising results.
Revised: Based on random dense initialization, the learnable mask is optimized to obtain subnetwork with promising results.

Original: Instead of considering network with random weights, thenetwork with the same shared parameters can also delivery representative capacity to some extent,which is investigated by Weight Agnostic Neural Network [8] and also inspires this research direction.
Revised: Instead of considering network with random weights, the network with the same shared parameters can also delivery representative capacity to some extent, which is investigated by Weight Agnostic Neural Network [12] and also inspires this research direction.

Original: We are highly motivated by these researches to validate how is the representative potential of randomweights with limited unique values and repetitive structures.
Revised: We are highly motivated by these researches to validate how is the representative potential of random weights with limited unique values by learning various masks.

Revised: 2.3 Weight Sharing

Revised: Our study is also related to several recent works about weight sharing.

Original: Methodology
Revised: This strategy has been explored and analyzed in convolutional neural networks for their efficiency [19, 43]. In addition, several works are also proposed for efficient transformer architecture using weight sharing strategy [22, 5, 1]. There are two main differences between these works and our study: 1) They follow the regular optimization strategy to learn the weight in a recurrent fashion, which is closer to the recurrent neural network. Our work follows a different setting.

Revised: We use fixed repetitive random weights to fill in the whole network and employs different masks to represent different feature mappings; 2) They mainly conduct cross-layer weight sharing for repetitive transformer structure.

Revised: In our work, we explore the potential of random weight vector with limited length as much smaller repetitive granularity to fill in the whole network, which is more challenging than cross-layer sharing strategy.

Original:  3.1 Instinctive Motivation
Revised: [Figure]  Parameter-Efficient Masking Networks 3.1 Instinctive Motivation

Original: Overparameterized randomly initialized neural network benefits network optimization to get higherperformance.
Revised: Overparameterized randomly initialized neural network benefits network optimization to get higher performance.

Original: Inevitably, the trained network contains redundant parameters but can be furthercompressed, which defines the conventional neural network pruning.
Revised: Inevitably, the trained network contains redundant parameters but can be further compressed, which defines the conventional neural network pruning.

Original: On the other side, the networkredundancy also ensures a large random network contains a huge number of possible subnetworks,thus, carefully selecting a specific subnetwork should obtain promising performances.
Revised: On the other side, the network redundancy also ensures a large random network contains a huge number of possible subnetworks, thus, carefully selecting a specific subnetwork should obtain promising performances.

Original: This pointof view has been proved by [20, 26].
Revised: This point of view has been proved by [31, 44].

Original: These works demonstrate the huge potential of certain subsetcombinations of a given random weights.
Revised: These works demonstrate the representative potential of certain subset combinations of a given random weights.

Original: Following this lane, we naturally ask a question: whatis the maximum  potential of a set of random weights?
Revised: Following this lane, we naturally ask a question: what is the maximum representative potential of a set of random weights?

Original: or in another word: can we use less randomweights with selected subnetwork to represent a usable network?
Revised: or in another word: can we use random weights with limited unique values to represent a usable network?

Original: We answer this question as positive:we can use less random weights to deliver a usable network.
Revised: We answer this question as positive and introduce our Parameter-Efficient Masking Networks (PEMN).

Original: Moreover, leveraging on 1) comparedwith trained network where the weight values cannot be predicted, we can pre-access the randomweights before we select the subnetwork; 2) selected subnetwork can be efficiently represented by abunch of masks, we can extremely reduce the network storage size and establish a new paradigm formodel compression.
Revised: Moreover, leveraging on 1) compared with trained network where the weight values cannot be predicted, we can pre-access the random weights before we select the subnetwork; 2) selected subnetwork can be efficiently represented by a bunch of masks, we can extremely reduce the network storage size and establish a new paradigm for network compression.

Original: 3.2 Sparse Selection
Revised: 3.2 Sparse Selection

Original: We follow [20] to conduct the sparse network selection.
Revised: We follow [31] to conduct the sparse network selection.

Original: We start from a randomly initialized neuralnetwork consisting of L layers.
Revised: We start from a randomly initialized neural network consisting of L layers.

Original: For each l ∈ { 1 , 2 , ..., L } , it has
Revised: For each l ∈ { 1 , 2 , ..., L } , it has

Original: I l +1 =

Original: σ ( F [ I l ; w l ]) , (1)

Original: [Figure]
Revised: [Equation]

Original: Regular

Original: One-Layer

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where I l and I l +1 are the input and output of layer l .
Revised: where I l and I l +1 are the input and output of layer l .

Original: σ is the activation.
Revised: σ is the activation.

Original: F represents the encodinglayer such as convolutional or linear layer with parameter w l = { w 1 l , w 2 l , ..., w d l l } , where d l is theparameter dimension of layer l .
Revised: F represents the encoding layer such as convolutional or linear layer with parameter w l = { w 1 l , w 2 l , ..., w d l l } , where d l is the parameter dimension of layer l .

Original: To perform the sparse selection, all the weights w = { w 1 , w 2 , ..., w L }are fixed and denoted as (cid:101) w .
Revised: To perform the sparse selection, all the weights w = { w 1 , w 2 , ..., w L } are fixed and denoted as (cid:101) w .

Original: To pick the fixed weights for subnetwork, each weight w jl is assigned alearnable element-wise score s jl to indicate its importance in the network.
Revised: To pick the fixed weights for subnetwork, each weight w jl is assigned a learnable element-wise score s jl to indicate its importance in the network.

Original: The Eq. 1 is rewrited as
Revised: The Eq. 1 is rewrited as

Original: [Equation]
Revised: [Equation]

Original: where s l = { s 1 l , s 2 l , ..., s d l l } is the score vector and h ( · ) is the indicator function to create the mask.
Revised: where s l = { s 1 l , s 2 l , ..., s d l l } is the score vector and h ( · ) is the indicator function to create the mask.

Original: Itoutputs 1 when the value of s jl belongs to the top K% highest scores and outputs 0 for others, where
Revised: It outputs 1 when the value of s jl belongs to the top K% highest scores and outputs 0 for others, where

Original: K is predefined sparse selection ratio.
Revised: K is predefined sparse selection ratio.

Original: Through optimizing s with fixed w , a subset of original denseweights is finally selected.
Revised: Through optimizing s with fixed w , a subset of original dense weights is finally selected.

Original: Since h ( · ) is a non-derivable function, the gradient of each s jl cannotbe obtained directly.
Revised: Since h ( · ) is a non-derivable function, the gradient of each s jl cannot be obtained directly.

Original: The straight-through gradient estimator [1] is applied to treat h ( · ) as identityfunction during gradient backwards pass.
Revised: The straight-through gradient estimator [3] is applied to treat h ( · ) as identity function during gradient backwards pass.

Original: Formally, the gradient of s is approximately computed as
Revised: Formally, the gradient of s is approximately computed as

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) I
Revised: where (cid:101) I

Original: l +1 =
Revised: l +1 =

Original: σ ( F [ I l ; w l ⊙ s l ]) , which is applied estimation.
Revised: σ ( F [ I l ; w l ⊙ s l ]) , which is applied estimation.

Original: (cid:101) g ( s jl ) is approximately estimatedgradient of weight score s j l .
Revised: (cid:101) g ( s jl ) is approximately estimated gradient of weight score s jl .

Original: In this way, the dense network is randomly initialized but fixed, but oneof its subnetwork can be selected using Backpropagation.
Revised: In this way, the dense network is randomly initialized but fixed, but one of its subnetwork can be selected using Backpropagation.

Original: In our work, we name this optimizationprocess as sparse selection .
Revised: In our work, we name this optimization process as sparse selection .

Original: 3.3 One Layer is All You Need
Revised: 3.3 Parameter-Efficient Strategy

Revised: Following the logic of parameter-efficient exploration from specific to general scenario, PEMN utilizes three strategies to construct the whole network based on given random weights: 1) One-layer (Sec. 3.3.1), 2) Max-layer padding (MP) (Sec. 3.4), and 3) Random vector padding (RP) (Sec. 3.4), which are detailedly introduced below. 3.3.1 One-Layer prototype layers parameters w pro P }

Revised: R l then 8:

Revised: Append w l into List pro 9: else 10:

Original:  Sparse selection initializes a dense network as a pool to pick certain weights.
Revised: Find w pro p , where R pro p = R l 11: Replace w l with w pro p 12: end if 13: end for 14: Return updated w as w ∗ Sparse selection initializes a dense network as a pool to pick certain weights.

Original: It provides a noveldirection to find admirable subnetwork without pretraining and pruning.
Revised: It provides a novel direction to find admirable subnetwork without pretraining and pruning.

Original: However, with increasing ofnetwork scales, the cost of restoring and transfering a neural network grows rapidly.
Revised: However, with increasing of network scales, the cost of restoring and transfering a neural network grows rapidly.

Original: Noticed thatmore popular network structures follow a similar design style: proposing a well-designed modelingblock and stacking it several times to boost network capacity, we are inspired to explore the feasibilityof finding a subnetwork by iteratively selecting different subnetworks in one set of repetitive modules.
Revised: Noticed that more popular network structures follow a similar design style: proposing a well-designed modeling block and stacking it several times to boost network capacity , we are inspired to explore the feasibility of finding subnetworks by iteratively selecting different masks in a series of repetitive modules.

Original: Formally, a L -layer randomly initialized network N L can be represented as a series of parameters:
Revised: Formally, a L -layer randomly initialized network N L can be represented as a series of parameters:

Original: [Equation]
Revised: [Equation]

Original: where  w l is used for various layers.
Revised: where l ∈ { 1 , 2 , ..., L } and w l is used for various layers.

Original: R l denotes different parameter spaces 
Revised: R l denotes different parameter spaces (e.g.,

Original: (e.g., R I × O l for linear,

Original: R  N × H × W l for CNN layer, where I / O are input/output dimensions and N / H / W are CNN kernel dimensions).
Revised: R I × O l for linear, R N × H × W l for CNN layer, where I / O are input/output dimensions and N / H / W are CNN kernel dimensions).

Original: From shallow to deep layer, we first sample the prototype layers for the whole networkwith unique parameter spaces, represented as w pro = [ w pro 1 , w pro 2 , ..., w pro P ] .
Revised: From shallow to deep layer, we first sample the prototype layers for the whole network with unique parameter spaces, represented as w pro = [ w pro 1 , w pro 2 , ..., w pro P ] .

Original: For each w pro p , weuse its parameters to replace its target layers w tar p which share the same parameter space:
Revised: For each w pro p , we use its parameters to replace its target layers w tar p which share the same parameter space:

Original: [Equation]
Revised: [Equation]

Revised: where T p is the number of target layers of prototype p .

Original: The whole network filled by several layers with unique weight size and Eq.
Revised: The whole network filled by several layers with unique weight size and Eq.

Original: 4 can be rewrited as
Revised: 4 can be rewrited as

Original: [Equation]
Revised: [Equation]

Original: We take a simple 5-layer MLP to clarify this operation: its dimensions are [512 , 100 , 100 , 100 , 10]with 4 weight matrices w 1 ∈ R 512 × 100 , w 2 ∈ R 100 × 100 , w 3 ∈ R 100 × 100 , and w 4 ∈ R 100 × 10 .
Revised: We take a simple 5-layer MLP to clarify this operation: its dimensions are [512 , 100 , 100 , 100 , 10] with 4 weight matrices w 1 ∈ R 512 × 100 , w 2 ∈ R 100 × 100 , w 3 ∈ R 100 × 100 , and w 4 ∈ R 100 × 10 .

Original: In thiscase, w 1 , w 2 and w 4 are three prototype layers.
Revised: In this case, w 1 , w 2 and w 4 are three prototype layers.

Original: w 2 has two target layers w 3 and itself.
Revised: w 2 has two target layers w 3 and itself.

Original: w 1 / w 4 onlyhas itself as target layer.
Revised: w 1 / w 4 only has itself as target layer.

Original: The general algorithm is summarized in Alg. 1.
Revised: The general algorithm is summarized in Alg. 1.

Original: Append w l into List pro 9: else 10:

Original: Find w pro p , where R pro p =

Original: R l 11:

Original: Replace w l with w pro p 12: end if 13: end for 14: Return updated w as w ∗

Original: In this way, any repetitive modules in a given network structure can be represented by one bunchof random weights.
Revised: In this way, any repetitive modules in a given network structure can be represented by one bunch of random weights.

Original: Using the sparse selectionstrategy, we iteratively pick subnetworks in thesame set of random weights to obtain diverse feature mappings.
Revised: Using the sparse selection strategy, we iteratively pick subnetworks in the same set of random weights to obtain diverse feature mappings.

Original: Therefore, the cost to representthe network significantly reduces, especially fordeep network with many repetitive blocks.
Revised: Therefore, the cost to represent the network significantly reduces, especially for deep network with many repetitive blocks.

Original: Inother words, one random layer with differentmasks can represent the majority of a completenetwork structure, which is named as one layeris all you need (one-layer) .
Revised: In other words, one random layer with different masks can represent the majority of a complete network structure, which is named as one-layer .

Original: 3.4 Random Weights Padding
Revised: 3.4 Random Weights Padding

Original: One-layer strategy efficiently handles networkswith many repetitive modules.
Revised: One-layer strategy efficiently handles networks with many repetitive modules.

Original: The majority ofthe whole network can be compressed into onerandom layeraccompanied with a set of masks.
Revised: The majority of the whole network can be compressed into one random layer with a set of masks.

Original: However, in real-world applications, various network architectures may not follow a tidy patternresulting in different shapes for different layers.
Revised: However, in real-world applications, various network architectures may not follow a tidy pattern resulting in different shapes for different layers.

Original: Hence, the one-layerstrategy is not flexible enoughto efficiently represent such networks.
Revised: Hence, the one-layer is not flexible enough to efficiently represent such networks.

Original: Naturally, we further propose a enhanced strategy, Random Weights Padding , to handle this scenario.
Revised: To handle it, we naturally propose a enhanced strategy, Random Weights Padding .

Original: We provide two versions, Max-Layer Padding and Random Vector Padding , to achieve it.
Revised: It consists of two versions, Max-layer padding and Random vector padding .

Original: We first formally rewrite Eq.
Revised: We first formally rewrite Eq.

Original: 4 as
Revised: 4 as

Original: [Equation]
Revised: [Equation]

Original: (7) where w l is flatten into a vector with dimension d l (e.g., d l = I × O for linear  and d l = N × H × Wfor CNN).
Revised: where w l is flatten into a vector with dimension d l (e.g., d l = I × O for linear layer and d l = N × H × W for CNN layer).

Original: Max-Layer Padding (MP).
Revised: Max-Layer Padding (MP).

Original: It chooses the layer w m as the prototype where d m =max ([ d 1 , d 2 , ..., d L ]) is the highest dimension.
Revised: It chooses the layer w m as the prototype where d m = max ([ d 1 , d 2 , ..., d L ]) is the highest dimension.

Original: All other layers in N L have fewer parameters thanw m .
Revised: All other layers in N L have fewer parameters than w m .

Original: We keep the prototype as it is and simply pick the first d l parameters from w m to replace theparameters in w l , which is described by
Revised: We keep the prototype as it is and simply pick the first d l parameters from w m to replace the parameters in w l , which is described by

Original: [Equation]
Revised: [Equation]

Original: Random Vector Padding (RP).
Revised: Random Vector Padding (RP).

Original: Instead of picking a complete layer as prototype, RP further reducesthe granularity of random prototype from a layer to a random weights vector with a relatively shortlength.
Revised: Instead of picking a complete layer as prototype, RP further reduces the granularity of random prototype from a layer to a random weights vector with a relatively short length.

Original: We let v pro ∈ R d v as the random weights vector with length d v .
Revised: We let v pro ∈ R d v as the random weights vector with length d v .

Original: For each layer l , we repeatv pro several times to reach the length of w l .
Revised: For each layer l , we repeat v pro several times to reach the length of w l .

Original: It can be formally described as
Revised: It can be formally described as

Original: [Equation]
Revised: [Equation]

Original: After the padding operation, weights in Eq.
Revised: After the padding operation, weights in Eq.

Original: 7 are reshaped back into the format of Eq.
Revised: 7 are reshaped back into the format of Eq.

Original: 4 to performas a network.
Revised: 4 to perform as a network.

Original: These two padding strategies are summarized in Alg. 2 and Alg. 3.
Revised: These two padding strategies are summarized in Alg. 2 and Alg. 3.

Original: In the series of one-layer , MP ,  RP , based on sparse selection , we explore using fewer unique weightsto represent the whole network.
Revised: In the series of one-layer , MP , and RP , based on sparse selection , we explore using fewer unique weights to represent the whole network.

Original: Leveraging on the property of the fixed weight values, the cost ofdelegating a network keeps decreasing by using a random vector with a bunch of masks.
Revised: Leveraging on the property of the fixed weight values, the cost of delegating a network keeps decreasing by using a random vector with a bunch of masks.

Original: By this mean,we fully explore the representative capacity of random weights with limited length.
Revised: By this mean, we fully explore the representative capacity of random weights with limited unique values.

Original: Furthermore,a novel model compression paradigm is correspondingly established by restoring a set of randomweights with different masks.
Revised: Furthermore, a novel model compression paradigm can be correspondingly established by restoring a set of random weights with different masks.

Original: Our three strategies compared to regular network setting are shown in 
Revised: Our three strategies compared to regular network setting are shown in Fig. [Table]

Original: Fig.
Revised: [Equation] [Table] [Equation] [Table]

Original: length
Revised: [Equation] [Table] [Figure] parameter settings.

Original: d l 7:
Revised: Dense means the model is trained in regular fashion. Mask is the sparse selection strategy. One-layer , MP, and RP are our strategies.

Original: Replace w l with  the repeated vector v pro 8: end for 9: Return updated w as w ∗
Revised: The decimal after RP means the number of unique parameters compared with MP . From Mask to RP 1e-5 , the unique values of network decrease. Different experimental settings illustrate the representative potential of random weights.

Original:  Experiments
Revised:  Experiments

Original: Our experiments conduct empirically validations on two aspects of our interests.
Revised: Our experiments conduct empirically validations on two aspects of our interests.

Original: Firstly, followingthe sparse selection line, how large is the representative potential of random weights with limitedunique values?
Revised: Firstly, we validate how large is the representative potential of random weights with limited unique values to test the effectiveness of our proposed Parameter-Efficient Masking Networks (PEMN).

Original: Secondly, leveraging on the pre-accessibility of random weights and lightweightstorage cost of binary mask, it is promising to establish a new model compression paradigm.
Revised: Secondly, leveraging on the pre-accessibility of random weights and lightweight storage cost of binary mask, it is promising to establish a new model compression paradigm.

Original: 4.1 Preparation
Revised: 4.1 Preparation

Original: We comprehensively use several classic or recently popular backbones for image classification taskto conduct general validations.
Revised: We comprehensively use several classic or recently popular backbones for image classification task to conduct general validations.

Original: Backbones include ResNet32, ResNet56 [13], ConvMixer [22], and
Revised: Backbones include ResNet32, ResNet56 [18], ConvMixer [33], and

Original: ViT [5].
Revised: ViT [9].

Original: We use CIFAR10 and CIFAR100 datasets [14] for our experiments.
Revised: We use CIFAR10 and CIFAR100 datasets [21] for our experiments.

Original: 4.2 Representative Random Weights 
Revised: 4.2 Representative Random Weights in PEMN

Original: We first explore the representative potential of random weights  based on our proposed strategies, One-Layer , Max-Layer Padding (MP) , and Random Vector Padding (RP) , in Sec.
Revised: We first explore the representative potential of random weights in PEMN based on our proposed strategies, One-Layer , Max-Layer Padding (MP) , and Random Vector Padding (RP) .

Original: We use a CNNbased architecture Convmixer [22] and a MLP based model ViT [5] to conduct our experiments on
Revised: We use a CNN based architecture Convmixer [33] and a MLP based model ViT [9] to conduct our experiments on

Original: CIFAR10 dataset [14].
Revised: CIFAR10 dataset [21].

Original: In Fig.
Revised: In Fig.

Original: 3, we show 8 pairs of experiments based on 2 backbones (ConvMixer, ViT) using 2 depthnumbers (6, 8) and 2 hidden dimensions (256, 512).
Revised: 3, we show 8 pairs of experiments based on 2 backbones (ConvMixer, ViT) using 2 depth numbers (6, 8) and 2 hidden dimensions (256, 512).

Original: Each pair includes a dense network performanceand a series of results obtained by sparse selection with different random weighting strategies.
Revised: Each pair includes a dense network performance and a series of results obtained by sparse selection with different random weighting strategies.

Original: Specifically, Mask learns the mask on the whole network.
Revised: Specifically, Mask learns the mask on the randomly initialized network.

Original: One-layer , MP , and RP represent ourproposed strategies.
Revised: One-layer , MP , and RP represent our proposed strategies.

Original: To simplify the comparison, we use a rate number after RP to show how manyunique parameters used in RP compared with MP . 
Revised: To simplify the comparison, we use a rate number after RP to show how many unique parameters used in RP compared with MP . From the left Mask to right [Figure] [Figure]

Revised: [Figure]

Original: From the left Mask to right RP 1e-5 , the numberof unique parameters gradually decreases.
Revised: [Figure] This pair of figures show that our proposed paradigm achieves admirable compression performance compared with baselines. In very high compression ratios, we can still maintain the test accuracy. RP 1e-5 , the number of unique parameters gradually decreases.

Original: Different settings show the similar patterns concluded as:1)
Revised: Different settings show the similar patterns concluded as: 1)

Original: Compared with regular trained dense model, sparse selection approach generally obtains promisingresults, even if with a performance drop caused by the constraint of fixing all the parameters; 2) Fromleft to right on X-axis, the performance gradually drops.
Revised: Compared with regular trained dense model, sparse selection approach generally obtains promising results, even if with a performance drop caused by the constraint of fixing all the parameters; 2) From left to right on X-axis, the performance gradually drops.

Original: This is caused by the decreasing numberof unique values in network, which makes network has less representative capacity; 3)
Revised: This is caused by the decreasing number of unique values in network, which makes network has less representative capacity; 3)

Original: However,performance drop arises when the number of unique parameters is extremely low (e.g., RP 1e-4 , RP 1e-5 ).
Revised: However, performance drop arises when the number of unique parameters is extremely low (e.g., RP 1e-4 , RP 1e-5 ).

Original: The results remain stable for the most of random weights strategies; 4) Larger depth and [Figure] parameter settings. Dense means the model is trained in regular fashion.
Revised: The results remain stable for the most of random weights strategies; 4) Larger depth and hidden dimension boost the model capacity for different configurations.

Original: Mask is the sparse selection strategy.

Original: One-layer , MP, and RP are our strategies.

Original: The decimal after RP means the number of unique parameters compared with MP .

Original: From Mask to RP 1e-5 , the unique values of network decrease.

Original: Different experimental settings illustrate the representative potential of random weights.

Original: hidden dimension boost the model capacity for different configurations.

Original: The performance drop ofrandom weights strategies from their dense counterpart is also decreased.
Revised: The performance drop of random weights strategies from their dense counterpart is also decreased.

Original: In addition, ViT showssome unstable fluctuation when fewer unique parameter, compared with ConvMixer with relativelystable patterns.
Revised: In addition, ViT shows some unstable fluctuation when fewer unique parameter, compared with ConvMixer with relatively stable patterns.

Original: This may caused by the difficulty of training MLP based network itself and will notaffect our main conclusions.
Revised: This may caused by the difficulty of training MLP based network itself and will not affect our main conclusions.

Original: The performance stability shown above illustrate the network representative capacity can be realizednot only by overparameterizing the model, but also carefully picking different combinations of limitedrandom parameters.
Revised: The performance stability shown above illustrates the network representative capacity can be realized not only by overparameterizing the model, but also carefully picking different combinations of random parameters with limited unique values.

Original: In this way,  we can represent a network using a random parameters prototypewith different learned masks, instead of typically restoring all the different parameters.
Revised: In this way, the effectiveness of our PEMN is validated and we can represent a network using a random parameters prototype with different learned masks, instead of typically restoring all the different parameters.

Original: This propertyinspires us to deliver a new model compression paradigm proposed in following section.
Revised: This property inspires us to introduce a new model compression paradigm proposed in following section.

Original: 4.3 A New  Compression Paradigm
Revised: 4.3 A New Model Compression Paradigm

Original: Practically, our work proposes a new network compression paradigm based on a group of randomweights with different masks.
Revised: Practically, our work proposes a new network compression paradigm based on a group of random weights with different masks.

Original: We first elaborate the network compression and storage processes toclarify our advantages then report the empirical results.
Revised: We first elaborate the network compression and storage processes to clarify our advantages then report the empirical results.

Original: 4.3.1 Sparse Network Storage
Revised: 4.3.1 Sparse Network Storage

Original: Previous works aim to remove redundant weights (e.g., unstructured pruning) among different layers.
Revised: Previous works aim to remove redundant weights (e.g., unstructured pruning) among different layers.

Original: The trivial weights are set to zero based on different criteria.
Revised: The trivial weights are set to zero based on different criteria.

Original: The ratio of zero-weight in the wholenetwork is regarded as sparsity ratio.
Revised: The ratio of zero-weight in the whole network is regarded as sparsity ratio.

Original: Different approaches are compared based on their final testaccuracy with a given sparsity ratio.
Revised: Different approaches are compared based on their final test accuracy with a given sparsity ratio.

Original: Different from this conventional fashion which restoring sparsetrained weights, we instead use fixed random weights with different masks to represent a network.
Revised: Different from this conventional fashion which restoring sparse trained weights, we instead use fixed random weights with different masks to represent a network.

Original: Tocompare these two paradigm, we calculate the required storage size as an integrated measurement.[Figure] [Figure] cc
Revised: To compare these two paradigm, we calculate the required storage size as an integrated measurement.

Original: [Figure]

Original: CIFAR

Original: [Figure]

Original: 97 Size

Original: [Figure]

Original: [Figure]

Original: This pair of figures show that our proposed paradigm achieves admirable compression performance compared with baselines.

Original: In very high compression ratios, we can still maintain the test accuracy.

Original: Assuming we have a trained network with p as parameter numbers and r as sparsity ratio.
Revised: Assuming we have a trained network with p as parameter numbers and r as sparsity ratio.

Original: Due toits sparsity, we only need to restore the non-zero weight values accompanied with their position [9]denoted as a binary mask.
Revised: Due to its sparsity, we only need to restore the non-zero weight values accompanied with their position [13] denoted as a binary mask.

Original: The storage cost can be separated into two parts, C w for weight values and
Revised: The storage cost can be separated into two parts, C w for weight values and

Original: C m for mask given by
Revised: C m for mask given by

Original: [Equation]
Revised: [Equation]

Original: For conventional setting, C w restores the values of kept sparse weights which is p · (1 − r ) for thewhole network.
Revised: For conventional setting, C w restores the values of kept sparse weights which is p · (1 − r ) for the whole network.

Original: It needs to be restore in float format.
Revised: It needs to be restore in float format.

Original: C m restores sparse positions of these weights,which can be restored into compressed sparse column (CSC) or compressed sparse row (CSR) formatswith cost around 2 p · (1 − r ) 
Revised: C m restores sparse positions of these weights, which can be restored into compressed sparse column (CSC) or compressed sparse row (CSR) formats with cost around 2 p · (1 − r ) [13].

Original: In our new paradigm, C w records the values of the given randomweights.
Revised: In our new paradigm, C w records the values of the given random weights.

Original: For example, the one-layer requires to record all weights of non-repetitive layers and oneprototype weights of all repetitive layers.
Revised: For example, the one-layer requires to record all weights of non-repetitive layers and one prototype weights of all repetitive layers.

Original: MP requires to keep the values of layer with the largestnumber of parameters.
Revised: MP requires to keep the values of layer with the largest number of parameters.

Original: RP only requires to restore the values of a random vector with given length.
Revised: RP only requires to restore the values of a random vector with given length.

Original: C m is also for the sparse positions to record the selected subnetwork.
Revised: C m is also for the sparse positions to record the selected subnetwork.

Original: 4.3.2 Compression Performance Validation
Revised: 4.3.2 Compression Performance Validation

Original:  We test the compreesion performance on CIFAR10 and CIFAR100 datasets using ConvMixer with6/8 depths, ResNet32, and ResNet56 backbones.
Revised: [Figure] We test the compression performance on CIFAR10 and CIFAR100 datasets using ConvMixer with 6/8 depths, ResNet32, and ResNet56 backbones.

Original: The compression ratio is based on the storage size aswe discussed instead of the conventional pruning ratio.
Revised: The compression ratio is based on the storage size as we discussed instead of the conventional pruning ratio.

Original: Since we propose a new strategy to compressnetwork, we involve two sparse network training baselines in our experiments.
Revised: Since we propose a new strategy to compress network, we involve two sparse network training baselines in our experiments.

Original: Specifically, we traina sparse network from scratch by removing random weight and minimum magnitude weights.
Revised: Specifically, we train a sparse network from scratch by removing random weight and minimum magnitude weights.

Original: Forcompression ratio, we set four settings for baselines: 90%, 92%, 94%, and 96%.
Revised: For compression ratio, we set four settings for baselines: 90%, 92%, 94%, and 96%.

Original: We directly refer tosettings, MP , RP from Sec.
Revised: We directly refer to settings, MP , RP from Sec.

Original: 4.2 for our paradigm.
Revised: 4.2 for our paradigm.

Original: Their compression ratio is calculated using thesame measurement as baselines.
Revised: Their compression ratio is calculated using the same measurement as baselines.

Original: In Fig.
Revised: In Fig.

Original: 4, we show 4 pairs of comparison based on 2 backbones (ResNet32/ResNet56) and 2 datasets(CIFAR10/CIFAR100).
Revised: 4, we show 4 pairs of comparison based on 2 backbones (ResNet32/ResNet56) anddatasets (CIFAR10/CIFAR100).

Original: Each pair includes dense model, 2 baselines with 4 compression ratios, andour results in 3 ratios.
Revised: Each pair includes dense model, 2 baselines with 4 compression ratios, and our results in 3 ratios.

Original: Two baselines are sparse network training by pruning random weights andminimum magnitude weights, respectively.
Revised: Two baselines are sparse network training by pruning random weights and minimum magnitude weights, respectively.

Original: For convenience, we do not follow exactly the samecompression ratios as baseline but directly use settings from Sec.4.2.
Revised: For convenience, we do not follow exactly the same compression ratios as baseline but directly use settings from Sec.4.2.

Original: For ResNet, we use MP , RP 1e-1 , and RP 1e-2 .
Revised: For ResNet, we use MP , RP 1e-1 , and RP 1e-2 .

Original: Their corresponding compression ratios are computed as shown in figures.
Revised: Their corresponding compression ratios are computed as shown in figures.

Original: Experiments based on different networks and datasets show the similar conclusions summarizedas: 1) our method outperforms the baselines by a significant margin, with even higher compressionratio; 2); Compared with conventional sparse network training where compressed model performancedecreases obviously along with increasing compression ratio, our method is relatively robust to thecompressed model size; 3) If we compare cross different models, we find compressed small modelby our method even performs better than baselines using larger model; 4) Network scale affects the compression performance, compared with ResNet32, ResNet56 basically contains more parametersand performance drop between compressed network with its dense counterpart is relatively small.[Figure]
Revised: Experiments based on different networks and datasets show the similar conclusions summarized as: 1) our method outperforms the baselines by a significant margin, with even higher compression ratio; 2); Compared with conventional sparse network training where compressed model performance decreases obviously along with increasing compression ratio, our method is relatively robust to the compressed model size; 3) If we compare cross different models, we find compressed small model by our method even performs better than baselines using larger model; 4) Network scale affects the compression performance, compared with ResNet32, ResNet56 basically contains more parameters and performance drop between compressed network with its dense counterpart is relatively small.

Original: In Fig.
Revised: In Fig.

Original: 5, we show compression performance on CIFAR10 dataset using ConvMixer with depth 6and 8.
Revised: 5, we show compression performance on CIFAR10 dataset using ConvMixer with depth 6 and 8.

Original: The settings are basically similar to Fig.
Revised: The settings are basically similar to Fig.

Original: We can also draw the similar conclusions: 1) Our method outperforms the baselines on ConvMixer with different depths; 2) Our methodcompresses network into lower size but main
Revised: We can also draw the similar conclusions: 1) Our method outperforms the baselines on ConvMixer with different depths; 2) Our method compresses network into lower size but maintains higher performance.

Original: In our Sec. 4, our experiments can be separatedinto two parts.
Revised: As a summary, in Sec. 4, our experiments can be separated into two parts.

Original: Firstly,  we investigate the representative potential of random weights which areused to fill in the complete network structure using different proposed strategies.
Revised: Firstly, to validate our PEMN, we investigate the representative potential of random weights which are used to fill in the complete network structure using different proposed strategies.

Original: Secondly, thepromising conclusion (Sec. 4.2) for this investigation naturally leads to the newly proposednetwork compression paradigm.
Revised: Secondly, the promising conclusion (Sec. 4.2) for this investigation naturally leads to the newly proposed network compression paradigm.

Original: Different fromconventional fashion restoring sparse weights,we instead restore the fixed random weights and different masks.
Revised: Different from conventional fashion restoring sparse weights, we instead restore the fixed random weights and different masks.

Original: Empirically, we validate theeffectiveness of our new paradigm for networkcompression.
Revised: Empirically, we validate the effectiveness of our new paradigm for network compression.

Original: Our experiments involve diversenetwork architectures to demonstrate the proposed paradigm can be generalized into differentnetwork designs.
Revised: Our experiments involve diverse network architectures to demonstrate the proposed paradigm can be generalized into different network designs.

Original:  Discussion and Conclusion
Revised:  Discussion and Conclusion

Original: [Figure]

Original: Discussion We summarize our intuitive logic and potential research direction in the future.
Revised: Discussion We summarize our intuitive logic and potential research direction in the future.

Original: Ourfoundamental insight is motivated by Supermasks [26] and Popup [20] showing random networkencodes informative pattern by selecting subnetworks.
Revised: Our fundamental insight is motivated by Supermasks [44] and Popup [31] showing random network encodes informative pattern by selecting subnetworks.

Original: They inspire us to understand neural network in a decoupled perspective: the informative output is delivered by certain weight-structurecombination.
Revised: They inspire us to understand neural network in a decoupled perspective: the informative output is delivered by certain weight-structure combination.

Original: Even if weights are fixed, the flexibility of learnable masks still provides promisingcapacity to represent diverse semantic information.
Revised: Even if weights are fixed, the flexibility of learnable masks still provides promising capacity to represent diverse semantic information.

Original: We are the first to fully explore the potentialof random weights, and practically, a new network compression paradigm is naturally established.
Revised: We are the first to fully explore the representative potential of random weights, and practically, a new network compression paradigm is naturally established.

Original: We further discuss some research directions in the future following this study.
Revised: We further discuss some research directions in the future following this study.

Original: Firstly, comparedwith conventional approaches need to record learned weights, our paradigm records random weightswhich can be pre-accessed, can this property be used for improve the model security?
Revised: Firstly, compared with conventional approaches need to record learned weights, our paradigm records random weights which can be pre-accessed, can this property be used for improve the model security?

Original: Moreover,leveraging on the property that repetitive random weights existing in networks for our strategies, is itpossible to specifically design hardware deployment configurations to achieve further compression oracceleration?
Revised: In addition, leveraging on the property that repetitive random weights existing in networks for our strategies, is it possible to specifically design hardware deployment configurations to achieve further compression or acceleration? Moreover, our PEMN is based on random initialized weights but cannot be directly deployed on pretrained models, which are more powerful these days. How to further improve our strategy on large-scale pretrained models is another interesting point to explore.

Original: We leave these topics in our future work.
Revised: We leave these topics in our future work.

Original: Conclusion We first explore the maximum representative potential of a set of fixed random weights,which leverages different learned masks to obtain different feature mappings.
Revised: Conclusion We first explore the maximum representative potential of a set of fixed random weights, which leverages different learned masks to obtain different feature mappings.

Original:  Specifically, wenaturally propose three strategies, one-layer , max-layer padding (MP) , and random vector padding(RP) , to fill in a complete network with given random weights.
Revised: Correspondingly, we introduce our proposed Parameter-Efficient Masking Networks (PEMN). Specifically, we naturally propose three strategies, one-layer , max-layer padding (MP) , and random vector padding (RP) , to fill in a complete network with given random weights.

Original: We find that alarge neural networkwith even limited unique parameters can achieve promising performance.
Revised: We find that a neural network with even limited unique parameters can achieve promising performance.

Original: It shows that parameterswith fewer unique values have great representative potential achieved by learning different masks.
Revised: It shows that parameters with fewer unique values have great representative potential achieved by learning different masks.

Original: In this way, we can represent a complete network by combining a set of random weights withdifferent masks.
Revised: Therefore, we can represent a complete network by combining a set of random weights with different masks.

Original: Inspired by this observation, we propose a novel network compression paradigm.
Revised: Inspired by this observation, we propose a novel network compression paradigm.

Original: Compared with traditional approaches, our paradigm canbe restore and transfer a network by onlykeeping a random vector with masks, instead of recording sparse weights for all layers.
Revised: Compared with traditional approaches, our paradigm can restore and transfer a network by only keeping a random vector with masks, instead of recording sparse weights for the whole network.

Original: Since thecost of restoring a mask is significantly lower than weight, we can achieve admirable compressionperformance.
Revised: Since the cost of restoring a mask is significantly lower than weight, we can achieve admirable compression performance.

Original: We conduct comprehensive experiments based on several popular and classic networkarchitectures to explore the random weights potential  and test the compression performance ofour new paradigm.
Revised: We conduct comprehensive experiments based on several popular network architectures to explore the random weights potential for PEMN and test the compression performance of our new paradigm.

Original: We expect our work can inspire further researches for both exploring networkrepresentative potential and network compression.
Revised: We expect our work can inspire further researches for both exploring network representative potential and network compression

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] • Did you include the license to the code and datasets?
Revised: [Yes] • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See supplementary material.(c)
Revised: [Yes] See supplementary material.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Seesupplementary material.
Revised: [Yes] See supplementary material.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A]3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [No](b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See supplementary material.
Revised: [Yes] See supplementary material.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See supplementary material.(d)
Revised: [Yes] See supplementary material.

Original:  Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See supplementary material.
Revised: [Yes] See supplementary material.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [No] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [No]
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Appendix 
Revised: Appendix A. Implementation Details

Revised: For all the backbones used in our experiments, we follow their default training settings. For ConvMixer [33], we use AdamW [27] optimizer with triangular learning rate scheduler. We set the maximum learning rate as 0.05 with weight decay as 0.005.

Original: Optionally include extra information (complete proofs, additional experiments and plots) in theappendix.
Revised: We set batch size as 512 and the number of total epochs is 100. We use different configurations for hidden dimension (256/512) and depth (6/8) in our experiments section. For ViT [9], we use Adam [20] optimizer with cosine learning rate scheduler. We set the maximum learning rate as 0.0001.

Revised: We set batch size as 256 and the number of total epochs as 200.

Revised: We use different configurations for hidden dimension (256/512) and depth (6/8) in our experiments section.

Revised: For ResNet [18], we follow the implementation configurations in Popup [31].

Revised: Specifically, we use SGD optimizer with maximum learning rate 0.1 and cosine scheduler.

Revised: The weight decay and momentum are set as 0.0005 and 0.9.

Revised: We train 100 epochs with 256 batch size.

Revised: For the sparse selection to pick the subnetworks, we follow the optimal choice in

Revised: Popup and set it as 0.5 for our experiments.

Revised: And we use kaiming uniform and kaiming normal [17] to initialize the scores and random weights, respectively.

Revised: B. Limitations and Potential Negative Societal Impacts

Revised: Firstly, our study focuses on exploring the random weight potential and representing a neural network with small storage cost.

Revised: However, it cannot further help for accelerating the training and inference.

Revised: This is a good point for us to further explore.

Revised: Secondly, our proposed network compression strategy leverages on the representative potential of random weights with different masks, which benefits to reducing storage cost.

Revised: However, following this strategy, it is hard to take advantages of powerful pretrained model these days.

Revised: How to tailor our insight to large-scale pretrained model is another good point to explore.

Revised: To the best of our knowledge, our study has no potential negative societal impacts.

Revised: C. More Experimental Evaluations

Revised: We further add experiments for CIFAR100 [21] and Tiny-ImageNet [23] datasets using ConvMixer as backbone to test our new network compression paradigm.

Revised: The results are shown in Fig.

Revised: We use 8 and 6 as depth number for CIFAR100 and Tiny-ImageNet datasets, respectively and other settings follow the same mentioned above.

Revised: The results in figure show that our compression strategy outperforms the baseline methods and validate the effectiveness of our propose network compression paradigm.

Revised: We also conduct experiments on large-scale ImageNet [6] dataset using ResNet50 [18] as backbone.

Revised: For simplicity, we customize two ImageNet subset for convenient evaluation.

Revised: We sample 100/200 classes from original ImageNet to construct our subsets.

Revised: We compare our compression strategy with magnitude pruning baseline and results are shown in Fig.

Revised: The results of our method are promising and demonstrate its effectiveness on challenging ImageNet dataset.

Revised: D. The Results of Repeated Experiments

Revised: We further supplement the repeated experimental results.

Revised: We supplement the main results from Fig.and Fig.

Revised: We repeated each experiment three times and report mean and std.

Revised: In Tab. 1, Tab. 2,

Revised: Tab. 3, Tab. 4, we provide the repeated results for Fig. 3.

Original: This section will often be part of  the supplemental material.
Revised: According to these results, we make several conclusions: 1) Our experimental performance are generally consistent across different datasets and different settings. The supplemented results follow the accuracy patterns and support the conclusions provided in our draft; 2) ConvMixer is more stable than ViT backbone across different settings; 3) Overall, along with the decreasing number of unique values in the network (from the left column to the right column of tables), the performance variations increase correspondingly. The limited unique weight values decreases the stability of the network.

Revised: In Tab. 5, Tab. 6, we provide the repeated results for Fig. 4.

Revised: The first four columns show the compression baselines (the first / second items represent random and magnitude pruning).

Revised: Based on the results shown above, we find our strategies generally outperform the model compression baselines and these results support our conclusion in the draft.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: ConvMixer on CIFAR100 dataset.

Revised: 97 Size

Revised: [Figure]

Revised: [Figure]

Revised: The straight lines on the top are performance of dense model with regular training.

Revised: Lines with different symbol shapes denote different settings.

Revised: Our three points are based on RP 1e-1 , RP 1e2 , and RP 1e-3 , respectively.

Revised: This figure shows that our proposed paradigm achieves admirable compression performance compared with baselines.

Revised: We can still maintain the test accuracy in very high compression ratios.

Revised: [Figure]

Revised: , and RP 1e-3 , respectively.

Revised: This figure shows that our proposed compression strategy achieves promising performance on challenging ImageNet dataset compared with baselines.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]