Original: Multi-Prize Lottery Ticket Hypothesis : F INDING A CCURATE B INARY N EURAL N ETWORKS BY P RUNING A R ANDOMLY W EIGHTED N ETWORK
Revised: Multi-Prize Lottery Ticket Hypothesis : F INDING A CCURATE B INARY N EURAL N ETWORKS BY P RUNING A R ANDOMLY W EIGHTED N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, Frankle & Carbin (2018) demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network.
Revised: Recently, Frankle & Carbin (2018) demonstrated that randomly-initialized dense networks contain subnetworks that once found can be trained to reach test accuracy comparable to the trained dense network.

Original: However, ﬁnding these high performing trainable subnetworks is expensive, requiring iterative process of training and pruning weights.
Revised: However, ﬁnding these high performing trainable subnetworks is expensive, requiring iterative process of training and pruning weights.

Original: In this paper, we propose (and prove) a stronger Multi-Prize
Revised: In this paper, we propose (and prove) a stronger Multi-Prize

Original: Lottery Ticket Hypothesis :
Revised: Lottery Ticket Hypothesis :

Original: A sufﬁciently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).
Revised: A sufﬁciently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).

Original: This provides a new paradigm for learning compact yet highly accurate binary neural networks by pruning and quantizing randomly weighted full precision neural networks.
Revised: This provides a new paradigm for learning compact yet highly accurate binary neural networks by pruning and quantizing randomly weighted full precision neural networks.

Original: These multi-prize tickets enjoy a number of desirable properties including drastically reduced memory size, faster test-time inference, and lower power consumption compared to their dense and full-precision counterparts.
Revised: These multi-prize tickets enjoy a number of desirable properties including drastically reduced memory size, faster test-time inference, and lower power consumption compared to their dense and full-precision counterparts.

Original: Furthermore, we propose an algorithm for ﬁnding multi-prize tickets and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets.
Revised: Furthermore, we propose an algorithm for ﬁnding multi-prize tickets and test it by performing a series of experiments on CIFAR-10 and ImageNet datasets.

Original: Empirical results indicate that as models grow deeper and wider,untrained multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to their signiﬁcantly larger and full-precision counterparts that have been weighttrained.
Revised: Empirical results indicate that as models grow deeper and wider, multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to their signiﬁcantly larger and full-precision counterparts that have been weight-trained.

Original: With minimal hyperparameter tuning, our binary weight multi-prize tickets outperform current state-of-the-art in binary neural networks.
Revised: With minimal hyperparameter tuning, our binary weight multi-prize tickets outperform current state-of-the-art in binary neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning (DL) has made a signiﬁcant breakthroughs in a wide range of applications (Goodfellow et al., 2016).
Revised: Deep learning (DL) has made a signiﬁcant breakthroughs in a wide range of applications (Goodfellow et al., 2016).

Original: These performance improvements can be attributed to the signiﬁcant growth in the model size and the availability of massive computational resources to train such models.
Revised: These performance improvements can be attributed to the signiﬁcant growth in the model size and the availability of massive computational resources to train such models.

Original: Therefore, these gains have come at the cost of large memory consumption, high inference time, and increased power consumption.
Revised: Therefore, these gains have come at the cost of large memory consumption, high inference time, and increased power consumption.

Original: This not only limits the potential applications where DL can make an impact but also have some serious consequences, such as, (a) generating huge carbon footprint, and (b) creating roadblocks to the democratization of AI.
Revised: This not only limits the potential applications where DL can make an impact but also have some serious consequences, such as, (a) generating huge carbon footprint, and (b) creating roadblocks to the democratization of AI.

Original: Note that signiﬁcant parameter redundancy and a large number of ﬂoating-point operations are key factors incurring the these costs.
Revised: Note that signiﬁcant parameter redundancy and a large number of ﬂoating-point operations are key factors incurring the these costs.

Original: Thus, for discarding the redundancy from DNNs, one can either (a) Prune: remove non-essential connections from an existing dense network, or (b) Quantize: constrain the full-precision (FP) weight and activation values to a set of discrete values which allows them to be represented using fewer bits.
Revised: Thus, for discarding the redundancy from DNNs, one can either (a) Prune: remove non-essential connections from an existing dense network, or (b) Quantize: constrain the full-precision (FP) weight and activation values to a set of discrete values which allows them to be represented using fewer bits.

Original: Further, one can exploit the complementary nature of pruning and quantization to combine their strengths.
Revised: Further, one can exploit the complementary nature of pruning and quantization to combine their strengths.

Original: Although pruning and quantization 1 are typical approaches used for compressing DNNs (Neill, 2020), it is not clear under what conditions and to what extent compression can be achieved without sacriﬁcing the accuracy.
Revised: Although pruning and quantization 1 are typical approaches used for compressing DNNs (Neill, 2020), it is not clear under what conditions and to what extent compression can be achieved without sacriﬁcing the accuracy.

Original: The most extreme form of quanitization is binarization, where weights and/or activations can only have two possible values, namely − 1(0) or +1 (the interest of this paper).
Revised: The most extreme form of quanitization is binarization, where weights and/or activations can only have two possible values, namely − 1(0) or +1 (the interest of this paper).

Original: In addition to saving memory, binarization results in more power efﬁcient networks with signiﬁcant computation acceleration since expensive multiply-accumulate operations (MACs) can be replaced by cheap XNOR and bit-counting operations (Qin et al., 2020a).
Revised: In addition to saving memory, binarization results in more power efﬁcient networks with signiﬁcant computation acceleration since expensive multiply-accumulate operations (MACs) can be replaced by cheap XNOR and bit-counting operations (Qin et al., 2020a).

Original: In light of these beneﬁts, it is of interest to question if conditions exists such that a binarized DNN can be pruned to achieve accuracy comparable to the dense FP DNN.
Revised: In light of these beneﬁts, it is of interest to question if conditions exists such that a binarized DNN can be pruned to achieve accuracy comparable to the dense FP DNN.

Original: More importantly, even if these favourable conditions are met then how do we ﬁnd these extremely compressed (or compact) and highly accurate subnetworks?
Revised: More importantly, even if these favourable conditions are met then how do we ﬁnd these extremely compressed (or compact) and highly accurate subnetworks?

Original: Traditional pruning schemes have shown that a pretrained DNN can be pruned without a signiﬁcant loss in the performance.
Revised: Traditional pruning schemes have shown that a pretrained DNN can be pruned without a signiﬁcant loss in the performance.

Original: Recently, (Frankle & Carbin, 2018) made a breakthrough by showing that dense network contain sparse subnetworks that can match the performance of the original network when trained from scratch with weights being reset to their initialization ( Lottery Ticket Hypothesis ).
Revised: Recently, (Frankle & Carbin, 2018) made a breakthrough by showing that dense network contain sparse subnetworks that can match the performance of the original network when trained from scratch with weights being reset to their initialization ( Lottery Ticket Hypothesis ).

Original: Although the original approach to ﬁnd these subnetworks still required training the dense network, some efforts (Wang et al., 2020b; You et al., 2019; Wang et al., 2020a) have been carried out to overcome this limitation.
Revised: Although the original approach to ﬁnd these subnetworks still required training the dense network, some efforts (Wang et al., 2020b; You et al., 2019; Wang et al., 2020a) have been carried out to overcome this limitation.

Original: Recently a more intriguing phenomenon has been reported – a dense network with random initialization contains subnetworks that achieve high accuracy, without any further training (Zhou et al., 2019; Ramanujan et al., 2020; Malach et al., 2020; Orseau et al., 2020).
Revised: Recently a more intriguing phenomenon has been reported – a dense network with random initialization contains subnetworks that achieve high accuracy, without any further training (Zhou et al., 2019; Ramanujan et al., 2020; Malach et al., 2020; Orseau et al., 2020).

Original: These trends highlight good progress being made towards efﬁciently and accurately pruning DNNs.
Revised: These trends highlight good progress being made towards efﬁciently and accurately pruning DNNs.

Original: In contrast to these positive developments for pruning, results on binarizing DNNs have been mostly negative.
Revised: In contrast to these positive developments for pruning, results on binarizing DNNs have been mostly negative.

Original: To the best of our knowledge, post-training schemes have not been successful in binarizing pretrained models without retraining.
Revised: To the best of our knowledge, post-training schemes have not been successful in binarizing pretrained models without retraining.

Original: Even with training binary neural networks (BNNs) from scratch (though inefﬁcient), the community has not been able to make BNNs achieve comparable results to their full precision counterparts.
Revised: Even with training binary neural networks (BNNs) from scratch (though inefﬁcient), the community has not been able to make BNNs achieve comparable results to their full precision counterparts.

Original: The main reason being that network structures and weight optimization techniques are predominantly developed for full precision DNNs and may not be suitable for training BNNs.
Revised: The main reason being that network structures and weight optimization techniques are predominantly developed for full precision DNNs and may not be suitable for training BNNs.

Original: Thus, closing the gap in accuracy between the full precision and the binarized version may require a paradigm shift.
Revised: Thus, closing the gap in accuracy between the full precision and the binarized version may require a paradigm shift.

Original: Furthermore, this also makes one wonder if efﬁciently and accurately binarizing DNNs similar to the recent trends in pruning is ever feasible.
Revised: Furthermore, this also makes one wonder if efﬁciently and accurately binarizing DNNs similar to the recent trends in pruning is ever feasible.

Original: In this paper, we show that a randomly initialized dense network contains extremely sparse binary subnetworks that without any training (i.e., efﬁcient ) have comparable performance to their trained dense and full-precision counterparts (i.e., accurate ).
Revised: In this paper, we show that a randomly initialized dense network contains extremely sparse binary subnetworks that without any training (i.e., efﬁcient ) have comparable performance to their trained dense and full-precision counterparts (i.e., accurate ).

Original: Based on this, we state our hypothesis:
Revised: Based on this, we state our hypothesis:

Original: Multi-Prize Lottery Ticket Hypothesis.
Revised: Multi-Prize Lottery Ticket Hypothesis.

Original: A sufﬁciently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).
Revised: A sufﬁciently over-parameterized neural network with random weights contains several subnetworks (winning tickets) that (a) have comparable accuracy to a dense target network with learned weights (prize 1), (b) do not require any further training to achieve prize 1 (prize 2), and (c) is robust to extreme forms of quantization (i.e., binary weights and/or activation) (prize 3).

Original: Contributions.
Revised: Contributions.

Original: First, we propose the multi-prize lottery ticket hypothesis as a new perspective on ﬁnding neural networks with drastically reduced memory size, much faster test-time inference and lower power consumption compared to their dense and full-precision counterparts.
Revised: First, we propose the multi-prize lottery ticket hypothesis as a new perspective on ﬁnding neural networks with drastically reduced memory size, much faster test-time inference and lower power consumption compared to their dense and full-precision counterparts.

Original: Next, we provide theoretical evidence of the existence of highly accurate sparse binary subnetworks within a randomly weighted DNN (i.e., proving the multi-prize lottery ticket hypothesis).
Revised: Next, we provide theoretical evidence of the existence of highly accurate sparse binary subnetworks within a randomly weighted DNN (i.e., proving the multi-prize lottery ticket hypothesis).

Original: Speciﬁcally, we mathematically prove that we can ﬁnd an ε -approximation of a fully-connected ReLU DNN with width n and depth (cid:96) using a sparse binary-weight DNN of sufﬁcient width.
Revised: Speciﬁcally, we mathematically prove that we can ﬁnd an ε -approximation of a fully-connected ReLU DNN with width n and depth (cid:96) using a sparse binary-weight DNN of sufﬁcient width.

Original: Our proof indicates that this can be accomplished by pruning and binarizing the weights of a randomly weighted neural network that is a factor O ( n 3 / 2 (cid:96)/ε ) wider and 2 (cid:96) deeper.
Revised: Our proof indicates that this can be accomplished by pruning and binarizing the weights of a randomly weighted neural network that is a factor O ( n 3 / 2 (cid:96)/ε ) wider and 2 (cid:96) deeper.

Original: To the best of our knowledge, this is the ﬁrst theoretical work proving the existence of highly accurateyet untrained 2 binary subnetworks within a sufﬁciently overparameterized  neural network.
Revised: To the best of our knowledge, this is the ﬁrst theoretical work proving the existence of highly accurate binary subnetworks within a sufﬁciently overparameterized randomly initialized neural network.

Original: Finally, we provide biprop ( bi narize- pr une op timizer) in Algorithm 1 to identify multi-prize tickets within randomly weighted DNNs and empirically test our hypothesis.
Revised: Finally, we provide biprop ( bi narize- pr une op timizer) in Algorithm 1 to identify multi-prize tickets within randomly weighted DNNs and empirically test our hypothesis.

Original: This provides a completely new way to learn sparse BNNs without relying on weight-optimization.
Revised: This provides a completely new way to learn sparse BNNs without relying on weight-optimization.

Original: Results.
Revised: Results.

Original: We explore two variants of multi-prize tickets – one with binary weights (MPT-1/32) and other with binary weights and activation (MPT-1/1) where x/y denotes x and y bits to represent weights and activation, respectively.
Revised: We explore two variants of multi-prize tickets – one with binary weights (MPT-1/32) and other with binary weights and activation (MPT-1/1) where x/y denotes x and y bits to represent weights and activation, respectively.

Original: MPTs we ﬁnd have 60 − 80% fewer parameters than the original network.
Revised: MPTs we ﬁnd have 60 − 80% fewer parameters than the original network.

Original: In addition, compared to their full-precision (i.e., 32/32) counterparts, MPT–1/32 can provide ∼ 32 × memory saving and ∼ 2 × computation saving, and MPT–1/1 can provide ∼ 32 × memory saving and ∼ 58 × computation saving due to their binary nature (Rastegari et al., 2016).
Revised: In addition, compared to their full-precision (i.e., 32/32) counterparts, MPT–1/32 can provide ∼ 32 × memory saving and ∼ 2 × computation saving, and MPT–1/1 can provide ∼ 32 × memory saving and ∼ 58 × computation saving due to their binary nature (Rastegari et al., 2016).

Original: We perform a series of experiments on on small and large scale datasets for image recognition, namely CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).
Revised: We perform a series of experiments on on small and large scale datasets for image recognition, namely CIFAR-10 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009).

Original: On CIFAR-10, we test the performance of multi-prize tickets against the trend of making the model deeper and wider.
Revised: On CIFAR-10, we test the performance of multi-prize tickets against the trend of making the model deeper and wider.

Original: We found that as models grow deeper and wider, both variants of multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to the dense and full precision original network with learned weights.
Revised: We found that as models grow deeper and wider, both variants of multi-prize tickets start to reach similar (and sometimes even higher) test accuracy compared to the dense and full precision original network with learned weights.

Original: In other words, the performance of multi-prize tickets improves with the amount of redundancy in the original network.
Revised: In other words, the performance of multi-prize tickets improves with the amount of redundancy in the original network.

Original: Note that our positive results are in sharp contrast with the existing belief of limited expressive power of BNNs.
Revised: Note that our positive results are in sharp contrast with the existing belief of limited expressive power of BNNs.

Original: We also carry out experiments with state-of-the-art (SOTA) architectures on CIFAR-10 and ImageNet datasets with an aim to investigate their redundancy.
Revised: We also carry out experiments with state-of-the-art (SOTA) architectures on CIFAR-10 and ImageNet datasets with an aim to investigate their redundancy.

Original: We ﬁnd that within most randomly weighted SOTA DNNs reside extremely compact (i.e., sparse and binary) subnetworks which are smaller than, but match the performance of trained target dense and full precision networks.
Revised: We ﬁnd that within most randomly weighted SOTA DNNs reside extremely compact (i.e., sparse and binary) subnetworks which are smaller than, but match the performance of trained target dense and full precision networks.

Original: Furthermore, with minimal hyperparameter tuning, our multi-prize tickets without any training achieve Top-1 accuracy comparable to (or higher than) SOTA BNNs.
Revised: Furthermore, with minimal hyperparameter tuning, our multi-prize tickets without any training achieve Top-1 accuracy comparable to (or higher than) SOTA BNNs.

Original: Speciﬁcally, our winning tickets MPT-1/32 set new state-of-the-art (in Top1 measure) w.r.t.
Revised: Speciﬁcally, our winning tickets MPT-1/32 set new state-of-the-art (in Top1 measure) w.r.t.

Original: trained BNN-1/32 – (a) on CIFAR-10 MPT-1/32 achieves 94.15% compared to the previous best 92.3% achieved by ProxQuant (Bai et al., 2018), and (b) on ImageNet MPT-1/ achieves 73.3% compared to the previous best 72.8% achieved using Quantization-Networks (Yang et al., 2019).
Revised: trained BNN-1/32 – (a) on CIFAR-10 MPT-1/32 achieves 94.15% compared to the previous best 92.3% achieved by ProxQuant (Bai et al., 2018), and (b) on ImageNet MPT-1/ achieves 73.3% compared to the previous best 72.8% achieved using Quantization-Networks (Yang et al., 2019).

Original: Furthermore, on both CIFAR-10 and ImageNet, MPT-1/32 subnetworks outperform their signiﬁcantly larger and full-precision counterparts that have been weight-trained.
Revised: Furthermore, on both CIFAR-10 and ImageNet, MPT-1/32 subnetworks outperform their signiﬁcantly larger and full-precision counterparts that have been weight-trained.

Original: MPT-1/1 performs comparable to SOTA on CIFAR-10, however, struggles on ImageNet.
Revised: MPT-1/1 performs comparable to SOTA on CIFAR-10, however, struggles on ImageNet.

Original: We discuss reasoning behind this and provide worthwhile future directions to further improve the performance.
Revised: We discuss reasoning behind this and provide worthwhile future directions to further improve the performance.

Original:  M ULTI -P RIZE L OTTERY T ICKETS : T HEORY AND A LGORITHMS
Revised:  M ULTI -P RIZE L OTTERY T ICKETS : T HEORY AND A LGORITHMS

Original: We ﬁrst prove the existence of MPTs in an overparameterized randomly weighted DNN.
Revised: We ﬁrst prove the existence of MPTs in an overparameterized randomly weighted DNN.

Original: For ease of presentation, we state an informal version of Theorem 2 which can be found in Appendix B.
Revised: For ease of presentation, we state an informal version of Theorem 2 which can be found in Appendix B.

Original: We then explore two variants of tickets (MPT-1/32 and MPT-1/1) and provide an algorithm to ﬁnd them.
Revised: We then explore two variants of tickets (MPT-1/32 and MPT-1/1) and provide an algorithm to ﬁnd them.

Original: 2. P ROVING THE M ULTI -P RIZE L OTTERY T ICKETS H YPOTHESIS
Revised: 2. P ROVING THE M ULTI -P RIZE L OTTERY T ICKETS H YPOTHESIS

Original: In this section we seek to answer the following question: What is the required amount of overparameterization such that a randomly weighted neural network can be compressed to a sparse binary subnetwork that approximates a dense trained target network?
Revised: In this section we seek to answer the following question: What is the required amount of overparameterization such that a randomly weighted neural network can be compressed to a sparse binary subnetwork that approximates a dense trained target network?

Original: Theorem 1.
Revised: Theorem 1.

Original: (Informal Statement of Theorem 2) Let ε, δ > 0 .
Revised: (Informal Statement of Theorem 2) Let ε, δ > 0 .

Original: For every fully-connected (FC) target network with ReLU activations of depth (cid:96) and width n with bounded weights, a random binary FC network with ReLU activations of depth 2 (cid:96) and width O (cid:0) ( (cid:96)n 3 / 2 /ε )
Revised: For every fully-connected (FC) target network with ReLU activations of depth (cid:96) and width n with bounded weights, a random binary FC network with ReLU activations of depth 2 (cid:96) and width O (cid:0) ( (cid:96)n 3 / 2 /ε )

Original: + (cid:96)n log( (cid:96)n/δ ) (cid:1) contains with probability (1 − δ ) a binary subnetwork that approximates the target network with error at most ε .
Revised: + (cid:96)n log( (cid:96)n/δ ) (cid:1) contains with probability (1 − δ ) a binary subnetwork that approximates the target network with error at most ε .

Original: Sketch of Proof.
Revised: Sketch of Proof.

Original: Consider a FC ReLU network F ( x )
Revised: Consider a FC ReLU network F ( x )

Original: =
Revised: =

Original: W ( (cid:96) ) σ ( W ( (cid:96) − 1) · · · σ ( W (1) x )) , where σ ( x ) = max { 0 , x } , x ∈ R d , W ( i
Revised: W ( (cid:96) ) σ ( W ( (cid:96) − 1) · · · σ ( W (1) x )) , where σ ( x ) = max { 0 , x } , x ∈ R d , W ( i

Original: ) ∈
Revised: ) ∈

Original: R k
Revised: R k

Original: i × k
Revised: i × k

Original: i − 1 , k 

Revised: i − 1 , k

Original: = d , and i ∈ [ (cid:96) ] .
Revised: = d , and i ∈ [ (cid:96) ] .

Original: Additionally, consider a
Revised: Additionally, consider a

Original: FC network with binary weights given by G ( x ) =
Revised: FC network with binary weights given by G ( x ) =

Original: B ( (cid:96) (cid:48) )
Revised: B ( (cid:96) (cid:48) )

Original: σ ( B ( (cid:96) (cid:48) − 1) · · · σ ( B (1) x )) , where B ( i ) ∈ {− 1 , +1 } k (cid:48)
Revised: σ ( B ( (cid:96) (cid:48) − 1) · · · σ ( B (1) x )) , where B ( i ) ∈ {− 1 , +1 } k (cid:48)

Original: i × k (cid:48)
Revised: i × k (cid:48)

Original: i − 1 , k (cid:48) 

Revised: i − 1 , k (cid:48)

Original: = d , and i ∈ [ (cid:96) (cid:48) ] .
Revised: = d , and i ∈ [ (cid:96) (cid:48) ] .

Original: Our goal is to determine a lower bound on the depth, (cid:96) (cid:48) , and the widths, { k  i } (cid:96) (cid:48)
Revised: Our goal is to determine a lower bound on the depth, (cid:96) (cid:48) , and the widths, { k (cid:48) i } (cid:96) (cid:48)

Original: i =0 , such that with probability (1 − δ ) the network G ( x ) contains a subnetwork ˜ G ( x ) satisfying (cid:107)
Revised: i =1 , such that with probability (1 − δ ) the network G ( x ) contains a subnetwork ˜ G ( x ) satisfying (cid:107)

Original: ˜ G ( x ) − F ( x ) (cid:107) ≤ ε , for any ε > 0 and δ ∈ (0 , 1) .
Revised: ˜ G ( x ) − F ( x ) (cid:107) ≤ ε , for any ε > 0 and δ ∈ (0 , 1) .

Original: We ﬁrst establish lower bounds on the width of a network of the form g ( x ) =
Revised: We ﬁrst establish lower bounds on the width of a network of the form g ( x ) =

Original: B (2) σ ( B (1) x ) such that with probability (1 − δ (cid:48) )
Revised: B (2) σ ( B (1) x ) such that with probability (1 − δ (cid:48) )

Original: there exists a subnetwork ˜ g ( x ) of g ( x ) s.t. (cid:107)
Revised: there exists a subnetwork ˜ g ( x ) of g ( x ) s.t. (cid:107)

Original: ˜ g
Revised: ˜ g

Original: ( x ) − σ ( W x ) (cid:107) ≤ ε (cid:48) , for any ε (cid:48) > 0 and δ (cid:48) ∈ (0 , 1) .
Revised: ( x ) − σ ( W x ) (cid:107) ≤ ε (cid:48) , for any ε (cid:48) > 0 and δ (cid:48) ∈ (0 , 1) .

Original: This process is carried out in detail in Lemmas 1, 2, and 3 in Appendix B.
Revised: This process is carried out in detail in Lemmas 1, 2, and 3 in Appendix B.

Original: We have now approximated a single layer FC real-valued network using a subnetwork of a two-layer FC binary network.
Revised: We have now approximated a single layer FC real-valued network using a subnetwork of a two-layer FC binary network.

Original: Hence, we can take (cid:96) (cid:48) =
Revised: Hence, we can take (cid:96) (cid:48) =

Original: 2 (cid:96) and Lemma 3 provides lower bounds on the width of each intermediate layer such that with probability (1 − δ ) there exists a subnetwork ˜ G ( x ) of G ( x ) satisfying (cid:107) ˜ G ( x ) − F ( x ) (cid:107) ≤ ε .
Revised: 2 (cid:96) and Lemma 3 provides lower bounds on the width of each intermediate layer such that with probability (1 − δ ) there exists a subnetwork ˜ G ( x ) of G ( x ) satisfying (cid:107) ˜ G ( x ) − F ( x ) (cid:107) ≤ ε .

Original: This is accomplished in Theorem 2 in Appendix B.
Revised: This is accomplished in Theorem 2 in Appendix B.

Original: To the best of our knowledge this is the ﬁrst theoretical result proving that a sparse binary-weight
Revised: To the best of our knowledge this is the ﬁrst theoretical result proving that a sparse binary-weight

Original: DNN that can approximate a real-valued target DNN.
Revised: DNN that can approximate a real-valued target DNN.

Original: As it has been established that real-valued
Revised: As it has been established that real-valued

Original: DNNs are universal approximators (Scarselli & Tsoi, 1998), our result carries the implication that sparse binary-weight DNNs are also universal approximators.
Revised: DNNs are universal approximators (Scarselli & Tsoi, 1998), our result carries the implication that sparse binary-weight DNNs are also universal approximators.

Original: In relation to the ﬁrst result establishing the existence of real-valued subnetworks in a randomly weighted DNN approximating a realvalued target DNN (Malach et al., 2020), the lower bound on the width established in Theorem 2 is better than their lower bound of O (cid:0) (cid:96) 2 n 2 log( (cid:96)n/δ ) /ε
Revised: In relation to the ﬁrst result establishing the existence of real-valued subnetworks in a randomly weighted DNN approximating a realvalued target DNN (Malach et al., 2020), the lower bound on the width established in Theorem 2 is better than their lower bound of O (cid:0) (cid:96) 2 n 2 log( (cid:96)n/δ ) /ε

Original: 2 (cid:1) .
Revised: 2 (cid:1) .

Original: 2. F INDING M ULTI -P RIZE W INNING T ICKETS
Revised: F INDING M ULTI -P RIZE W INNING T ICKETS

Original: Given the existence of multi-prize winning tickets from Theorem 2, a natural question arises – How should we ﬁnd them?
Revised: Given the existence of multi-prize winning tickets from Theorem 2, a natural question arises – How should we ﬁnd them?

Original: In this section, we answer this question by introducing an algorithm for ﬁnding multi-prize tickets.
Revised: In this section, we answer this question by introducing an algorithm for ﬁnding multi-prize tickets.

Original: 3 Speciﬁcally, we explore two variants of multi-prize tickets in this paper – 1)
Revised: 2 Speciﬁcally, we explore two variants of multi-prize tickets in this paper – 1)

Original: MPT-1/32 where weights are quantized to 1-bit with activations being real valued (i.e., 32-bits) and 2) MPT-1/1 where both weights and activations are quantized to 1-bit.
Revised: MPT-1/32 where weights are quantized to 1-bit with activations being real valued (i.e., 32-bits) and 2) MPT-1/1 where both weights and activations are quantized to 1-bit.

Original: We ﬁrst outline a generic process for identifying MPTs along with some theoretical motivation for our approach.
Revised: We ﬁrst outline a generic process for identifying MPTs along with some theoretical motivation for our approach.

Original: Given a neural network g ( x ; W ) with weights W ∈ R m , we can mathematically express a subnetwork of g using a binary mask M ∈ { 0 , 1 } m as g ( x ; M (cid:12) W ) , where (cid:12) denotes the Hadamard product.
Revised: Given a neural network g ( x ; W ) with weights W ∈ R m , we can mathematically express a subnetwork of g using a binary mask M ∈ { 0 , 1 } m as g ( x ; M (cid:12) W ) , where (cid:12) denotes the Hadamard product.

Original: Hence, a binary subnetwork can be expressed as g ( x ; M (cid:12) B ) , where B ∈ {− 1 , +1 } m .
Revised: Hence, a binary subnetwork can be expressed as g ( x ; M (cid:12) B ) , where B ∈ {− 1 , +1 } m .

Original: Our detailed theoretical analysis of Lemma 1 in Appendix B indicates that rescaling the binary weights to {− α, α } using a gain term α ∈ R is necessary to achieve good performance of the resulting subnetwork.
Revised: Our detailed theoretical analysis of Lemma 1 in Appendix B indicates that rescaling the binary weights to {− α, α } using a gain term α ∈ R is necessary to achieve good performance of the resulting subnetwork.

Original: We note that the use of gain terms is common in binary neural networks (Qin et al., 2020a).
Revised: We note that the use of gain terms is common in binary neural networks (Qin et al., 2020a).

Original: Combining all this allows us to represent a binary subnetwork as g ( x ; α ( M (cid:12) B )) .
Revised: Combining all this allows us to represent a binary subnetwork as g ( x ; α ( M (cid:12) B )) .

Original: Now we focus on how to update M , B , and α .
Revised: Now we focus on how to update M , B , and α .

Original: Suppose f ( x ; W ∗ ) is a target network with optimized weights W ∗ that we wish to approximate.
Revised: Suppose f ( x ; W ∗ ) is a target network with optimized weights W ∗ that we wish to approximate.

Original: Assuming g ( x ; · ) is κ -Lipschitz continuous yields
Revised: Assuming g ( x ; · ) is κ -Lipschitz continuous yields

Original: [Equation]
Revised: [Equation]

Original: From (1) it follows that the error of the MPT  is bounded above by the error of the subnetwork of g with the original weights and the error from binarizing the current subnetwork.
Revised: Hence, the MPT error is bounded above by the error of the subnetwork of g with the original weights and the error from binarizing the current subnetwork.

Original: This informs our approach for identifying MPTs: 1) Update a pruning mask M that reduces thethe subnetwork error, and 2) apply binarization with a gain term that minimizes the binarization error.
Revised: This informs our approach for identifying MPTs: 1) Update a pruning mask M that reduces the subnetwork error (lines 7 – 9 in Algorithm 1), and 2) apply binarization with a gain term that minimizes the binarization error (lines 4 and 10).

Original: We ﬁrst discuss how to update M .
Revised: We ﬁrst discuss how to update M .

Revised: While we could search for M by minimizing the subnetwork error in (1), this would require the use of a pretrained target network (i.e., f ( x ; W ∗ ) ).

Original: Instead of minimizing the subnetwork error in (1), we  aim to minimize the training loss w.r.t.
Revised: To avoid requiring a target network in our method we instead aim to minimize the training loss w.r.t.

Original: M in the current binary subnetwork.
Revised: M in the current binary subnetwork.

Original: This allows our method to search for MPTs without requiring the use of a pretrained target network (i.e., f ( x ; W ∗ ) ).

Original: Directly optimizing over the pruning mask is a combinatorial problem.
Revised: Directly optimizing over the pruning mask is a combinatorial problem.

Original: Instead, to update the pruning mask efﬁciently, we optimize over a set of scores S ∈ R m corresponding to each randomly initialized weight in the network.
Revised: So to update the pruning mask efﬁciently we optimize over a set of scores S ∈ R m corresponding to each randomly initialized weight in the network.

Original: In this approach, each component of the randomly initialized weights is assigned a pruning score.
Revised: In this approach, each component of the randomly initialized weights is assigned a pruning score.

Original: The pruning scores are updated via backpropagation by computing the gradient of the loss function, L , with respect to the pruning scores.
Revised: The pruning scores are updated via backpropagation by computing the gradient of the loss function over minibatches with respect to the pruning scores.

Original: Then the magnitude of the scores in absolute value are used to identify the P percent of weights in each layer that are least important to Input : Neural network g bit activations Network widths {
Revised: Then the magnitude of the scores in absolute value are used to identify the P percent of weights in each layer that are least important to the success of the binary subnetwork. The components of

Original: Loss Training Randomly Initialize FP Parameters : Network weights { W ( j ) } (cid:96)j =1 ; Pruning scores { S ( j ) } (cid:96)j =1 .
Revised: [Table]

Original: Initialize Layerwise Pruning Masks : { M ( j ) } (cid:96)j =1 each to 1 . Initialize Binary Subnetwork Weights : { B ( j ) } (cid:96)j =1 ← { sign( W ( j ) ) } (cid:96)j =1 . Initialize Layerwise Gain Terms : { α ( j ) } (cid:96)j =1 ← {(cid:107) M
Revised: [Table]

Original: ( j ) (cid:12) W ( j ) (cid:107) 1 / (cid:107) M ( j ) (cid:107) 1 } (cid:96)j =1 . for k = 1 to N epochs do
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Update pruning scores at layer j {

Original: [Table]
Revised: [Table]

Original: i
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ← Sorting of indices { i }

Original: [Table]
Revised: [Table]

Original: .t.

Original: [Table]
Revised: [Table]

Original: j

Original: [Table]|
Revised: [Table]

Original: Index sort over values | S ( j ) |

Original: [Table]
Revised: [Table]

Original: Update pruning mask at layer j

Original: [Table]
Revised: [Table]

Revised: ) (cid:107) 1 / (cid:107) M

Original:  Update gain term at layer j  Output : Return Binarized Subnetwork g ( x ; { α ( j ) ( M ( j ) (cid:12) B ( j ) ) } (cid:96)j =1 ) .the success of the binary subnetwork.
Revised: ( j ) (cid:107) Update gain term at layer j 11: Output : Return Binarized Subnetwork g ( x ; { α ( j ) ( M ( j ) (cid:12) B ( j ) ) } (cid:96)j =1 ) .

Original: The components of the pruning mask corresponding to these indices are set to 0 and the remaining components are set to 1 .
Revised: the pruning mask corresponding to these indices are set to 0 and the remaining components are set to 1 .

Original: To avoid unintentionally pruning an entire layer of the network, we use a pruning mask for each layer that prunes P percent of the weights in that layer.
Revised: To avoid unintentionally pruning an entire layer of the network, we use a pruning mask for each layer that prunes P percent of the weights in that layer.

Original: The choice to use pruning scores to update the mask M was due to the fact that it is computationally efﬁcient.
Revised: The choice to use pruning scores to update the mask M was due to the fact that it is computationally efﬁcient.

Original: The use of pruning scores is a well-established optimization technique used in a range of applications (Joshi & Boyd, 2009; Ramanujan et al., 2020).
Revised: The use of pruning scores is a well-established optimization technique used in a range of applications (Joshi & Boyd, 2009; Ramanujan et al., 2020).

Original: We now consider how to update B and α .
Revised: We now consider how to update B and α .

Original: By keeping M ﬁxed, we can derive the following closed form expressions that minimize the binarization error in (1): B ∗ = sign( W ) and α ∗ = (cid:107) M (cid:12) W (cid:107) 1 / (cid:107) M (cid:107) 1 .
Revised: By keeping M ﬁxed, we can derive the following closed form expressions that minimize the binarization error in (1): B ∗ = sign( W ) and α ∗ = (cid:107) M (cid:12) W (cid:107) 1 / (cid:107) M (cid:107) 1 .

Original: These closed form expressions indicate that only the gain term needs to be recomputed after each update to M .
Revised: These closed form expressions indicate that only the gain term needs to be recomputed after each update to M .

Original: Hence, B = sign( W ) throughout our entire approach.
Revised: Hence, B = sign( W ) throughout our entire approach.

Original: We update a gain term for each layer of the subnetwork in our approach.
Revised: We update a gain term for each layer of the subnetwork in our approach based on the formula for α ∗ .

Original: More details on the derivation of B ∗ and α ∗ are provided in Appendix C.
Revised: More details on the derivation of B ∗ and α ∗ are provided in Appendix C.

Original: Pseudocode for our method biprop ( bi narize- pr une op timizer) is provided in Algorithm 1 and crossentropy loss is used in our experiments.
Revised: Pseudocode for our method biprop ( bi narize- pr une op timizer) is provided in Algorithm 1 and crossentropy loss is used in our experiments.

Original: Note that the process for identifying MPT-1/32 and MPT-1/ differs only in computation of the gradient.
Revised: Note that the process for identifying MPT-1/32 and MPT-1/ differs only in computation of the gradient.

Original: Next, we explain how these gradients can be computed.
Revised: Next, we explain how these gradients can be computed.

Original: 2.2. U PDATING P RUNING S CORES FOR B INARY -W EIGHT T ICKETS (MPT-1/32)
Revised: 2.2. U PDATING P RUNING S CORES FOR B INARY -W EIGHT T ICKETS (MPT-1/32)

Original: As an example, for a FC network where the state at each layer is deﬁned recursively by
Revised: As an example, for a FC network where the state at each layer is deﬁned recursively by

Original: [Equation] x and U ( j ) = α ( j ) ( B ( j ) (cid:12) M ( j ) )
Revised: U (1) = α (1) ( B (1) (cid:12) M (1) ) x and U ( j ) = α ( j ) ( B ( j ) (cid:12) M ( j ) )

Original: σ ( U ( j − 1) )
Revised: σ ( U ( j − 1) )

Original: we have ∂L ∂S ( j ) p,q = ∂L ∂U ( j ) q ∂U ( j ) q ∂M ( j ) p,q ∂M ( j ) p,q ∂S ( j ) p,q .
Revised: we have ∂L ∂S ( j ) p,q = ∂L ∂U ( j ) q ∂U ( j ) q ∂M ( j ) p,q ∂M ( j ) p,q ∂S ( j ) p,q .

Original: We use the straight-through estimator (Bengio et al., 2013) for ∂M ( j ) p,q ∂S ( j ) p,q which yields∂L ∂S
Revised: We use the straight-through estimator (Bengio et al., 2013) for ∂M ( j ) p,q ∂S ( j ) p,q which yields

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , where ∂L ∂U ( j ) q is computed via backpropagation.
Revised: (cid:17) , where ∂L ∂U ( j ) q is computed via backpropagation.

Original: 2.2. U PDATING P RUNING S CORES FOR B INARY -A CTIVATION T ICKETS (MPT-1/1)
Revised: 2.2. U PDATING P RUNING S CORES FOR B INARY -A CTIVATION T ICKETS (MPT-1/1)

Original: Note that MPT-1/1 uses the sign activation function.
Revised: Note that MPT-1/1 uses the sign activation function.

Original: From Section 2.2.1, it immediately follows that
Revised: From Section 2.2.1, it immediately follows that

Original: [Equation]
Revised: [Equation]

Original: (cid:17) .

Original: However, updating
Revised: However, updating

Original: [Equation]
Revised: [Equation]

Original:  via backpropagation requires a gradient estimator for the sign activation function.
Revised: q via backpropagation requires a gradient estimator for the sign activation function.

Original: To motivate our choice of estimator note that we can approximate the sign function using a quadratic spline parameterized by some t > 0 :
Revised: To motivate our choice of estimator note that we can approximate the sign function using a quadratic spline parameterized by some t > 0 :

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] In (2), q i ( x ) = a i x 2 + b i x + c i and suitable values for the coefﬁcients are derived using the following zero- and ﬁrst-order constraints: q 1 ( − t )
Revised: In (2), q i ( x ) = a i x 2 + b i x + c i and suitable values for the coefﬁcients are derived using the following zero- and ﬁrst-order constraints: q 1 ( − t )

Original: = − 1 , q 1 (0) = 0 , q 2 (0) = 0 , q 2 ( t ) = 1 , q (cid:48) 1 ( − t ) = 0 , q (cid:48) 1 (0) =
Revised: = − 1 , q 1 (0) = 0 , q 2 (0) = 0 , q 2 ( t ) = 1 , q (cid:48) 1 ( − t ) = 0 , q (cid:48) 1 (0) =

Original: q (cid:48) 2 (0) , and q (cid:48) 2 ( t )
Revised: q (cid:48) 2 (0) , and q (cid:48) 2 ( t )

Original: = 0 .
Revised: = 0 .

Original: This yields q 1 ( x ) =
Revised: This yields q 1 ( x ) =

Original: ( x/t ) 

Revised: ( x/t )

Original: + 2( x/t ) and q 2 ( x ) = − ( x/t ) 2 + 2( x/t ) .
Revised: + 2( x/t ) and q 2 ( x ) = − ( x/t ) 2 + 2( x/t ) .

Original: As s t ( x ) approximates sign( x ) , we can use s (cid:48) t ( x ) as our gradient estimator.
Revised: As s t ( x ) approximates sign( x ) , we can use s (cid:48) t ( x ) as our gradient estimator.

Original: Since q (cid:48) 1 ( x ) = 2 t (1 + xt ) and q (cid:48) 2 ( x ) = 2 t (1 − xt ) it follows that s (cid:48) t ( x ) = (cid:104) 2 t (cid:16) 1 − | x | t (cid:17)(cid:105) 1 { x ∈ [ − t,t ] } ( x ) .
Revised: Since q (cid:48) 1 ( x ) = 2 t (1 + xt ) and q (cid:48) 2 ( x ) = 2 t (1 − xt ) it follows that s (cid:48) t ( x ) = (cid:104) 2 t (cid:16) 1 − | x | t (cid:17)(cid:105) 1 { x ∈ [ − t,t ] } ( x ) .

Original: The choice to approximate sign using a quadratic spline instead of a cubic spline results in a gradient estimator that can be implemented efﬁciently in PyTorch as torch.clamp(2*(1-torch.abs(x)/t)/t,min=0.0) .
Revised: The choice to approximate sign using a quadratic spline instead of a cubic spline results in a gradient estimator that can be implemented efﬁciently in PyTorch as torch.clamp(2*(1-torch.abs(x)/t)/t,min=0.0) .

Original: We note that lim t → 0 s t ( x ) = sign( x ) , which suggests that smaller values of t yield more suitable approximations.
Revised: We note that lim t → 0 s t ( x ) = sign( x ) , which suggests that smaller values of t yield more suitable approximations.

Original: Our experiments use s (cid:48) 1 ( x ) as the gradient estimator since we found it to work well in practice.
Revised: Our experiments use s (cid:48) 1 ( x ) as the gradient estimator since we found it to work well in practice.

Original: Finally, we note that taking t = 1 in our gradient estimator yields the same value as the gradient estimator in (Liu et al., 2018a), however, our implementation in PyTorch is 6 × more memory efﬁcient.
Revised: Finally, we note that taking t = 1 in our gradient estimator yields the same value as the gradient estimator in (Liu et al., 2018a), however, our implementation in PyTorch is 6 × more memory efﬁcient.

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: The primary goal of the experiments in Section 3.1 is to empirically verify our Multi-Prize Lottery Ticket Hypothesis .
Revised: The primary goal of the experiments in Section 3.1 is to empirically verify our Multi-Prize Lottery Ticket Hypothesis .

Original: As a secondary objective, we would like to determine tunable factors that make randomly-initialized networks amenable to containing readily identiﬁable Multi-Prize Tickets (MPTs).
Revised: As a secondary objective, we would like to determine tunable factors that make randomly-initialized networks amenable to containing readily identiﬁable Multi-Prize Tickets (MPTs).

Original: Thus, we test our hypothesis against the general trend of increasing the model size (depth and width) and monitor the accuracy of the identiﬁed MPTs.
Revised: Thus, we test our hypothesis against the general trend of increasing the model size (depth and width) and monitor the accuracy of the identiﬁed MPTs.

Original: After verifying our Multi-Prize Lottery
Revised: After verifying our Multi-Prize Lottery

Original: Ticket Hypothesis , we consider the performance of MPTs compared to state-of-the-arts in binary neural networks and their dense counterparts on CIFAR-10 and ImageNet datasets in Section 3.2.
Revised: Ticket Hypothesis , we consider the performance of MPTs compared to state-of-the-arts in binary neural networks and their dense counterparts on CIFAR-10 and ImageNet datasets in Section 3.2.

Original: 3. W HERE CAN WE EXPECT TO FIND MULTI - PRIZE TICKETS ?
Revised: 3. W HERE CAN WE EXPECT TO FIND MULTI - PRIZE TICKETS ?

Original: In this section, we empirically test the effect of overparameterization on the performance of MPTs.
Revised: In this section, we empirically test the effect of overparameterization on the performance of MPTs.

Original: We overparameterize networks by making them (a) deeper (Sec. 3.1.1) and (b) wider (Sec. 3.1.2).
Revised: We overparameterize networks by making them (a) deeper (Sec. 3.1.1) and (b) wider (Sec. 3.1.2).

Original: We use VGG (Simonyan & Zisserman, 2014) variants as our network architectures for searching for
Revised: We use VGG (Simonyan & Zisserman, 2014) variants as our network architectures for searching for

Original: MPTs.
Revised: MPTs.

Original: In each randomly weighted network, we ﬁnd winning tickets MPT-1/32 and MPT-1/1 for different pruning rates using Algorithm 1.
Revised: In each randomly weighted network, we ﬁnd winning tickets MPT-1/32 and MPT-1/1 for different pruning rates using Algorithm 1.

Original: We choose our baselines as dense full-precision models with learned weights.
Revised: We choose our baselines as dense full-precision models with learned weights.

Original: In all experiments, we use three independent initializations and report the average of Top-1 accuracy with with error bars extending to the lowest and highest Top-1 accuracy.
Revised: In all experiments, we use three independent initializations and report the average of Top-1 accuracy with with error bars extending to the lowest and highest Top-1 accuracy.

Original: Additional experiment conﬁguration details are provided in Appendix A.
Revised: Additional experiment conﬁguration details are provided in Appendix A.

Original: 3.1. D O WINNING TICKETS EXIST IN DEEP NETWORKS ?
Revised: 3.1. D O WINNING TICKETS EXIST IN DEEP NETWORKS ?

Original: In this experiment, we empirically test the following hypothesis: As a network grows deeper, the performance ofuntrained multi-prize tickets in the randomly initialized network will approach the performance of the same network with learned weights.
Revised: In this experiment, we empirically test the following hypothesis: As a network grows deeper, the performance of multi-prize tickets in the randomly initialized network will approach the performance of the same network with learned weights.

Original: We are further interested in exploring the required network depth for our hypothesis to be true.
Revised: We are further interested in exploring the required network depth for our hypothesis to be true.

Original: In Figure 2, we vary the depth of VGG architectures ( d = 2 to 8 ) and compare the Top-1 accuracy of MPTs (at different pruning rates) with weight-trained dense network.
Revised: In Figure 2, we vary the depth of VGG architectures ( d = 2 to 8 ) and compare the Top-1 accuracy of MPTs (at different pruning rates) with weight-trained dense network.

Original: We notice that there exist a range of pruning rates where the performance of MPTs are very similar, and beyond this range the performance drops quickly.
Revised: We notice that there exist a range of pruning rates where the performance of MPTs are very similar, and beyond this range the performance drops quickly.

Original: Interestingly, as the network depth increases, more parameters can be pruned without hurting the performance of MPTs.
Revised: Interestingly, as the network depth increases, more parameters can be pruned without hurting the performance of MPTs.

Original: For example, MPT-1/32 can match the performance of trained Conv-8 while having only ∼ 20% of its parameter count.
Revised: For example, MPT-1/32 can match the performance of trained Conv-8 while having only ∼ 20% of its parameter count.

Original: Interestingly, the performance gap between MPT-1/32 and MPT-1/1 does not change much with depth across different pruning rates.
Revised: Interestingly, the performance gap between MPT-1/32 and MPT-1/1 does not change much with depth across different pruning rates.

Original: We further note that the performance of MPTs improve when increasing the depth and both start to approach the performance of the dense model with learned weights.
Revised: We further note that the performance of MPTs improve when increasing the depth and both start to approach the performance of the dense model with learned weights.

Original: This gain starts to plateau beyond a certain depth, suggesting that the MPTs might be approaching the limit of their achievable accuracy.
Revised: This gain starts to plateau beyond a certain depth, suggesting that the MPTs might be approaching the limit of their achievable accuracy.

Original: Surprisingly, MPT-1/32 performs equally good (or better) than the weight-trained model regardless of having 50 − 80% lesser parameters and weights being binarized.
Revised: Surprisingly, MPT-1/32 performs equally good (or better) than the weight-trained model regardless of having 50 − 80% lesser parameters and weights being binarized.

Original: 3.1. D O WINNING TICKETS EXIST IN WIDE NETWORKS ?
Revised: 3.1. D O WINNING TICKETS EXIST IN WIDE NETWORKS ?

Original: Similar to the previous experiment, in this experiment, we empirically test the following hypothesis: As a network grows wider, the performance ofuntrained multi-prize tickets in the randomly initialized network will approach the performance of the same network with learned weights.
Revised: Similar to the previous experiment, in this experiment, we empirically test the following hypothesis: As a network grows wider, the performance of multi-prize tickets in the randomly initialized network will approach the performance of the same network with learned weights.

Original: We are further interested in exploring the required layer width for our hypothesis to be true.
Revised: We are further interested in exploring the required layer width for our hypothesis to be true.

Original: In Figures 3 and 4, we vary the width of different VGG architectures and compare the Top-1 accuracy of MPT-1/32 and MPT-1/1 tickets (at different pruning rates) with weight-trained dense network.
Revised: In Figures 3 and 4, we vary the width of different VGG architectures and compare the Top-1 accuracy of MPT-1/32 and MPT-1/1 tickets (at different pruning rates) with weight-trained dense network.

Original: A width multiplier of value 1 corresponds to the models in Figure 2.
Revised: A width multiplier of value 1 corresponds to the models in Figure 2.

Original: Performance of all the models improves when increasing the width and the performance of both MPT-1/32 and MPT-1/1 start to approach the performance of the dense model with learned weights.
Revised: Performance of all the models improves when increasing the width and the performance of both MPT-1/32 and MPT-1/1 start to approach the performance of the dense model with learned weights.

Original: Although, this gain starts to plateau beyond a certain width.
Revised: Although, this gain starts to plateau beyond a certain width.

Original: For both MPT-1/32 and MPT-1/1, as the width and depth increase the performance at different pruning rates approach the same value.
Revised: For both MPT-1/32 and MPT-1/1, as the width and depth increase the performance at different pruning rates approach the same value.

Original: This observed phenomenon yields a more signiﬁcant gain in the performance for MPTs with higher pruning rates.
Revised: This observed phenomenon yields a more signiﬁcant gain in the performance for MPTs with higher pruning rates.

Original: Similar to the previous experiment, the performance of MPT-1/32 matches (or exceeds) the performance of dense models for a large range of pruning rates.
Revised: Similar to the previous experiment, the performance of MPT-1/32 matches (or exceeds) the performance of dense models for a large range of pruning rates.

Original: Furthermore, in the high width regime, a large number of weights ( ∼ 90% ) can be pruned without having a noticeable impact on the performance of MPTs.
Revised: Furthermore, in the high width regime, a large number of weights ( ∼ 90% ) can be pruned without having a noticeable impact on the performance of MPTs.

Original: We also notice that the performance gap between MPT-1/32 and MPT-1/1 decreases signiﬁcantly with an increase the width which is in sharp contrast with the with the depth experiments where the performance gap between MPT-1/32 and MPT-1/1 appeared to be largely independent of the depth.
Revised: We also notice that the performance gap between MPT-1/32 and MPT-1/1 decreases signiﬁcantly with an increase the width which is in sharp contrast with the with the depth experiments where the performance gap between MPT-1/32 and MPT-1/1 appeared to be largely independent of the depth.

Original: Key Takeaways.
Revised: Key Takeaways.

Original: Our experiments verify Multi-Prize Lottery Ticket Hypothesis and additionally convey the signiﬁcance of choosing appropriate network depth and layer width for a given pruning rate.
Revised: Our experiments verify Multi-Prize Lottery Ticket Hypothesis and additionally convey the signiﬁcance of choosing appropriate network depth and layer width for a given pruning rate.

Original: In particular, we ﬁnd that a network with a large width can be pruned more aggressively without sacriﬁcing much accuracy, while the accuracy of a network with smaller widths suffers when pruning a large percentage of the weights.
Revised: In particular, we ﬁnd that a network with a large width can be pruned more aggressively without sacriﬁcing much accuracy, while the accuracy of a network with smaller widths suffers when pruning a large percentage of the weights.

Original: Similar patterns hold for the depth of the networks as well.
Revised: Similar patterns hold for the depth of the networks as well.

Original: The amount of overparametrization needed to approach the performance of dense networks seems to differ for MPT variants – MPT-1/1 requires higher depth and width compared to MPT-1/32.
Revised: The amount of overparametrization needed to approach the performance of dense networks seems to differ for MPT variants – MPT-1/1 requires higher depth and width compared to MPT-1/32.

Original: 3. H OW R EDUNDANT S TATE - OF - THE -A RT D EEP N EURAL N ETWORKS A CTUALLY A RE ?
Revised: 3. H OW R EDUNDANT S TATE - OF - THE -A RT D EEP N EURAL N ETWORKS A CTUALLY A RE ?

Original: Having shown that MPTs can perform equally good (or better) than overparameterized networks, this experiment aims to answer:
Revised: Having shown that MPTs can perform equally good (or better) than overparameterized networks, this experiment aims to answer:

Original: Are state-of-the-art weight-trained DNNs overparametrized enough that signiﬁcantly smaller and untrained MPTs can match (or beat) their performance?
Revised: Are state-of-the-art weight-trained DNNs overparametrized enough that signiﬁcantly smaller multi-prize tickets can match (or beat) their performance?

Original: Experimental Conﬁguration.
Revised: Experimental Conﬁguration.

Original: Instead of focusing on extremely large DNNs, we experiment with small to moderate size DNNs.
Revised: Instead of focusing on extremely large DNNs, we experiment with small to moderate size DNNs.

Original: Speciﬁcally, we analyze the redundancy of following backbone models: (1) VGG-Small and ResNet-18 on CIFAR-10, and (2) WideResNet-34 and WideResNet50 on ImageNet.
Revised: Speciﬁcally, we analyze the redundancy of following backbone models: (1) VGG-Small and ResNet-18 on CIFAR-10, and (2) WideResNet-34 and WideResNet50 on ImageNet.

Original: As we will show later that even these models are highly redundant, thus, our ﬁnding automatically extends to larger models.
Revised: As we will show later that even these models are highly redundant, thus, our ﬁnding automatically extends to larger models.

Original: In this process, we also perform a comprehensive comparison of the performance of our multi-prize winning tickets with state-of-the-art in binary neural networks (BNNs).
Revised: In this process, we also perform a comprehensive comparison of the performance of our multi-prize winning tickets with state-of-the-art in binary neural networks (BNNs).

Original: Details on the experimental conﬁguration are provided in Appendix A.
Revised: Details on the experimental conﬁguration are provided in Appendix A.

Original: In this experiment, we use Algorithm 1 to ﬁnd multi-prize tickets within randomly initialized backbone networks.
Revised: In this experiment, we use Algorithm 1 to ﬁnd multi-prize tickets within randomly initialized backbone networks.

Original: We compare the Top-1 accuracy and number of non-zero parameters for our MPT1/32 and MPT-1/1 tickets with selected baselines in binary neural networks (Qin et al., 2020a).
Revised: We compare the Top-1 accuracy and number of non-zero parameters for our MPT1/32 and MPT-1/1 tickets with selected baselines in binary neural networks (Qin et al., 2020a).

Original: A more comprehensive comparison can be found in Appendix D. Results for CIFAR-10 are shown in
Revised: A more comprehensive comparison can be found in Appendix D. Results for CIFAR-10 are shown in

Original: Tables 1 and 2, and results for ImageNet are shown in Tables 3 and 4.
Revised: Tables 1 and 2, and results for ImageNet are shown in Tables 3 and 4.

Original: Next to each MPT method we include the percentage of weights pruned in parentheses.
Revised: Next to each MPT method we include the percentage of weights pruned in parentheses.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: trained binary-1/32 networks on ImageNet.
Revised: binary-1/32 networks on ImageNet.

Original: [Table]
Revised: [Table]

Original: Our results highlight that SOTA DNN models are extremely redundant.
Revised: Our results highlight that SOTA DNN models are extremely redundant.

Original: For similar parameter count, our binaryand untrained MPT-1/32 models outperform even full-precision models with learned weights.
Revised: For similar parameter count, our binary MPT-1/32 models outperform even full-precision models with learned weights.

Original: When compared to state-of-the-art in BNNs, with minimal hyperparameter tuning our multi-prize tickets achieve comparable (or higher) Top-1 accuracy.
Revised: When compared to state-of-the-art in BNNs, with minimal hyperparameter tuning our multi-prize tickets achieve comparable (or higher) Top-1 accuracy.

Original: Speciﬁcally, our winning tickets MPT-1/32 outperform trained binary weight networks – (a) on CIFAR-10 MPT-1/32 with 80% pruned ResNet-18 model achieves 94.15% compared to the previous best 92.3% achieved using unpruned ResNet-56 (Bai et al., 2018), and ( ) on ImageNet MPT-1/
Revised: Speciﬁcally, our winning tickets MPT-1/32 outperform trained binary weight networks – (a) on CIFAR-10 MPT-1/32 with 80% pruned ResNetmodel achieves 94.15% compared to the previous best 92.3% achieved using unpruned ResNet56 (Bai et al., 2018), and (b) on ImageNet MPT-1/32 with 70% pruned Wide ResNet-50 achieves

Original: 70% pruned ResNet-50 achieves 73.3% compared to the previous best 72.8% achieved using unpruned [-ResNet
-]{+ResNet-
+}
Revised: 73.3% compared to the previous best 72.8% achieved using unpruned [-ResNet
-]{+ResNet-
+}

Original: (Yang et al., 2019).
Revised: (Yang et al., 2019).

Original: Note that on CIFAR-10, MPT-1/32 outperforms its signiﬁcantly larger dense and full precision counterpart with learned weights that achieves 93 .
Revised: Note that on CIFAR-10, MPT-1/32 outperforms its signiﬁcantly larger dense and full precision counterpart with learned weights that achieves 93 .

Original: 02% accuracy.
Revised: 02% accuracy.

Original: On ImageNet, a randomly weighted Wide ResNet-50 contains MPT-1/32 which is smaller than, but outperforms full precision ResNet-34 with learned weights (i.e., 73.27% Top-1 accuracy).
Revised: On ImageNet, a randomly weighted Wide ResNet-50 contains MPT-1/32 which is smaller than, but outperforms full precision ResNet34 with learned weights (i.e., 73.27% Top-1 accuracy).

Original: Moving forward, we believe that there are several avenues for addressing the inability of MPT-1/ to achieve state-of-the-art results.
Revised: Moving forward, we believe that there are several avenues for addressing the inability of MPT-1/ to achieve state-of-the-art results.

Original: As this was our initial attempt to identify MPT-1/1, we used the baseline conﬁgurations of these networks with minimal modiﬁcations.
Revised: As this was our initial attempt to identify MPT-1/1, we used the baseline conﬁgurations of these networks with minimal modiﬁcations.

Original: Hence, it is reasonable that our accuracy falls between the performance of BNN (Courbariaux et al., 2016) and
Revised: Hence, it is reasonable that our accuracy falls between the performance of BNN (Courbariaux et al., 2016) and

Original: XNOR-Net
Revised: XNOR-Net

Original: (Rastegari et al., 2016) which also make minimal modiﬁcations to the networks they are training.
Revised: (Rastegari et al., 2016) which also make minimal modiﬁcations to the networks they are training.

Original: Searches for MPT-1/1 in state-of-the-art BNNs are likely to yield MPT-1/1 networks with improved performance.
Revised: Searches for MPT-1/1 in state-of-the-art BNNs are likely to yield MPT-1/1 networks with improved performance.

Original: Additionally, alternative approaches for updating the pruning mask in biprop could alleviate issues with back-propagating gradients through binary activation networks.
Revised: Additionally, alternative approaches for updating the pruning mask in biprop could alleviate issues with back-propagating gradients through binary activation networks.

Original: Finally, specialized accelerator designs for exploiting the sparsity of MPTs is a worthwhile future direction.
Revised: Finally, specialized accelerator designs for exploiting the sparsity of MPTs is a worthwhile future direction.

Original:  D ISCUSSION AND I MPLICATIONS
Revised:  D ISCUSSION AND I MPLICATIONS

Original: Existing compression approaches (e.g., pruning and binarization) typically rely on some form of weight-training.
Revised: Existing compression approaches (e.g., pruning and binarization) typically rely on some form of weight-training.

Original: This paper showed that a sufﬁciently overparametrized randomly weighted network contains binary subnetworks that achieve high accuracy (comparable to dense and full precision original network with learned weights) without any training.
Revised: This paper showed that a sufﬁciently overparametrized randomly weighted network contains binary subnetworks that achieve high accuracy (comparable to dense and full precision original network with learned weights) without any training.

Original: We referred to this ﬁnding as MultiPrize Lottery Ticket Hypothesis.
Revised: We referred to this ﬁnding as the Multi-Prize Lottery Ticket Hypothesis.

Original: We also proved the existence of such winning tickets and presented a generic procedure to ﬁnd them.
Revised: We also proved the existence of such winning tickets and presented a generic procedure to ﬁnd them.

Original: Our comparison with state-of-the-art neural networks corroborated our hypothesis.
Revised: Our comparison with state-of-the-art neural networks corroborated our hypothesis.

Original: Our work has several important practical and theoretical implications.
Revised: Our work has several important practical and theoretical implications.

Original: Algorithmic.
Revised: Algorithmic.

Original: Our biprop framework enjoys certain advantages over traditional weightoptimization.
Revised: Our biprop framework enjoys certain advantages over traditional weightoptimization.

Original: First, contemporary experience suggests that sparse BNN training from scratch is challenging.
Revised: First, contemporary experience suggests that sparse BNN training from scratch is challenging.

Original: Both sparseness and binarization bring their own challenges for gradient-based weight training – getting stuck at bad local minima in the sparse regime, incompatibility of backpropagation due to discontinuity in activation function, etc.
Revised: Both sparseness and binarization bring their own challenges for gradient-based weight training – getting stuck at bad local minima in the sparse regime, incompatibility of backpropagation due to discontinuity in activation function, etc.

Original: Although we used gradient-based approaches in this paper, biprop is ﬂexible to accommodate different class of algorithms that might avoid the pitfalls of gradient-based weight training.
Revised: Although we used gradient-based approaches in this paper, biprop is ﬂexible to accommodate different class of algorithms that might avoid the pitfalls of gradient-based weight training.

Original: Next, in contrast to weight-optimization that requires large model size and massive compute resources to achieve high performance, our hypothesis suggests that one can achieve similar performance without ever training the large model.
Revised: Next, in contrast to weight-optimization that requires large model size and massive compute resources to achieve high performance, our hypothesis suggests that one can achieve similar performance without ever training the large model.

Original: Therefore, strategies such as fast ticket search (You et al., 2019) or forward ticket selection (Ye et al., 2020) can be developed to enable more efﬁcient ways of ﬁnding–or even designing–MPTs.
Revised: Therefore, strategies such as fast ticket search (You et al., 2019) or forward ticket selection (Ye et al., 2020) can be developed to enable more efﬁcient ways of ﬁnding–or even designing–MPTs.

Original: Finally, as opposed to weight-optimization, biprop by design achieves compact yet accurate models.
Revised: Finally, as opposed to weight-optimization, biprop by design achieves compact yet accurate models.

Original: Theoretical.
Revised: Theoretical.

Original: MPTs achieve similar performance as the model with learned weights.
Revised: MPTs achieve similar performance as the model with learned weights.

Original: First, this observation notes the beneﬁt of overparameterization in the neural network learning and reinforces the idea that an important task of gradient descent (and learning in general) may be to effectively compress overparametrized models to ﬁnd multi-prize tickets.
Revised: First, this observation notes the beneﬁt of overparameterization in the neural network learning and reinforces the idea that an important task of gradient descent (and learning in general) may be to effectively compress overparametrized models to ﬁnd multi-prize tickets.

Original: Next, our results highlight the expressive power of MPTs – since we showed that compressed subnetworks can approximate any target neural network who are known to be universal approximators, our MPTs are also universal approximators.
Revised: Next, our results highlight the expressive power of MPTs – since we showed that compressed subnetworks can approximate any target neural network who are known to be universal approximators, our MPTs are also universal approximators.

Original: Finally,  multi-prize lottery ticket hypothesis also uncovers the generalization properties of DNNs.
Revised: Finally, the multi-prize lottery ticket hypothesis also uncovers the generalization properties of DNNs.

Original: Generalization theory for DL is still in its infancy and its not clear what and how DNNs learn (Neyshabur et al., 2017).
Revised: Generalization theory for DL is still in its infancy and its not clear what and how DNNs learn (Neyshabur et al., 2017).

Original: Multi-prize lottery ticket hypothesis may serve as a valuable tool for answering such questions as it indicates the dependence of generalization on the compressiblity.
Revised: Multi-prize lottery ticket hypothesis may serve as a valuable tool for answering such questions as it indicates the dependence of generalization on the compressiblity.

Original: Practical.
Revised: Practical.

Original: Huge storage and heavy computation requirements of state-of-the-art deep neural networks inevitably limit their applications in practice.
Revised: Huge storage and heavy computation requirements of state-of-the-art deep neural networks inevitably limit their applications in practice.

Original: Multi-prize tickets are signiﬁcantly lighter, faster, and efﬁcient and still perform equally good.
Revised: Multi-prize tickets are signiﬁcantly lighter, faster, and efﬁcient while maintaining performance.

Original: This unlocks a range of potential applications DL could be applied to, e.g., applications with resource-constrained devices (e.g. , the mobile phones, embedded devices, etc.).
Revised: This unlocks a range of potential applications DL could be applied to (e.g., applications with resource-constrained devices such as mobile phones, embedded devices, etc.).

Original: Our results also indicate that existing SOTA models might be spending far more compute and power than actually needed to achieve a certain performance.
Revised: Our results also indicate that existing SOTA models might be spending far more compute and power than is needed to achieve a certain performance.

Original: In other words, 
Revised: In other words, SOTA

Original: SOTA DL models have terrible energy efﬁciency and signiﬁcant carbon footprint (Strubell et al., 2019).
Revised: DL models have terrible energy efﬁciency and signiﬁcant carbon footprint (Strubell et al., 2019).

Original: In this regards, MPTs have  potential to enable environmental friendly artiﬁcial intelligence.
Revised: In this regard, MPTs have the potential to enable environmentally friendly artiﬁcial intelligence.

Original: A H YPERPARAMETER C ONFIGURATIONS A. H YPERPARAMETERS FOR S ECTION 3. 
Revised: A H YPERPARAMETER C ONFIGURATIONS A. H YPERPARAMETERS FOR S ECTION 3.

Original: Experimental Conﬁguration.
Revised: Experimental Conﬁguration.

Original: For MPT-1/32 tickets, the network structure is not modiﬁed from the original.
Revised: For MPT-1/32 tickets, the network structure is not modiﬁed from the original.

Original: For MPT-1/1 tickets, the network structure is modiﬁed by moving the max-pooling layer directly after the convolution layer and adding a batch-normalization layer before the binary activation function, as is common in many BNN architectures (Rastegari et al., 2016).
Revised: For MPT-1/1 tickets, the network structure is modiﬁed by moving the max-pooling layer directly after the convolution layer and adding a batch-normalization layer before the binary activation function, as is common in many BNN architectures (Rastegari et al., 2016).

Original: We choose our baselines as dense full precision models with learned weights.
Revised: We choose our baselines as dense full precision models with learned weights.

Original: The baselines were obtained by training backbone networks using the Adam optimizer with learning rate of 0 .
Revised: The baselines were obtained by training backbone networks using the Adam optimizer with learning rate of 0 .

Original: 0003 for 100 epochs and with a batch size of 60 .
Revised: 0003 for 100 epochs and with a batch size of 60 .

Original: In each randomly weighted backbone network, we ﬁnd winning tickets
Revised: In each randomly weighted backbone network, we ﬁnd winning tickets

Original: MPT-1/32 and MPT-1/1 for different pruning rates using Algorithm 1.
Revised: MPT-1/32 and MPT-1/1 for different pruning rates using Algorithm 1.

Original: We use Adam optimizer with an initial learning rate of 0 . 1 and a cosine decay learning rate policy for 250 epochs with a batch size of 128 and a weight decay value of 0 . 0001 .
Revised: We use Adam optimizer with an initial learning rate of 0 . 1 and a cosine decay learning rate policy for 250 epochs with a batch size of 128 and a weight decay value of 0 . 0001 .

Original: For both the weight-optimized and MPT networks, the weights are initialized using the Kaiming Normal distribution (He et al., 2015).
Revised: For both the weight-optimized and MPT networks, the weights are initialized using the Kaiming Normal distribution (He et al., 2015).

Original: A. H YPERPARAMETERS FOR S ECTION 3. 
Revised: A. H YPERPARAMETERS FOR S ECTION 3.

Original: Experimental Conﬁguration.
Revised: Experimental Conﬁguration.

Original: CIFAR-10 Experiments: MPT-1/32 Models: VGG-Small (Learning Rate Policy optimizer: adam, lr: 0.1, lr policy: cosine lr, Network training conﬁg epochs: 250, weight decay: 0.0001, momentum: 0.9, batchsize: 128).
Revised: CIFAR-10 Experiments: MPT-1/32 Models: VGG-Small (Learning Rate Policy optimizer: adam, lr: 0.1, lr policy: cosine lr, Network training conﬁg epochs: 250, weight decay: 0.0001, momentum: 0.9, batchsize: 128).

Original: ResNet-18: (Learning Rate Policy optimizer: sgd, lr: 0.1, lr policy: cosine lr, Network training conﬁg epochs: 100, weight decay: 0.0005, momentum: 0.9, batchsize: 256)
Revised: ResNet-18: (Learning Rate Policy optimizer: sgd, lr: 0.1, lr policy: cosine lr, Network training conﬁg epochs: 100, weight decay: 0.0005, momentum: 0.9, batchsize: 256)

Original: MPT-1/1 Models: VGG-Small (Learning Rate Policy: optimizer: adam, lr: 0.0002, lr policy: cosine lr; Network training conﬁguration: epochs: 600, weight decay: 9.6, momentum: 0.9, batchsize: 128)
Revised: MPT-1/1 Models: VGG-Small (Learning Rate Policy: optimizer: adam, lr: 0.0002, lr policy: cosine lr; Network training conﬁguration: epochs: 600, weight decay: 9.6, momentum: 0.9, batchsize: 128)

Original: ImageNet Experiments: MPT-1/32 Models: Wide ResNet-34 and Wide ResNet-50 (Learning Rate
Revised: ImageNet Experiments: MPT-1/32 Models: Wide ResNet-34 and Wide ResNet-50 (Learning Rate

Original: Policy: optimizer: sgd, lr: 0.256, lr policy: cosine lr, warmup length: 5; Network training conﬁguration: epochs: 100, weight decay: 0.000030517578125, momentum: 0.875, batch size: 256, label smoothing: 0.1)
Revised: Policy: optimizer: sgd, lr: 0.256, lr policy: cosine lr, warmup length: 5; Network training conﬁguration: epochs: 100, weight decay: 0.000030517578125, momentum: 0.875, batch size: 256, label smoothing: 0.1)

Original: MPT-1/1 Models: Wide ResNet-34 (Learning Rate Policy optimizer: adam, lr: 0.256, lr policy: cosine lr, warmup length: 5; Network training conﬁguration: epochs: 200, weight decay: 0.000030517578125, momentum: 0.875, batchsize: 256, label smoothing: 0.1)
Revised: MPT-1/1 Models: Wide ResNet-34 (Learning Rate Policy optimizer: adam, lr: 0.256, lr policy: cosine lr, warmup length: 5; Network training conﬁguration: epochs: 200, weight decay: 0.000030517578125, momentum: 0.875, batchsize: 256, label smoothing: 0.1)

Original: The weights are initialized using the Kaiming Normal distribution (He et al., 2015) for all the models except for MPT-1/32 on ImageNet where we use Signed Kaiming Constant following Ramanujan et al.
Revised: The weights are initialized using the Kaiming Normal distribution (He et al., 2015) for all the models except for MPT-1/32 on ImageNet where we use Signed Kaiming Constant following Ramanujan et al.

Original: B E XISTENCE OF B INARY -W EIGHT S UBNETWORK A PPROXIMATING T ARGET N ETWORK
Revised: B E XISTENCE OF B INARY -W EIGHT S UBNETWORK A PPROXIMATING T ARGET N ETWORK

Original: In the following analysis, note that we write Bin ( {− 1 , +1 } m × n ) to denote matrices of dimension m × n whose components are independently sampled from a binomial distribution with elements {− 1 , +1 } and probability p = 1 / 2 .
Revised: In the following analysis, note that we write Bin ( {− 1 , +1 } m × n ) to denote matrices of dimension m × n whose components are independently sampled from a binomial distribution with elements {− 1 , +1 } and probability p = 1 / 2 .

Original: Lemma 1.
Revised: Lemma 1.

Original: Let s ∈ [ d ] , α ∈ (cid:104) − 1 √ s , 1 √ s (cid:105) ,
Revised: Let s ∈ [ d ] , α ∈ (cid:104) − 1 √ s , 1 √ s (cid:105) ,

Original: i ∈
Revised: i ∈

Original: [ d ] , and ε, δ ≥ 0 be given.
Revised: [ d ] , and ε, δ ≥ 0 be given.

Original: Let B ∈ {− 1 , +1 }
Revised: Let B ∈ {− 1 , +1 }

Original: k × d be chosen randomly from Bin ( {− 1 , 1 } k × d ) and u ∈ {− 1 , +1 } k be chosen randomly from
Revised: k × d be chosen randomly from Bin ( {− 1 , 1 } k × d ) and u ∈ {− 1 , +1 } k be chosen randomly from

Original: Bin ( {− 1 , +1 } k ) .
Revised: Bin ( {− 1 , +1 } k ) .

Original: If k
Revised: If k

Original: [Equation]
Revised: [Equation]

Original: then with probability at least 1 − δ there exist masks ˜ m ∈ { 0 , 1 } k and M ∈ { 0 , 1 } k × d such that the function g : R d → R deﬁned by
Revised: then with probability at least 1 − δ there exist masks ˜ m ∈ { 0 , 1 } k and M ∈ { 0 , 1 } k × d such that the function g : R d → R deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for all (cid:107) x (cid:107) ∞ ≤ 1 .
Revised: for all (cid:107) x (cid:107) ∞ ≤ 1 .

Original: Furthermore, (cid:107) ˜ m (cid:107) 

Revised: Furthermore, (cid:107) ˜ m (cid:107)

Original: = (cid:107) M (cid:107) 0 ≤ 2 ε √ s , and max 1 ≤ j ≤ k (cid:107) M j, : (cid:107) 0 ≤ 1 .
Revised: = (cid:107) M (cid:107) 0 ≤ 2 ε √ s , and max 1 ≤ j ≤ k (cid:107) M j, : (cid:107) 0 ≤ 1 .

Original: Proof.
Revised: Proof.

Original: If | α | ≤ ε then taking M = 0 yields the desired result.
Revised: If | α | ≤ ε then taking M = 0 yields the desired result.

Original: Suppose that | α | > ε .
Revised: Suppose that | α | > ε .

Original: Then there exists a c i ∈ N such that
Revised: Then there exists a c i ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: Hence, it follows that
Revised: Hence, it follows that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal inequality follows from (6) and the hypothesis that (cid:107) x (cid:107) ∞ ≤ 1 .
Revised: where the ﬁnal inequality follows from (6) and the hypothesis that (cid:107) x (cid:107) ∞ ≤ 1 .

Original: Our goal now is to show that with probability 1 − δ the random initialization of u and B yield masks ˜ m and M such that g ( x ) =
Revised: Our goal now is to show that with probability 1 − δ the random initialization of u and B yield masks ˜ m and M such that g ( x ) =

Original: c i ε sign( α ) x i .
Revised: c i ε sign( α ) x i .

Original: Now ﬁx i ∈ [ d ] and take k (cid:48) =
Revised: Now ﬁx i ∈ [ d ] and take k (cid:48) =

Original: k 2 .
Revised: k 2 .

Original: First, we consider the probability
Revised: First, we consider the probability

Original: P (
Revised: P (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: and B
Revised: and B

Original: [Equation]
Revised: [Equation]

Original: ,i =
Revised: ,i =

Original: [Equation]
Revised: [Equation]

Original: ( α
Revised: ( α

Original: [Equation]
Revised: [Equation]

Original: }| < c
Revised: }| < c

Original: [Equation]
Revised: [Equation]

Original: As u and B : ,i are each sampled from a binomial distribution with k (cid:48) trials, the distribution that the pair ( u j , B j,i ) is sampled from is a multinomial distribution with four possible events each having a probability of 1 / 4 .
Revised: As u and B : ,i are each sampled from a binomial distribution with k (cid:48) trials, the distribution that the pair ( u j , B j,i ) is sampled from is a multinomial distribution with four possible events each having a probability of 1 / 4 .

Original: Since we are only interested in the event ( u j , B j,i ) =
Revised: Since we are only interested in the event ( u j , B j,i ) =

Original: (+1 , sign( α )) occurring, we can instead consider a binomial distribution where P (( u j , B j,i )
Revised: (+1 , sign( α )) occurring, we can instead consider a binomial distribution where P (( u j , B j,i )

Original: =
Revised: =

Original: (+1 , sign( α ))
Revised: (+1 , sign( α ))

Original: = 14 and
Revised: = 14 and

Original: P (( u j , B j,i ) (cid:54) =
Revised: P (( u j , B j,i ) (cid:54) =

Original: (+1 , sign( α ))
Revised: (+1 , sign( α ))

Original: = 34 .
Revised: = 34 .

Original: Hence, using Hoeffding’s inequality we have that
Revised: Hence, using Hoeffding’s inequality we have that

Original: P ( |{ j ∈ [ k (cid:48) ] : u j = +1 and B j,i = sign( α ) }| < c
Revised: P ( |{ j ∈ [ k (cid:48) ] : u j = +1 and B j,i = sign( α ) }| < c

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal inequality follows since exp() is an increasing function and −
Revised: where the ﬁnal inequality follows since exp() is an increasing function and −

Original: 2 c 2 i k (cid:48) < 0 .
Revised: 2 c 2 i k (cid:48) < 0 .

Original: From (6) and the fact that | α | ≤ 1 √ s , it follows that
Revised: From (6) and the fact that | α | ≤ 1 √ s , it follows that

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining our hypothesis in (3) with (12) yields that
Revised: Combining our hypothesis in (3) with (12) yields that

Original: [Equation]
Revised: [Equation]

Original: Substituting (13) into (11) yields
Revised: Substituting (13) into (11) yields

Original: [Equation]
Revised: [Equation]

Original: Additionally, it follows from the same argument that
Revised: Additionally, it follows from the same argument that

Original: [Equation]
Revised: [Equation]

Original: From (14) and (15) it follows with probability at least 1 − δ that there exist sets S
Revised: From (14) and (15) it follows with probability at least 1 − δ that there exist sets S

Original: + :=
Revised: + :=

Original: { j : u j = +1 and B j,i = sign( α ) } and S − :=
Revised: { j : u j = +1 and B j,i = sign( α ) } and S − :=

Original: { j : u j = − 1 and B j,i = − sign( α ) } satisfying | S + | = | S − | =
Revised: { j : u j = − 1 and B j,i = − sign( α ) } satisfying | S + | = | S − | =

Original: c i and S + ∩ S − = ∅ .
Revised: c i and S + ∩ S − = ∅ .

Original: Using these sets, we deﬁne the components of the mask ˜ m
Revised: Using these sets, we deﬁne the components of the mask ˜ m

Original: and M by
Revised: and M by

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Using the deﬁnition of g ( x ) in (4) we now have that
Revised: Using the deﬁnition of g ( x ) in (4) we now have that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal equality follows from the identity σ ( a ) − σ ( − a )
Revised: where the ﬁnal equality follows from the identity σ ( a ) − σ ( − a )

Original: = a , for all a ∈ R .
Revised: = a , for all a ∈ R .

Original: This concludes the proof of (5).
Revised: This concludes the proof of (5).

Original: Lastly, by our choice of ˜ m in (16), M in (17), and (12), it follows that
Revised: Lastly, by our choice of ˜ m in (16), M in (17), and (12), it follows that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: The next step is to consider an analogue for Lemma A.2 from (Malach et al., 2020) which we provide in Lemma 2.
Revised: The next step is to consider an analogue for Lemma A.2 from (Malach et al., 2020) which we provide in Lemma 2.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let s ∈ [ d ] , w ∗ ∈ (cid:104) − 1 √ s , 1 √ s (cid:105)
Revised: Let s ∈ [ d ] , w ∗ ∈ (cid:104) − 1 √ s , 1 √ s (cid:105)

Original: d with (cid:107) w ∗ (cid:107) 0 ≤ s , and ε, δ > 0 be given.
Revised: d with (cid:107) w ∗ (cid:107) 0 ≤ s , and ε, δ > 0 be given.

Original: Let B ∈ {− 1 , +1 }
Revised: Let B ∈ {− 1 , +1 }

Original: k × d be chosen randomly from Bin ( {− 1 , 1 } k × d ) and u ∈ {− 1 , +1 } k be chosen randomly from Bin ( {− 1 , +1 } k ) .
Revised: k × d be chosen randomly from Bin ( {− 1 , 1 } k × d ) and u ∈ {− 1 , +1 } k be chosen randomly from Bin ( {− 1 , +1 } k ) .

Original: If
Revised: If

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then with probability at least 1 − δ there exist masks ˜ m ∈ { 0 , 1 } k and M ∈ { 0 , 1 } k × d such that the function g : R d → R deﬁned by
Revised: then with probability at least 1 − δ there exist masks ˜ m ∈ { 0 , 1 } k and M ∈ { 0 , 1 } k × d such that the function g : R d → R deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: , for
Revised: , for

Original: [Equation]
Revised: [Equation]

Original: Furthermore,
Revised: Furthermore,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Assume k = s · (cid:108) 

Revised: Assume k = s · (cid:108)

Original: √ s ε + 16 log (cid:0) 2 sδ (cid:1)(cid:109) and set k (cid:48)
Revised: √ s ε + 16 log (cid:0) 2 sδ (cid:1)(cid:109) and set k (cid:48)

Original: =
Revised: =

Original: ks .
Revised: ks .

Original: Note that if k > s · (cid:108) 

Revised: Note that if k > s · (cid:108)

Original: √ s ε + 16 log (cid:0) 2 sδ (cid:1)(cid:109)
Revised: √ s ε + 16 log (cid:0) 2 sδ (cid:1)(cid:109)

Original: then the excess neurons can be masked yielding the desired value for k .
Revised: then the excess neurons can be masked yielding the desired value for k .

Original: We decompose u , ˜ m , B , and M into s equal size submatrices by deﬁning
Revised: We decompose u , ˜ m , B , and M into s equal size submatrices by deﬁning

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: for i ∈ [ s ] .
Revised: for i ∈ [ s ] .

Original: Note that these submatrices satisfy
Revised: Note that these submatrices satisfy

Original: [Equation]
Revised: [Equation]

Original: Now let I :=
Revised: Now let I :=

Original: { i ∈ [ d ] : w ∗
Revised: { i ∈ [ d ] : w ∗

Original: i (cid:54) = 0 } .
Revised: i (cid:54) = 0 } .

Original: By our hypothesis that (cid:107) w ∗ (cid:107) 0 ≤ s , it follows that |I| ≤ s .
Revised: By our hypothesis that (cid:107) w ∗ (cid:107) 0 ≤ s , it follows that |I| ≤ s .

Original: WLOG, assume that I ⊆
Revised: WLOG, assume that I ⊆

Original: [ s ] .
Revised: [ s ] .

Original: Now ﬁx i ∈ [ s ] and deﬁne g i : R d → R by
Revised: Now ﬁx i ∈ [ s ] and deﬁne g i : R d → R by

Original: [Equation]
Revised: [Equation]

Original: By (23), taking ε (cid:48) =
Revised: By (23), taking ε (cid:48) =

Original: εs and δ (cid:48) =
Revised: εs and δ (cid:48) =

Original: δs yields that k (cid:48) ≥ 16 ε (cid:48) √ s + 16 log (cid:0) 2 δ (cid:48) (cid:1) .
Revised: δs yields that k (cid:48) ≥ 16 ε (cid:48) √ s + 16 log (cid:0) 2 δ (cid:48) (cid:1) .

Original: Hence, it follows from
Revised: Hence, it follows from

Original: Lemma 1 that with probability at least 1 − δ (cid:48) there exist ˜ m
Revised: Lemma 1 that with probability at least 1 − δ (cid:48) there exist ˜ m

Original: ( i ) ∈ { 0 , 1 } k (cid:48) and M ( i ) ∈ { 0 , 1 } k (cid:48)
Revised: ( i ) ∈ { 0 , 1 } k (cid:48) and M ( i ) ∈ { 0 , 1 } k (cid:48)

Original: ×
Revised: ×

Original: d such that
Revised: d such that

Original: [Equation]
Revised: [Equation]

Original: (32) for every x ∈ R d with (cid:107) x (cid:107) ∞ ≤ 1 , and (cid:107) ˜ m
Revised: (32) for every x ∈ R d with (cid:107) x (cid:107) ∞ ≤ 1 , and (cid:107) ˜ m

Original: ( i ) (cid:107) 

Revised: ( i ) (cid:107)

Original: = (cid:107) M
Revised: = (cid:107) M

Original: ( i ) (cid:107) 0 ≤
Revised: ( i ) (cid:107) 0 ≤

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of g ( x ) in (24)
Revised: By the deﬁnition of g ( x ) in (24)

Original: [Equation]
Revised: [Equation]

Original: using (30) yields
Revised: using (30) yields

Original: [Equation]
Revised: [Equation]

Original: Hence, combining (32) for all i ∈ [ s ]
Revised: Hence, combining (32) for all i ∈ [ s ]

Original: [Equation]
Revised: [Equation]

Original: it follows that with probability at least 1 − δ we have
Revised: it follows that with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: it follows from (30) and (33) that
Revised: it follows from (30) and (33) that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof
Revised: which concludes the proof

Original: [Equation]
Revised: [Equation]

Original: We now state and prove an analogue to Lemma A.5 in (Malach et al., 2020) which is the last lemma we will need to establish the desired result.
Revised: We now state and prove an analogue to Lemma A.5 in (Malach et al., 2020) which is the last lemma we will need to establish the desired result.

Original: Lemma 3.
Revised: Lemma 3.

Original: Let s ∈ [ d ] , W ∗ ∈ (cid:104) − 1 √ s , 1 √ s (cid:105) n × d with (cid:107) W ∗ (cid:107) 0 ≤ s , F : R d → R n deﬁned by
Revised: Let s ∈ [ d ] , W ∗ ∈ (cid:104) − 1 √ s , 1 √ s (cid:105) n × d with (cid:107) W ∗ (cid:107) 0 ≤ s , F : R d → R n deﬁned by

Original: F i ( x ) =
Revised: F i ( x ) =

Original: σ ( (cid:104) w ∗ i , x (cid:105) ) , and ε, δ > 0 be given.
Revised: σ ( (cid:104) w ∗ i , x (cid:105) ) , and ε, δ > 0 be given.

Original: Let B ∈ {− 1 , +1 }
Revised: Let B ∈ {− 1 , +1 }

Original: k × d be chosen randomly from
Revised: k × d be chosen randomly from

Original: Bin ( {− 1 , 1 } k × d ) and U ∈ {− 1 , +1 } k × n be chosen randomly from Bin ( {− 1 , +1 } k × n ) .
Revised: Bin ( {− 1 , 1 } k × d ) and U ∈ {− 1 , +1 } k × n be chosen randomly from Bin ( {− 1 , +1 } k × n ) .

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then with probability at least 1 − δ there exist masks ˜ M ∈ { 0 , 1 } k × n and M ∈ { 0 , 1 }
Revised: then with probability at least 1 − δ there exist masks ˜ M ∈ { 0 , 1 } k × n and M ∈ { 0 , 1 }

Original: k × d such that the function G : R d → R n deﬁned by
Revised: k × d such that the function G : R d → R n deﬁned by

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Furthermore, (cid:107) ˜ M (cid:107)
Revised: Furthermore, (cid:107) ˜ M (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ns
Revised: ns

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Assume k = ns · (cid:108) 16 √ nsε + 16 log (cid:0) 2 nsδ (cid:1)(cid:109) and set k (cid:48)
Revised: Assume k = ns · (cid:108) 16 √ nsε + 16 log (cid:0) 2 nsδ (cid:1)(cid:109) and set k (cid:48)

Original: =
Revised: =

Original: kn .
Revised: kn .

Original: Note that if k > ns · (cid:108) 16 √ nsε + 16 log (cid:0) 2 nsδ (cid:1)(cid:109) then excess neurons can be masked to yield the desired value for k .
Revised: Note that if k > ns · (cid:108) 16 √ nsε + 16 log (cid:0) 2 nsδ (cid:1)(cid:109) then excess neurons can be masked to yield the desired value for k .

Original: As in the proof of Lemma 2, we can split U , ˜ M , B , and M into n submatrices, denoted
Revised: As in the proof of Lemma 2, we can split U , ˜ M , B , and M into n submatrices, denoted

Original: U ( i ) ∈ {− 1 , +1 } k (cid:48) × n , ˜ M
Revised: U ( i ) ∈ {− 1 , +1 } k (cid:48) × n , ˜ M

Original: ( i ) ∈ {− 1 , +1 } k (cid:48)
Revised: ( i ) ∈ {− 1 , +1 } k (cid:48)

Original: × n , B ( i ) ∈ {− 1 , +1 } k (cid:48) × d , and M ( i ) ∈ {− 1 , +1 } k (cid:48)
Revised: × n , B ( i ) ∈ {− 1 , +1 } k (cid:48) × d , and M ( i ) ∈ {− 1 , +1 } k (cid:48)

Original: × d for i ∈ [ n ] , such that
Revised: × d for i ∈ [ n ] , such that

Original: [Equation]
Revised: [Equation]

Original: To simplify notation in the following deﬁnition, we deﬁne the vectors ˜ m
Revised: To simplify notation in the following deﬁnition, we deﬁne the vectors ˜ m

Original: ( i ) :=
Revised: ( i ) :=

Original: ˜ M ( i ) : ,i and ˜ u ( i ) :
Revised: ˜ M ( i ) : ,i and ˜ u ( i ) :

Original: = ˜
Revised: = ˜

Original: U
Revised: U

Original: ( i ) : ,i .
Revised: ( i ) : ,i .

Original: Now we deﬁne the functions
Revised: Now we deﬁne the functions

Original: g i : R d → R by
Revised: g i : R d → R by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for each i ∈ [ n ] .
Revised: for each i ∈ [ n ] .

Original: Taking ε (cid:48)
Revised: Taking ε (cid:48)

Original: = ε √ n and δ (cid:48)
Revised: = ε √ n and δ (cid:48)

Original: =
Revised: =

Original: δn , it follows from (37) that k (cid:48) ≥ s · (cid:108) 16 √ s ε (cid:48)
Revised: δn , it follows from (37) that k (cid:48) ≥ s · (cid:108) 16 √ s ε (cid:48)

Original: + 16 log (cid:0) 2 sδ (cid:48) (cid:1)(cid:109) .
Revised: + 16 log (cid:0) 2 sδ (cid:48) (cid:1)(cid:109) .

Original: As the hypotheses of Lemma 2 are satisﬁed, with probability at least 1 − δn there exist masks ˜ m
Revised: As the hypotheses of Lemma 2 are satisﬁed, with probability at least 1 − δn there exist masks ˜ m

Original: ( i ) and M ( i ) with
Revised: ( i ) and M ( i ) with

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: For each i ∈ [ n ] , note that this results in choosing the columns of the mask ˜ M ( i ) by
Revised: For each i ∈ [ n ] , note that this results in choosing the columns of the mask ˜ M ( i ) by

Original: [Equation]
Revised: [Equation]

Original: : otherwise
Revised: : otherwise

Original: [Equation]
Revised: [Equation]

Original: Combining this choice with (40) yields
Revised: Combining this choice with (40) yields

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By the deﬁnition of G ( x ) in (38), it follows from (45) that
Revised: By the deﬁnition of G ( x ) in (38), it follows from (45) that

Original: [Equation]
Revised: [Equation]

Original: Combining (43) and (46), we have with probability at least 1 − δ that
Revised: Combining (43) and (46), we have with probability at least 1 − δ that

Original: [Equation]
Revised: [Equation]

Original: Finally, it follows from (42) and (44) that
Revised: Finally, it follows from (42) and (44) that

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: We are now ready to prove the main result in Theorem 2.
Revised: We are now ready to prove the main result in Theorem 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let (cid:96), n, s ∈ N , W (1) ∗ ∈
Revised: Let (cid:96), n, s ∈ N , W (1) ∗ ∈

Original: (cid:104)
Revised: (cid:104)

Original: − 1 √ s , 1 √ s (cid:105) d × n , { W ( i ) ∗ } (cid:96) − 1 i =2 ∈ (cid:104) − 1 √ n , 1 √ n (cid:105) n × n , and W ( (cid:96) )
Revised: − 1 √ s , 1 √ s (cid:105) d × n , { W ( i ) ∗ } (cid:96) − 1 i =2 ∈ (cid:104) − 1 √ n , 1 √ n (cid:105) n × n , and W ( (cid:96) )

Original: ∗ ∈ (cid:104) − 1 √ n , 1 √ n (cid:105) 1 × n .
Revised: ∗ ∈ (cid:104) − 1 √ n , 1 √ n (cid:105) 1 × n .

Original: Assume that for each i ∈ [ (cid:96) ] we have (cid:107) W ( i ) ∗ (cid:107) 2 ≤ 1 and max j (cid:107) W
Revised: Assume that for each i ∈ [ (cid:96) ] we have (cid:107) W ( i ) ∗ (cid:107) 2 ≤ 1 and max j (cid:107) W

Original: ( i ) ∗ j (cid:107) 0 ≤ s .
Revised: ( i ) ∗ j (cid:107) 0 ≤ s .

Original: Deﬁne F ( x ) :
Revised: Deﬁne F ( x ) :

Original: = F ( (cid:96) ) ◦ · · · ◦ F (1) ( x ) where F ( i ) ( x ) = σ ( W ( i ) ∗ x ) for i ∈ [ (cid:96) − 1] and F ( (cid:96) ) ( x ) =
Revised: = F ( (cid:96) ) ◦ · · · ◦ F (1) ( x ) where F ( i ) ( x ) = σ ( W ( i ) ∗ x ) for i ∈ [ (cid:96) − 1] and F ( (cid:96) ) ( x ) =

Original: W ( (cid:96) ) ∗ x .
Revised: W ( (cid:96) ) ∗ x .

Original: Fix ε, δ ∈ (0 , 1) .
Revised: Fix ε, δ ∈ (0 , 1) .

Original: Let B (1) ∈ {− 1 , +1 } k × d be sampled from Bin ( {− 1 , +1 } k × d ) , { B ( i ) } (cid:96)i =2 ∈ {− 1 , +1 } k × n be sampled from Bin ( {− 1 , +1 } k × n ) , { U ( i ) } (cid:96) − 1 i = ∈ {− 1 , +1 } k × n be sampled from
Revised: Let B (1) ∈ {− 1 , +1 } k × d be sampled from Bin ( {− 1 , +1 } k × d ) , { B ( i ) } (cid:96)i =2 ∈ {− 1 , +1 } k × n be sampled from Bin ( {− 1 , +1 } k × n ) , { U ( i ) } (cid:96) − 1 i = ∈ {− 1 , +1 } k × n be sampled from

Original: Bin ( {− 1 , +1 } k × n ) and U ( (cid:96) ) ∈ {− 1 , +1 } k × 1 sampled from Bin ( {− 1 , +1 } k × 1 ) .
Revised: Bin ( {− 1 , +1 } k × n ) and U ( (cid:96) ) ∈ {− 1 , +1 } k × 1 sampled from Bin ( {− 1 , +1 } k × 1 ) .

Original: If k ≥ ns · (cid:24) 32 (cid:96) √ ns ε + 16 log (cid:18) 2 ns(cid:96) δ (cid:19)(cid:25) , (49) then with probability at least 1 − δ there exist binary masks { M ( i ) } (cid:96)i =1 and { ˜ M ( i ) } (cid:96)i =1 for { B ( i ) } (cid:96)i =1 and { U ( i ) }
Revised: If k ≥ ns · (cid:24) 32 (cid:96) √ ns ε + 16 log (cid:18) 2 ns(cid:96) δ (cid:19)(cid:25) , (49) then with probability at least 1 − δ there exist binary masks { M ( i ) } (cid:96)i =1 and { ˜ M ( i ) } (cid:96)i =1 for { B ( i ) } (cid:96)i =1 and { U ( i ) }

Original: (cid:96)i =1 , respectively, such that the function G : R
Revised: (cid:96)i =1 , respectively, such that the function G : R

Original: d
Revised: d

Original: → R deﬁned by
Revised: → R deﬁned by

Original: [Equation]
Revised: [Equation]

Original: 50) where
Revised: 50) where

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Additionally, (cid:107) ˜
Revised: Additionally, (cid:107) ˜

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let i ∈ [ (cid:96) − 1] .
Revised: Let i ∈ [ (cid:96) − 1] .

Original: Using Lemma 3 with ε (cid:48)
Revised: Using Lemma 3 with ε (cid:48)

Original: =
Revised: =

Original: ε 2 (cid:96) and δ (cid:48) = δ(cid:96) , with probability at least 1 − δ(cid:96) there exist M ( i ) and ˜ M ( i ) such that
Revised: ε 2 (cid:96) and δ (cid:48) = δ(cid:96) , with probability at least 1 − δ(cid:96) there exist M ( i ) and ˜ M ( i ) such that

Original: [Equation]
Revised: [Equation]

Original: , for all
Revised: , for all

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The remainder of the proof follows from applying the same argument as in the proof of Theorem A. from (Malach et al., 2020).
Revised: The remainder of the proof follows from applying the same argument as in the proof of Theorem A. from (Malach et al., 2020).

Original: C M OTIVATION FOR F RAMEWORK TO I DENTIFY MPT S
Revised: C M OTIVATION FOR F RAMEWORK TO I DENTIFY MPT S

Original: Suppose that f ( x ; W ∗ ) with optimized weights W ∗ is a target network that we wish to approximate.
Revised: Suppose that f ( x ; W ∗ ) with optimized weights W ∗ is a target network that we wish to approximate.

Original: Let g ( x ; W ) denote the network in which we want to identify a MPT-1/32 that is an εapproximation of f ( x ; W ∗ ) , for some ε > 0 .
Revised: Let g ( x ; W ) denote the network in which we want to identify a MPT-1/32 that is an εapproximation of f ( x ; W ∗ ) , for some ε > 0 .

Original: Now assume that g ( x ; · ) is Lipschitz continuous with constant κ , B ∈ {− 1 , +1 } m are binary parameters for g , and α ∈ R is gain term.
Revised: Now assume that g ( x ; · ) is Lipschitz continuous with constant κ , B ∈ {− 1 , +1 } m are binary parameters for g , and α ∈ R is gain term.

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: If we take M to be a ﬁxed binary mask, we can minimize the error of binarizing the subnetwork parameters M (cid:12) W by solving the optimization problem
Revised: If we take M to be a ﬁxed binary mask, we can minimize the error of binarizing the subnetwork parameters M (cid:12) W by solving the optimization problem

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where M , W , and B are stacked into vectors of some length, say n .
Revised: where M , W , and B are stacked into vectors of some length, say n .

Original: As the pruning mask M is applied to both W and B , solving problem (57) is equivalent to solving problem (2) in (Rastegari et al., 2016) with a different dimension.
Revised: As the pruning mask M is applied to both W and B , solving problem (57) is equivalent to solving problem (2) in (Rastegari et al., 2016) with a different dimension.

Original: Hence, it immediately follows that one closed form solution for B in problem (57) is
Revised: Hence, it immediately follows that one closed form solution for B in problem (57) is

Original: B ∗ = sign( W ) .
Revised: [Equation]

Original: Taking the derivative of the cost function in (57) with respect to α and setting it equal to zero yields
Revised: Taking the derivative of the cost function in (57) with respect to α and setting it equal to zero yields

Original: [Equation]
Revised: [Equation]

Original: Recalling that M ∈ { 0 , 1 } n and using (58), we have
Revised: Recalling that M ∈ { 0 , 1 } n and using (58) [Table] we have

Original: [Equation]
Revised: [Equation]

Original: B ∗ )
Revised: ∗ )

Original: =
Revised: =

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: ( M i B ∗ i ) 2 = n (cid:88) i =
Revised: ( M i B ∗ i ) 2 = n (cid:88) i =

Original: M 2 i (sign( W i )) 2 = n (cid:88) i =
Revised: M 2 i (sign( W i )) 2 = n (cid:88) i =

Original: M i
Revised: M i

Original: = (cid:107) M (cid:107)(60) and ( M (cid:12) W ) (cid:124)
Revised: = (cid:107) M (cid:107)(60) and ( M (cid:12) W ) (cid:124)

Original: ( M (cid:12) B ∗ )
Revised: ( M (cid:12) B ∗ )

Original: = n (cid:88) i =
Revised: = n (cid:88) i =

Original: M 2 i W i sign( W i ) =
Revised: M 2 i W i sign( W i ) =

Original: n (cid:88) i =
Revised: n (cid:88) i =

Original: M i | W i | = (cid:107) M (cid:12) W (cid:107) 1 .
Revised: M i | W i | = (cid:107) M (cid:12) W (cid:107) 1 .

Original: Substituting (60) and (61) into (59) and solving for α yields the closed form solution
Revised: Substituting (60) and (61) into (59) and solving for α yields the closed form solution

Original: [Equation]
Revised: [Equation]

Original: [Equation] Hence, α ∗ and B ∗ minimize the right hand side of (56) and, consequently, reduce the approximation error of the MPT-1/32.
Revised: Hence, α ∗ and B ∗ minimize the right hand side of (56) and, consequently, reduce the approximation error of the MPT-1/32.

Original: So when the binarization error, (cid:107) ( M (cid:12) W ) − α ( M (cid:12) sign( W )) (cid:107) , and the subnetwork error, (cid:107) g
Revised: So when the binarization error, (cid:107) ( M (cid:12) W ) − α ( M (cid:12) sign( W )) (cid:107) , and the subnetwork error, (cid:107) g

Original: ( x ; M (cid:12) W )
Revised: ( x ; M (cid:12) W )

Original: − f ( x ; W ∗ ) (cid:107) , are sufﬁciently small then the binarized subnetwork g ( x ; α ( M (cid:12) sign( W ))) serves as a good approximation to the target network.
Revised: − f ( x ; W ∗ ) (cid:107) , are sufﬁciently small then the binarized subnetwork g ( x ; α ( M (cid:12) sign( W ))) serves as a good approximation to the target network.

Original: These closed form expressions for the gain term and the binarized weights are the updates used for the gain term and binary subnetwork weights in biprop after updating the binary pruning mask.
Revised: These closed form expressions for the gain term and the binarized weights are the updates used for the gain term and binary subnetwork weights in biprop after updating the binary pruning mask.

Original: D C OMPARISON OF MPT S WITH BINARY NEURAL NETWORK SOTA
Revised: D C OMPARISON OF MPT S WITH BINARY NEURAL NETWORK SOTA

Original: Here we provide a more exhaustive comparison of MPT–1/32 and MPT–1/1 on CIFAR-10 and ImageNet to SOTA methods – BinaryConnect (Courbariaux et al., 2015), BNN (Courbariaux et al., 2016), DoReFa-Net (Zhou et al., 2016), LQ-Nets (Zhang et al., 2018), BWN and XNOR-Net (Rastegari et al., 2016), ABC-Net (Lin et al., 2017), IR-Net (Qin et al., 2020b), LAB (Hou et al., 2016),
Revised: Here we provide a more exhaustive comparison of MPT–1/32 and MPT–1/1 on CIFAR-10 and ImageNet to SOTA methods – BinaryConnect (Courbariaux et al., 2015), BNN (Courbariaux et al., 2016), DoReFa-Net (Zhou et al., 2016), LQ-Nets (Zhang et al., 2018), BWN and XNOR-Net (Rastegari et al., 2016), ABC-Net (Lin et al., 2017), IR-Net (Qin et al., 2020b), LAB (Hou et al., 2016),

Original: ProxQuant (Bai et al., 2018), DSQ (Gong et al., 2019), and BBG (Shen et al., 2020).
Revised: ProxQuant (Bai et al., 2018), DSQ (Gong et al., 2019), and BBG (Shen et al., 2020).

Original: Results for
Revised: Results for

Original: CIFAR-10 can be found in Tables 5 and 6 and results for ImageNet can be found in Tables 7 and 8.
Revised: CIFAR-10 can be found in Tables 5 and 6 and results for ImageNet can be found in Tables 7 and 8.

Original: Next to the MPT method we include the layer width multiplier (if larger than 1) and the percentage of weights pruned in parentheses.
Revised: Next to the MPT method we include the layer width multiplier (if larger than 1) and the percentage of weights pruned in parentheses.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: E R ELATED W ORK E. P RUNING
Revised: E R ELATED W ORK E. P RUNING

Original: We categorize pruning methods based on whether a model is pruned either after the training or before the training (see (Neill, 2020) for a comprehensive review).
Revised: We categorize pruning methods based on whether a model is pruned either after the training or before the training (see (Neill, 2020) for a comprehensive review).

Original: Post-Training Pruning.
Revised: Post-Training Pruning.

Original: The traditional pruning methods leverage a three-stage pipeline – pretraining (a large model), pruning [Table] and ﬁne-tuning.
Revised: The traditional pruning methods leverage a three-stage pipeline – pretraining (a large model), pruning, and ﬁne-tuning.

Original: The main distinction lies among these approaches is what type of criteria is used for pruning.
Revised: The main distinction lies among these approaches is what type of criteria is used for pruning.

Original: One of the most popular approach is the magnitude-based pruning where the weights with the magnitude below a certain threshold are discarded (Hagiwara, 1993).
Revised: One of the most popular approach is the magnitude-based pruning where the weights with the magnitude below a certain threshold are discarded (Hagiwara, 1993).

Original: Further, certain penalty term (e [Table] g., l 1 , l 2 or lasso weight regularization) can be used during training to encourage a model to learn certain smaller magnitude weights and removing them posttraining (Weigend et al., 1991).
Revised: Further, certain penalty term (e.g., l 1 , l 2 or lasso weight regularization) can be used during training to encourage a model to learn certain smaller magnitude weights and removing them posttraining (Weigend et al., 1991).

Original: Models can also be pruned by measuring the importance of weights by computing the sensitivity of the loss function when weights are removed and prune those which cause the smallest change in the loss (LeCun et al., 1990).
Revised: Models can also be pruned by measuring the importance of weights by computing the sensitivity of the loss function when weights are removed and prune those which cause the smallest change in the loss (LeCun et al., 1990).

Original: Pruning Before Training.
Revised: Pruning Before Training.

Original: Thus far, we have have discussed methods for pruning pretrained
Revised: Thus far, we have have discussed methods for pruning pretrained

Original: DNNs.
Revised: DNNs.

Original: Recently, (Frankle & Carbin, 2018) proposed the Lottery Ticket Hypothesis and showed that randomly-initialized neural networks contain sparse subnetworks that can be effectively trained from scratch when reset to their initialization.
Revised: Recently, (Frankle & Carbin, 2018) proposed the Lottery Ticket Hypothesis and showed that randomly-initialized neural networks contain sparse subnetworks that can be effectively trained from scratch when reset to their initialization.

Original: Further, (Liu et al., 2018b) showed that the training an over-parameterized model is often not necessary to obtain an efﬁcient ﬁnal model and network architecture itself is more important than the remaining weights after pruning pretrained networks.
Revised: Further, (Liu et al., 2018b) showed that the training an over-parameterized model is often not necessary to obtain an efﬁcient ﬁnal model and network architecture itself is more important than the remaining weights after pruning pretrained networks.

Original: These ﬁndings has revived interest in ﬁnding approaches for searching sparse and trainable subnetworks.
Revised: These ﬁndings has revived interest in ﬁnding approaches for searching sparse and trainable subnetworks.

Original: For example, (Lee et al., 2018; Wang et al., 2020b; You et al., 2019; Wang et al., 2020a) explored efﬁcient approaches to search for these sparse and trainable subnetworks.
Revised: For example, (Lee et al., 2018; Wang et al., 2020b; You et al., 2019; Wang et al., 2020a) explored efﬁcient approaches to search for these sparse and trainable subnetworks.

Original: Along this line of work, a striking ﬁnding was reported by (Zhou et al., 2019; Ramanujan et al., 2020) showing that randomly-initialized neural networks contain sparse subnetworks that achieve good performance without any training.
Revised: Along this line of work, a striking ﬁnding was reported by (Zhou et al., 2019; Ramanujan et al., 2020) showing that randomly-initialized neural networks contain sparse subnetworks that achieve good performance without any training.

Original: (Malach et al., 2020; Pensia et al., 2020) provided theoretical evidences for this phenomenon and showed that one can approximate any target neural network, by pruning a sufﬁciently over-parameterized network of random weights.
Revised: (Malach et al., 2020; Pensia et al., 2020) provided theoretical evidences for this phenomenon and showed that one can approximate any target neural network, by pruning a sufﬁciently over-parameterized network of random weights.

Original: E. B INARIZATION
Revised: E. B INARIZATION

Original: Similar to pruning, we categorize binarization methods based on whether a model is binarized either after the training or during the training (see (Qin et al., 2020a) for a comprehensive review).
Revised: Similar to pruning, we categorize binarization methods based on whether a model is binarized either after the training or during the training (see (Qin et al., 2020a) for a comprehensive review).

Original: Post-Training Binarization.
Revised: Post-Training Binarization.

Original: To the best of our knowledge, none of the post-training schemes have been successful in binarizing pretrained models with or without retraining to achieve reasonable test accuracy.
Revised: To the best of our knowledge, none of the post-training schemes have been successful in binarizing pretrained models with or without retraining to achieve reasonable test accuracy.

Original: Most existing works (Han et al., 2015; Zhou et al., 2017) are limited to ternary weight quantization.
Revised: Most existing works (Han et al., 2015; Zhou et al., 2017) are limited to ternary weight quantization.

Original: Training
Revised: Training

Original: -Aware Binarization
Revised: -Aware Binarization

Original: There are several efforts to improve the performance of BNN training.
Revised: There are several efforts to improve the performance of BNN training.

Original: This is a challenging problem as binarization introduces discontinuities which makes differentiation during backpropogation difﬁcult.
Revised: This is a challenging problem as binarization introduces discontinuities which makes differentiation during backpropogation difﬁcult.

Original: Binaryconnect (Courbariaux et al., 2015) established how to train networks with binary weights within the familiar back-propagation paradigm.
Revised: Binaryconnect (Courbariaux et al., 2015) established how to train networks with binary weights within the familiar back-propagation paradigm.

Original: BinaryNet (Courbariaux et al., 2016) further quantize both the weights and the activations to 1-bit values.
Revised: BinaryNet (Courbariaux et al., 2016) further quantize both the weights and the activations to 1-bit values.

Original: Unfortunately, these early schemes resulted in a staggering drop in the accuracy compared to their full precision counterparts.
Revised: Unfortunately, these early schemes resulted in a staggering drop in the accuracy compared to their full precision counterparts.

Original: In an attempt to improve the performance, XNOR-Net (Rastegari et al., 2016) proposed to add a real-valued channel-wise scaling factor.
Revised: In an attempt to improve the performance, XNOR-Net (Rastegari et al., 2016) proposed to add a real-valued channel-wise scaling factor.

Original: Dorefa-Net
Revised: Dorefa-Net

Original: (Zhou et al., 2016) extends XNOR-Net to accelerate the training process using quantized gradients.
Revised: (Zhou et al., 2016) extends XNOR-Net to accelerate the training process using quantized gradients.

Original: ABC-Net (Lin et al., 2017) improved the performance by using more weight bases and activation bases at the cost of increase in memory and computation.
Revised: ABC-Net (Lin et al., 2017) improved the performance by using more weight bases and activation bases at the cost of increase in memory and computation.

Original: There have also been efforts in making modiﬁcations to the network architectures to make them amenable for the binary neural network training.
Revised: There have also been efforts in making modiﬁcations to the network architectures to make them amenable for the binary neural network training.

Original: For example,
Revised: For example,

Original: Bireal-Net (Liu et al., 2018a)
Revised: Bireal-Net (Liu et al., 2018a)

Original: added layer-wise identity short-cut, and AutoBNN (Shen et al., 2020) proposed to widen or squeeze the channels in an automatic manner.
Revised: added layer-wise identity short-cut, and AutoBNN (Shen et al., 2020) proposed to widen or squeeze the channels in an automatic manner.

Original: (Han et al., 2020) proposed to learn to binarize neurons with noisy supervision.
Revised: (Han et al., 2020) proposed to learn to binarize neurons with noisy supervision.

Original: Some efforts also have been carried out to designing gradient estimators extending straight-through estimator (STE) (Bengio et al., 2013) for accurate gradient back-propagation.
Revised: Some efforts also have been carried out to designing gradient estimators extending straight-through estimator (STE) (Bengio et al., 2013) for accurate gradient back-propagation.

Original: DSQ (Gong et al., 2019) used differentiable soft quantization to have accurate gradients in backward propagation.
Revised: DSQ (Gong et al., 2019) used differentiable soft quantization to have accurate gradients in backward propagation.

Original: On the other hand, PCNN Gu et al.
Revised: On the other hand, PCNN Gu et al.

Original: (2019) proposed a new discrete back-propagation via projection algorithm to build BNNs.
Revised: (2019) proposed a new discrete back-propagation via projection algorithm to build BNNs.

Original: E. O THER R ELATED D IRECTIONS
Revised: E. O THER R ELATED D IRECTIONS

Original: Gaier & Ha (2019) proposed a search method for neural network architectures that can already perform a task without any explicit weight training, i.e., each weight in the network has the same shared value.
Revised: Gaier & Ha (2019) proposed a search method for neural network architectures that can already perform a task without any explicit weight training, i.e., each weight in the network has the same shared value.

Original: Recent work in randomly wired neural networks (Xie et al., 2019) showed that constructing neural networks with random graph algorithms often outperforms a manually engineered architecture.
Revised: Recent work in randomly wired neural networks (Xie et al., 2019) showed that constructing neural networks with random graph algorithms often outperforms a manually engineered architecture.

Original: As opposed to ﬁxed wirings in (Xie et al., 2019), (Wortsman et al., 2019) learned the network parameters as well as the structure.
Revised: As opposed to ﬁxed wirings in (Xie et al., 2019), (Wortsman et al., 2019) learned the network parameters as well as the structure.

Original: This show that ﬁnding a good architecture is akin to ﬁnding a sparse subnetwork of the complete graph.
Revised: This show that ﬁnding a good architecture is akin to ﬁnding a sparse subnetwork of the complete graph.