Original: B LOCK S WAP : F ISHER - GUIDED B LOCK S UBSTITUTION FOR N ETWORK C OMPRESSION ON A B UDGET
Revised: B LOCK S WAP : F ISHER - GUIDED B LOCK S UBSTITUTION FOR N ETWORK C OMPRESSION ON A B UDGET

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks.
Revised: The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks.

Original: However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive.
Revised: However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive.

Original: In this work, we develop BlockSwap : a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential .
Revised: In this work, we develop BlockSwap : a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential .

Original: These networks can then be used as students and distilled with the original large network as a teacher.
Revised: These networks can then be used as students and distilled with the original large network as a teacher.

Original: We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classiﬁcation, and COCO for detection, and provide a comprehensive ablation study of our approach.
Revised: We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classiﬁcation, and COCO for detection, and provide a comprehensive ablation study of our approach.

Original: BlockSwap quickly explores possible block conﬁgurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. 8 minutes on a single CPU for CIFAR-10). 
Revised: BlockSwap quickly explores possible block conﬁgurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. under 5 minutes on a single GPU for CIFAR-10). Code is available at https://github.com/BayesWatch/pytorch-blockswap .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original:  Convolutional Neural Networks(CNNs) are extremely popular, and demonstrate strong performance on a variety of challenging tasks.
Revised: Deep Convolutional Neural Networks are extremely popular, and demonstrate strong performance on a variety of challenging tasks.

Original: Because of this, there exists a large range of scenarios in the wild in which practitioners wish to deploy these networks e.g. pedestrian detection in a vehicle’s computer, human activity recognition with wearables (Radu et al., 2018).
Revised: Because of this, there exist a large range of scenarios in the wild in which practitioners wish to deploy these networks e.g. pedestrian detection in a vehicle’s computer, human activity recognition with wearables (Radu et al., 2018).

Original: CNNs are largely over-parameterised; a fact that can be exploited when specialising networks for different resource budgets.
Revised: These networks are largely over-parameterised; a fact that can be exploited when specialising networks for different resource budgets.

Original: For example, there is a wealth of work demonstrating that it is possible to replace expensive convolutional blocks in a large network with cheap alternatives e.g. those using grouped convolutions (Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018) or bottleneck structures (He et al., 2016; Sandler et al., 2018; Peng et al., 2018).
Revised: For example, there is a wealth of work demonstrating that it is possible to replace expensive convolutional blocks in a large network with cheap alternatives e.g. those using grouped convolutions (Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018) or bottleneck structures (He et al., 2016; Sandler et al., 2018; Peng et al., 2018).

Original: This creates a smaller network which may be used as a student , and trained through distillation (Ba & Caruana, 2014;
Revised: This creates a smaller network which may be used as a student , and trained through distillation (Ba & Caruana, 2014;

Original: Hinton et al., 2015) with the original large network as a teacher to retain performance.
Revised: Hinton et al., 2015) with the original large network as a teacher to retain performance.

Original: Typically, each network block is replaced with a single cheap alternative, producing a single-blocktype network for a given budget.
Revised: Typically, each network block is replaced with a single cheap alternative, producing a single-blocktype network for a given budget.

Original: By cheapening each block equally, one relies on the assumption that each block is of equal importance.
Revised: By cheapening each block equally, one relies on the assumption that each block is of equal importance.

Original: We posit instead that for each budget, there exist more powerful mixed-blocktype networks that assign non-uniform importance to each block by cheapening them to different extents.
Revised: We posit instead that for each budget, there exist more powerful mixed-blocktype networks that assign non-uniform importance to each block by cheapening them to different extents.

Original: We now have a paradigm of choice; for a given budget, it is not obvious which cheap alternatives to use for our student network, nor where to place them.
Revised: We now have a paradox of choice; for a given budget, it is not obvious which cheap alternatives to use for our student network, nor where to place them.

Original: Let’s assume we have a large network that consists of B expensive convolutional blocks, and a candidate pool of C cheap blocks that we could substitute each of these out for, and we are given a limit on the number of parameters (and therefore memory) that the network can use. 
Revised: Let’s assume we have a large network that consists of B expensive convolutional blocks, and a candidate pool of C cheap blocks that we could substitute each of these out for, and we are given a limit on the number of parameters (and therefore memory) that the network can use. Which of [Figure]

Original: Which of the C B mixed-blocktype networks is appropriate?
Revised: [Figure] the C B mixed-blocktype networks is appropriate?

Original: It rapidly becomes infeasible to exhaustively consider these networks, even for single digit C .
Revised: It rapidly becomes infeasible to exhaustively consider these networks, even for single digit C .

Original: We could turn to neural architecture search (NAS) techniques (Zoph & Le, 2017;  Pham et al., 2018;
Revised: We could turn to neural architecture search (NAS) techniques (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018;

Original: Tan et al., 2019; Wu et al., 2019; Liu et al., 2019a) but these introduce a gigantic computational overhead; even one of the fastest such approaches (Pham et al., 2018) still requires half a day of search time on a GPU.
Revised: Tan et al., 2019; Wu et al., 2019; Liu et al., 2019a) but these introduce a gigantic computational overhead; even one of the fastest such approaches (Pham et al., 2018) still requires half a day of search time on a GPU.

Original: Moreover, in real-world scenarios, allocated budgets can change quickly; we would like to be able to ﬁnd a good network ina matter of minutes, not hours or days.[Figure]
Revised: Moreover, in real-world scenarios, allocated budgets can change quickly; we would like to be able to ﬁnd a good network in of , not hours or days.

Original: Our goal in this paper is, given a desired parameter budget, to quickly identify a suitable mixedblocktype version of the original network that makes for a powerful student.
Revised: Our goal in this paper is, given a desired parameter budget, to quickly identify a suitable mixedblocktype version of the original network that makes for a powerful student.

Original: We present a simple method— BlockSwap —to achieve this.
Revised: We present a simple method— BlockSwap —to achieve this.

Original: First, we randomly sample a collection of candidate mixedblocktype architectures that satisfy a desired parameter budget.
Revised: First, we randomly sample a collection of candidate mixedblocktype architectures that satisfy the parameter budget.

Original: A single minibatch is then pushed through each candidate network to calculate its Fisher potential : the sum of the total  Fisher information for each of its blocks.
Revised: A single minibatch is then pushed through each candidate network to calculate its Fisher potential : the sum of the total (empirical) Fisher information for each of its blocks.

Original: Finally, the network with the highest potential is selected as a student and trained through the distillation method of attention transfer (Zagoruyko & Komodakis, 2017) with the original teacher.
Revised: Finally, the network with the highest potential is selected as a student and trained through the distillation method of attention transfer (Zagoruyko & Komodakis, 2017) with the original teacher.

Original: Our method is illustrated in Figure 1.
Revised: Our method is illustrated in Figure 1.

Original: In Section 3 we describe the block substitutions used in BlockSwap, distillation via attention transfer, and Fisher information.
Revised: In Section 3 we describe the block substitutions used in BlockSwap, distillation via attention transfer, and Fisher information.

Original: We describe our method in Section 4 as well as providing a comprehensive ablation study.
Revised: We elaborate on our method in Section 4 as well as providing a comprehensive ablation study.

Original: Finally, we experimentally verify the potency of BlockSwap on CIFAR-10 (Section 5) as well as ImageNet (Section 6) and COCO (Section 7).
Revised: Finally, we experimentally verify the potency of BlockSwap on CIFAR-10 (Section 5) as well as ImageNet (Section 6) and COCO (Section 7).

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: We introduce BlockSwap, an algorithm for reducing large neural networks by performing block-wise substitution.
Revised: We introduce BlockSwap, an algorithm for reducing large neural networks by performing block-wise substitution.

Original: We show that this outperforms other top-down approaches such as depth/width scaling, parameter pruning, and random substitution.
Revised: We show that this outperforms other top-down approaches such as depth/width scaling, parameter pruning, and random substitution.

Original: We outline a simple method for quickly evaluating candidate models via Fisher information, which matches the performance of bottom-up approaches while reducing search time from days to minutes.
Revised: We outline a simple method for quickly evaluating candidate models via Fisher information, which matches the performance of bottom-up approaches while reducing search time from days to minutes.

Original: We conduct ablation studies to validate our  ranking metric is highly correlated to  ﬁnal error.
Revised: We conduct ablation studies to validate our methodology, highlighting the beneﬁts of block mixing, and conﬁrming that our ranking metric is highly correlated to the ﬁnal error.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural networks tend to be overparameterised: Denil et al.
Revised: Neural networks tend to be overparameterised: Denil et al.

Original: (2013) accurately predict most of the weights in a network from a small subset; Frankle & Carbin (2019) hypothesise that within a large network, there exists a fortuitously initialised subnetwork that drives its performance.
Revised: (2013) accurately predict most of the weights in a network from a small subset; Frankle & Carbin (2019) hypothesise that within a large network, there exists a fortuitously initialised subnetwork that drives its performance.

Original: However, it remains difﬁcult to exploit this overparameterisation without taking a hit in performance.
Revised: However, it remains difﬁcult to exploit this overparameterisation without taking a hit in performance.

Original: One means to combat this is to use a large teacher network to regularise the training of a small student network; a process known as distillation.
Revised: One means to combat this is to use a large teacher network to regularise the training of a small student network; a process known as distillation.

Original: The small network is trained from scratch, but is also forced to match the outputs (Ba & Caruana, 2014; Hinton et al., 2015) or activation statistics (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Kim et al., 2018) of the teacher using an additional loss term.
Revised: The small network is trained from scratch, but is also forced to match the outputs (Ba & Caruana, 2014; Hinton et al., 2015) or activation statistics (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Kim et al., 2018) of the teacher using an additional loss term.

Original: When utilising distillation one must decide how to create a student network.
Revised: When utilising distillation one must decide how to create a student network.

Original: A simple approach would be to reduce the depth of the original large network, although this can prove detrimental (Urban et al., 2017).
Revised: A simple approach would be to reduce the depth of the original large network, although this can prove detrimental (Urban et al., 2017).

Original: An effective strategy is to create a student by replacing all the teacher’s convolutions with grouped alternatives (Crowley et al., 2018).
Revised: An effective strategy is to create a student by replacing all the teacher’s convolutions with grouped alternatives (Crowley et al., 2018a).

Original: Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying a much cheaper convolution on each split.
Revised: Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying a much cheaper convolution on each split.

Original: They were originally used in AlexNet (Krizhevsky et al., 2012) due to GPU memory limitations, and have appeared in several subsequent architectures (Ioffe & Szegedy, 2015; Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018).
Revised: They were originally used in AlexNet (Krizhevsky et al., 2012) due to GPU memory limitations, and have appeared in several subsequent architectures (Ioffe & Szegedy, 2015; Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018).

Original: However, as the number of groups increases, fewer channels are mixed, which hinders representational capacity.
Revised: However, as the number of groups increases, fewer channels are mixed, which hinders representational capacity.

Original: MobileNet (Howard et al., 2017) compensates for this by following its heavily-grouped depthwise convolutions by a pointwise ( 1 × 1 ) convolution to allow for channel mixing.
Revised: MobileNet (Howard et al., 2017) compensates for this by following its heavily-grouped depthwise convolutions by a pointwise ( 1 × 1 ) convolution to allow for channel mixing.

Original: The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search (NAS).
Revised: The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search (NAS).

Original: Zoph & Le (2017) use an RNN to generate network  descriptions and ﬁlter the options using reinforcement learning.
Revised: Zoph et al. (2018) use an RNN to generate network block descriptions and ﬁlter the options using reinforcement learning.

Original: Storing such a large quantity of possible networks is expensive, and their evaluation strategy utilised 450 GPUs over the course of 3 days .
Revised: These blocks are stacked to form a full neural network. This is an extremely expensive process, utilising 450 GPUs over the course of 3 days .

Original: To address this, Pham et al.
Revised: To address this, Pham et al.

Original: (2018) propose giving all models access to a shared set of weights, achieving similar performance to Zoph & Le (2017) with a single GPU in less than 24 hours.
Revised: (2018) propose giving all models access to a shared set of weights, achieving similar performance to Zoph et al. (2018) with a single GPU in less than 24 hours.

Original: Subsequent works have made extensive use of this technique (Liu et al., 2019a;
Revised: Subsequent works have made extensive use of this technique (Liu et al., 2019a;

Original: Luo et al., 2018; Chen et al., 2018b).
Revised: Luo et al., 2018; Chen et al., 2018b).

Original: However, it has been observed that under the constrained architecture search space of the above methods, random architecture search provides a competitive baseline (Li & Talwalkar, 2019; Sciuto et al., 2019).
Revised: However, it has been observed that under the constrained architecture search space of the above methods, random architecture search provides a competitive baseline (Li & Talwalkar, 2019; Yu et al., 2020).

Original: In particular, Sciuto et al.
Revised: In particular, Yu et al.

Original: (2019) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to ﬁnd suboptimal architectures.
Revised: (2020) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to ﬁnd suboptimal architectures.

Original: NAS techniques predominantly take a bottom-up approach; they ﬁnd a powerful building block, and form neural networks using stacks of these blocks.
Revised: NAS techniques predominantly take a bottom-up approach; they ﬁnd a powerful building block, and form neural networks using stacks of these blocks.

Original: Other works have taken a top-down approach to ﬁnd architectures using pruning (Chen et al., 2018a; Lee et al., 2019; Liu et al., 2019b;  Frankle & Carbin, 2019).
Revised: Other works have taken a top-down approach to ﬁnd architectures using pruning (Chen et al., 2018a; Lee et al., 2019; Liu et al., 2019b; Crowley et al., 2018b; Frankle & Carbin, 2019).

Original: Chen et al.
Revised: Chen et al.

Original: (2018a) take a pre-trained network and apply a principled framework to compress it under operational constraints such as latency.
Revised: (2018a) take a pre-trained network and apply a principled framework to compress it under operational constraints such as latency.

Original: In SNIP (Lee et al., 2019), one randomly initialises a large network and quantiﬁes the sensitivity of each connection using gradient magnitudes.
Revised: In SNIP (Lee et al., 2019), one randomly initialises a large network and quantiﬁes the sensitivity of each connection using gradient magnitudes.

Original: The lowest sensitivity connections are removed to produce a sparse architecture which is then trained as normal.
Revised: The lowest sensitivity connections are removed to produce a sparse architecture which is then trained as normal.

Original:  P RELIMINARIES 3. S UBSTITUTE B LOCKS
Revised:  P RELIMINARIES 3. S UBSTITUTE B LOCKS

Original: Here, we will brieﬂy elaborate on the block substitutions used for BlockSwap .
Revised: Here, we will brieﬂy elaborate on the block substitutions used for BlockSwap .

Original: A tabular comparison is given in Appendix A.
Revised: A tabular comparison is given in Appendix A.

Original: The block choices are deliberately simplistic.
Revised: The block choices are deliberately simplistic.

Original: We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of a speciﬁc highly-engineered block e.g. one from the NAS literature.
Revised: We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of a speciﬁc highly-engineered block e.g. one from the NAS literature.

Original: Standard Block – S

Original: The blocks considered are variations of the standard block used in residual networks.
Revised: The blocks considered are variations of the standard block used in residual networks.

Original: In the majority of these blocks, the input has the same number of channels as the output, so we describe their parameter cost assuming this is the case.
Revised: In the majority of these blocks, the input has the same number of channels as the output, so we describe their parameter cost assuming this is the case.

Original: This standard block contains two convolutional layers, each using N lots of N × k × k ﬁlters where k is the kernel size.
Revised: This standard block contains two convolutional layers, each using N lots of N × k × k ﬁlters where k is the kernel size.

Original: Assuming the costs of batch-norm (BN) layers and shortcut convolutions (where applicable) are negligible, the block uses a total of 2 N 2 k 2 parameters.
Revised: Assuming the costs of batch-norm (BN) layers and shortcut convolutions (where applicable) are negligible, the block uses a total of 2 N 2 kparameters.

Original: Grouped+Pointwise Block – G(g)
Revised: Grouped+Pointwise Block – G(g)

Original: In a grouped convolution, the N channel input is split along the channel dimension into g groups, each of which has N/g channels.
Revised: In a grouped convolution, the N channel input is split along the channel dimension into g groups, each of which has N/g channels.

Original: Each group goes through its own convolution which outputs N/g channels, and all the outputs are concatenated along the channel dimension.
Revised: Each group goes through its own convolution which outputs N/g channels, and all the outputs are concatenated along the channel dimension.

Original: This uses g × ( N/g ) 2 k 2 =
Revised: This uses g × ( N/g ) 2 k 2 =

Original: ( N 2 k 2 ) /g parameters.
Revised: ( N 2 k 2 ) /g parameters.

Original: To compensate for reduced channel mixing, each convolution is further followed by a pointwise ( 1 × 1 ) convolution, incurring an extra cost of N 2 .
Revised: To compensate for reduced channel mixing, each convolution is further followed by a pointwise ( 1 × 1 ) convolution, incurring an extra cost of N 2 .

Original: For this block, each full convolution has been replaced with a grouped+pointwise convolution, and so the block uses 2(( N 2 k 2 ) /g
Revised: For this block, each full convolution has been replaced with a grouped+pointwise convolution, and so the block uses 2(( N 2 k 2 ) /g

Original: + N 2 ) parameters.
Revised: + N 2 ) parameters.

Original: Bottleneck Block – B(b)
Revised: Bottleneck Block – B(b)

Original: In this block, a pointwise convolution is used to reduce the number of channels of the input by a factor of b before a standard convolution is applied.
Revised: In this block, a pointwise convolution is used to reduce the number of channels of the input by a factor of b before a standard convolution is applied.

Original: Then, another pointwise convolution brings the channel size back up.
Revised: Then, another pointwise convolution brings the channel size back up.

Original: This uses ( N/b ) 2 k 2 + 2 N 2 /b parameters.
Revised: This uses ( N/b ) 2 k 2 + 2 N 2 /b parameters.

Original: Bottleneck Grouped+Pointwise Block – BG(b, g)
Revised: Bottleneck Grouped+Pointwise Block – BG(b, g)

Original: This is identical to the bottleneck block, except the standard convolution is further split into g groups, and so uses ( N/bg ) 2 k 2 + 2 N 2 /b parameters.
Revised: This is identical to the bottleneck block, except the standard convolution is further split into g groups, and so uses ( N/bg ) 2 k 2 + 2 N 2 /b parameters.

Original: 3. D ISTILLATION VIA A TTENTION T RANSFER
Revised: 3. D ISTILLATION VIA A TTENTION T RANSFER

Original: Attention transfer (Zagoruyko & Komodakis, 2017) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points are made to be similar to those produced by a large teacher network.
Revised: Attention transfer (Zagoruyko & Komodakis, 2017) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points are made to be similar to those produced by a large teacher network.

Original: A formal deﬁnition follows: Consider a choice of layers i = 1 , 2 , ...,
Revised: A formal deﬁnition follows: Consider a choice of layers i = 1 , 2 , ...,

Original: L in a teacher network with L layers, and the corresponding layers in the student network.
Revised: L in a teacher network with L layers, and the corresponding layers in the student network.

Original: At each chosen layer i of the teacher network, collect the spatial map of the activations for channel j into the vector a t ij .
Revised: At each chosen layer i of the teacher network, collect the spatial map of the activations for channel j into the vector a tij .

Original: Let A t i collect a t ij for all j .
Revised: Let A ti collect a tij for all j .

Original: Likewise for the student network we correspondingly collect into a sij and A si .
Revised: Likewise for the student network we correspondingly collect into a sij and A si .

Original: Now given some choice of mapping f
Revised: Now given some choice of mapping f

Original: ( A i ) that maps each collection of the form
Revised: ( A i ) that maps each collection of the form

Original: A i into a vector, attention transfer involves learning the student network by minimising
Revised: A i into a vector, attention transfer involves learning the student network by minimising

Original: [Equation]
Revised: [Equation]

Original: where β is a hyperparameter, and L CE is the standard cross-entropy loss.
Revised: where β is a hyperparameter, and L CE is the standard cross-entropy loss.

Original: In Zagoruyko & Komodakis (2017) the authors use f ( A i ) =
Revised: In Zagoruyko & Komodakis (2017) the authors use f ( A i ) =

Original: (1 /N A i ) (cid:80) N
Revised: (1 /N A i ) (cid:80) N

Original: Ai j =1 a 2 ij , where N A
Revised: Ai j =1 a 2 ij , where N A

Original: i is the number of channels at layer i .
Revised: i is the number of channels at layer i .

Original: 3. F ISHER I NFORMATION
Revised: 3. F ISHER I NFORMATION

Original: Theis et al.
Revised: Theis et al.

Original: (2018) derive a second order approximation of the change in loss that would occur on the removal of a particular channel activation in a neural network.
Revised: (2018) derive a second order approximation of the change in loss that would occur on the removal of a particular channel activation in a neural network; they demonstrate that this is equivalent to calculating an empirical estimate of the Fisher information for a binary mask parameter that is used to toggle that channel on or off.

Original: They use this signal ∆ c to identify the least important activation channels, and remove their corresponding weights whileperforming channel pruning.
Revised: They use this signal ∆ c to identify the least important activation channels, and remove their corresponding weights while pruning.

Original: Formally, let us consider a single channel of an activation in a network due to some input minibatch of N examples.
Revised: Formally, let us consider a single channel of an activation in a network due to some input minibatch of N examples.

Original: Let us denote the values for this channel as A : a N × W × H tensor where W and H are the channel’s spatial width and height.
Revised: Let us denote the values for this channel as a : a N × W × H tensor where W and H are the channel’s spatial width and height.

Original: Let us refer to the entry corresponding to example n in the mini-batch at location ( i, j ) as A nij .
Revised: Let us refer to the entry corresponding to example n in the mini-batch at location ( i, j ) as a nij .

Original: If the network has a lossfunction L, then we can back-propagate to get the gradient of the loss with respect to this activation channel ∂ L ∂A .
Revised: We can backpropagate the network’s loss L to obtain the gradient of L with respect to this activation channel ∂ L ∂a .

Original: Let us denote this gradient as g and index it as g nij .
Revised: Let us denote this gradient as g and index it as g nij .

Original: ∆ c can then be computed by
Revised: ∆ c can then be computed by

Original: [Equation]
Revised: [Equation]

Revised: In this work, we are interested in obtaining the Fisher information for a whole block.

Revised: We approximate this quantity by summing ∆ c for every output channel in a block as

Revised: [Equation]

Original: Using an approximation to the Taylor expansion of the loss around an activation originated in LeCun et al.
Revised: Using an approximation to the Taylor expansion of a change in loss to gauge the saliency of individual parameters originated in LeCun et al.

Original: and has inspired many works in pruning (Hassibi & Stork, 1993; Molchanov et al., 2017; Guo et al., 2016; Srinivas & Babu, 2015) and quantisation (Choi et al., 2017; Hou et al., 2017).
Revised: and has inspired many works in pruning (Hassibi & Stork, 1993; Molchanov et al., 2017; Guo et al., 2016; Srinivas & Babu, 2015) and quantisation (Choi et al., 2017; Hou et al., 2017).

Original:  M ETHOD
Revised:  M ETHOD

Original: Let us denote a large teacher network T composed of B blocks each of type S as T
Revised: Let us denote a large teacher network T composed of B blocks each of type S as T

Original: =
Revised: =

Original: [ S 1 , S 2 , ..., S B ] .
Revised: [ S 1 , S 2 , ..., S B ] .

Original: Each of these may be replaced by a cheap block C r , chosen from a list of candidates C 1 , C 2 , ..., C N of various representational capacities.
Revised: Each of these may be replaced by a cheap block C r , chosen from a list of candidates C 1 , C 2 , ..., C N of various representational capacities.

Original: We wish to construct a smaller model M =
Revised: We wish to construct a smaller model M =

Original: [ C r 1 , C r 2 , ..., C rB ] that is powerful, and within a given parameter budget.
Revised: [ C r 1 , C r 2 , ..., C rB ] that is powerful, and within a given parameter budget.

Original: But, the space of possible block conﬁgurations is very large ( ∼ 10 23 in Section 5).
Revised: But, the space of possible block conﬁgurations is very large ( ∼ 10 23 in Section 5).

Original: Even when using a cheap network evaluation strategy it is not possible to exhaustively search through the available network architectures.
Revised: Even when using a cheap network evaluation strategy it is not possible to exhaustively search through the available network architectures.

Original: What we require is a method to quickly propose and score possible conﬁgurations with as little training as possible.
Revised: What we require is a method to quickly propose and score possible conﬁgurations with as little training as possible.

Original:  BlockSwap therefore relies on randomly sampling block conﬁgurations that satisfy a parameter budget using rejection sampling—we generate  architecture proposals at random and only save those that satisfy our parameter budget— and then ranking the  saved samples by  Fisher potential to decide which architectures to train.
Revised: We develop BlockSwap to achieve this. First, we obtain candidate architectures through rejection sampling: we generate mixed-blocktype architecture proposals at random and only save those that satisfy our parameter budget. As these are only proposals, this step does not require instantiating any networks; parameter count can be inferred directly from the proposed conﬁguration, making this a very cheap operation. Second, we score each of our saved candidates by its Fisher potential to determine which network to train.

Original: We obtain a score for each network as follows: we  place probes after the last convolution in each block.
Revised: We obtain this score for each candidate as follows: we initialise the network from scratch and then place probes after the last convolution in each block.

Original: When a single minibatch of training data is  passed through the network and the resulting cross-entropy loss is backpropagated, the probe measures the  Fisher potential of the block  by summing ∆ c (Equation 2) for each channel in the layer it is placed after.
Revised: A single minibatch of training data is then passed through the network, and the resulting cross-entropy loss is backpropagated. The probe measures the total Fisher information of the block ∆ b by summing ∆ c (Equation 2) for each channel in the layer it is placed after.

Original: For this step, minibatch size is set equal to the size used during training.
Revised: For this step, minibatch size is set equal to the size used during training.

Original: We then sum the Fisher potential of each block to give usa score for the network.
Revised: We then sum this quantity across all blocks to give us the Fisher potential.

Original: The intuition for this metric is as follows: the Fisher information measures the sensitivity of the loss to each parameter at initialisation.
Revised: The intuition for this metric is as follows: the Fisher potential is the trace of the Fisher Information matrix of the activations.

Original: Where parameters are insensitive at initialisation, they commonly remain insensitive as the other parameters are adjusted to adapt to the signal. Hence the accumulated Fisher potential measures the parameter wastage: how much of the parameter space is likely to be redundant for learning.
Revised: It is an aggregate of the total information each block contains about the class label (under a simplifying conditional independence assumption). During training it is this information about the class that drives learning, and the initial learning steps are key.

Original: We wish to choose architectures which are efﬁcient (low parameter numbers) and have low wastage (use the parameters they have well).
Revised: Hence higher information values tend to result in higher efﬁciency block utilisation.

Original: Once we have scored each candidate architecture using the Fisher potential, we select thebest one  and train it using attention transfer.
Revised: Once we have scored each candidate architecture using the Fisher potential, we select the one with the highest score and train it using attention transfer.

Original: The training hyperparameters can be mirrored from T .
Revised: The training hyperparameters can be mirrored from T .

Original: We use the following blocks (deﬁned in Section 3.1) as candidate blocks:
Revised: We use the following blocks (deﬁned in Section 3.1) as candidate blocks:

Original: • B ( b ) for b ∈ { 2 , 4 } • G ( g ) for g ∈ { 2 , 4 , 8 , 16 , N/ 16 , N/ 8 , N/ 4 , N/ 2 , N } • BG (2 , g ) for g ∈ { 2 , 4 , 8 , 16 , M/ 16 , M/ 8 , M/ 4 , M/ 2 , M }
Revised: • B ( b ) for b ∈ { 2 , 4 } • G ( g ) for g ∈ { 2 , 4 , 8 , 16 , N/ 16 , N/ 8 , N/ 4 , N/ 2 , N } • BG (2 , g ) for g ∈ { 2 , 4 , 8 , 16 , M/ 16 , M/ 8 , M/ 4 , M/ 2 , M }

Original: where N is the number of channels in a block, and M is the number of channels after a bottleneck— this is N/ 2 when b = 2 .
Revised: where N is the number of channels in a block, and M is the number of channels after a bottleneck— this is N/ 2 when b = 2 .

Original: We also use the standard block S as a candidate choice, so as to not force blocks to reduce capacity where it is imperative.
Revised: We also use the standard block S as a candidate choice, so as to not force blocks to reduce capacity where it is imperative.

Original: 4. W HY MIX BLOCK TYPES ?
Revised: 4. W HY MIX BLOCK TYPES ?

Original: A reasonable question to ask is whether mixed-blocktype architectures will perform better than the use of a single blocktype.
Revised: A reasonable question to ask is whether mixed-blocktype architectures will perform better than the use of a single blocktype.

Original: As a simple check for this, we chose parameter budgets of 200K, 400K, and 800K and at each budget, compared single-blocktype models to one hundred randomly assembled, mixed-blocktype models.
Revised: As a simple check for this, we chose parameter budgets of 200K, 400K, and 800K and at each budget, compared single-blocktype models to one hundred randomly assembled, mixed-blocktype models.

Original: The models are WideResNets (Zagoruyko & Komodakis, 2016) with substituted blocks trained on CIFAR-10.[Figure]
Revised: The models are WideResNets (Zagoruyko & Komodakis, 2016) with substituted blocks trained on CIFAR-10.

Original: 2,4) trains to 5.0% error, while the mean error of randomly constructed networks at the same parameter budget lies at around 5.15% error.

Original: This implies that random architecture search will not work here, and that a better strategy is needed.

Original: In Figure 2 we plot the ﬁnal test error of the random architectures (blue densities) against their single-blocktype counterparts (the red dotted lines).
Revised: In Figure 2 we plot the ﬁnal test error of the random architectures (blue densities) against their single-blocktype counterparts (the red dotted lines).

Original: We observe that for each of the given parameter budgets there exist combinations of mixed blocks that are more powerful than their single blocktype equivalents.
Revised: We observe that for each of the given parameter budgets there exist combinations of mixed blocks that are more powerful than their single blocktype equivalents.

Original: However, single-blocktype networks typically sit below the mean of the density of the random networks.
Revised: However, single-blocktype networks typically sit below the mean of the density of the random networks.

Original: This suggests that, though random search has been shown to work well in  architecture search settings (Sciuto et al., 2019), it will yield sub-optimal structures in this case. 
Revised: This suggests that, though random search has been shown to work well in other architecture search settings (Yu et al., 2020), it will yield sub-optimal structures in this case. [Figure]

Original: 4. W HY USE A SINGLE MINIBATCH ? H OW DOES YOUR RANKING METRIC COMPARE  ?
Revised: 4. W HY USE A SINGLE MINIBATCH ? H OW DOES YOUR RANKING METRIC COMPARE TO ALTERNATIVES ?

Original: Though there are no common metrics for architecture ranking, neural architecture search methods often use information from early learning stages to evaluate possible candidates.
Revised: Though there are no common metrics for architecture ranking, neural architecture search methods often use information from early learning stages to evaluate possible candidates.

Original: In order to assess the fastest, most accurate indicator of ﬁnal error, we examine our 100 random mixed-blocktype networks for parameter budgets of 200K, 400K, and 800K and record how ﬁnal error correlates with (i) train accuracy, (ii) total (cid:96) 2 norms, (iii) summed absolute gradient norms  (Golub et al., 2019), and (iv) Fisher potential after 1, 10, and 100 minibatches of training in Table 1.
Revised: In order to assess the fastest, most accurate indicator of ﬁnal error, we examine our 100 random mixed-blocktype networks for parameter budgets of 200K, 400K, and 800K and record how ﬁnal error correlates with (i) training accuracy, (ii) total (cid:96) 2 norms of weights, (iii) summed absolute gradient norms of weights (Golub et al., 2019), and (iv) Fisher potential after 1, 10, and 100 minibatches of training in Table 1.

Original: We found that Fisher potential was by far the most robust metric, and that there were diminishing returns for adding more than a single minibatch.
Revised: We found that Fisher potential was by far the most robust metric, and that there were diminishing returns as the number of minibatches increases. [Table] [Table]

Original: 4. H OW MANY SAMPLES ARE NEEDED ?
Revised: H OW MANY SAMPLES ARE NEEDED ?

Original: Figure 2 suggests that sampling 100 random architectures will reliably yield at least one mixedblocktype network that outperforms a similarly parameterised single blocktype network.
Revised: Figure 2 suggests that sampling 100 random architectures will reliably yield at least one mixedblocktype network that outperforms a similarly parameterised single blocktype network.

Original: However, accepting that there is some noise to our ranking metric, we assume that we will need to take more than 100 samples in order to reliably detect powerful architectures.
Revised: However, accepting that there is some noise in our ranking metric, we assume that we will need to take more than 100 samples in order to reliably detect powerful architectures.

Original: As an illustration, at the budget of 400K parameters a single blocktype alternative has a test error of 4 . 45% , whereas BlockSwap ﬁnds networks with ﬁnal test errors of 4 . 85% . 4 .
Revised: As an illustration, at the budget of 400K parameters a single blocktype alternative has a test error of 4 . 45% , whereas BlockSwap ﬁnds networks with ﬁnal test errors of 4 . 85% . 4 .

Original: 54% , and 4 .
Revised: 54% , and 4 .

Original: 21% after 10, 100, and 1000 samples respectively.
Revised: 21% after 10, 100, and 1000 samples respectively.

Original: We empirically found that 1000 samples was enough to be robust to various parameter budgets on the tasks we considered.[Table] [Table] [Table]
Revised: We empirically found that 1000 samples was enough to be robust to various parameter budgets on the tasks we considered.

Original:  W HAT DO “ GOOD NETWORKS ” LOOK LIKE ?
Revised: 4. W HAT DO “ GOOD NETWORKS ” LOOK LIKE ?

Original: For each of the three parameter budgets (200K, 400K, 800K) we inspected the most common block choices for “good” and “bad” networks, where good and bad are networks with ﬁnal error greater two standard deviations below or above the mean respectively.
Revised: For each of the three parameter budgets (200K, 400K, 800K) we inspected the most common block choices for “good” and “bad” networks, where good and bad are networks with ﬁnal error greater two standard deviations below or above the mean respectively.

Original: We found that overwhelmingly, the types of blocks used in both good and bad networks was very similar, implying that it is the placement of the blocks instead of the types of the blocks that matters .
Revised: We found that overwhelmingly, the types of blocks used in both good and bad networks was very similar, implying that it is the placement of the blocks instead of the types of the blocks that matters .

Original: We examine this further in Appendix B
Revised: We examine this further in Appendix B.

Original: [Table]

Original:  CIFAR E XPERIMENTS
Revised:  CIFAR E XPERIMENTS

Original: Here, we evaluate student networks obtained using BlockSwap on the CIFAR-10 image classiﬁcation dataset (Krizhevsky, 2009).
Revised: Here, we evaluate student networks obtained using BlockSwap on the CIFAR-10 image classiﬁcation dataset (Krizhevsky, 2009).

Original: We benchmark these against competing student networks for a range of parameter budgets.
Revised: We benchmark these against competing student networks for a range of parameter budgets.

Original: To recapitulate, the BlockSwap networks are found by taking 1000 random samples from the space of possible block combinations that satisfy our constraint (in this case, parameter budget).
Revised: To recapitulate, the BlockSwap networks are found by taking 1000 random samples from the space of possible block combinations that satisfy our constraint (in this case, parameter budget).

Original: These points are ranked by Fisher potential after a single minibatch of training data, and the network with the highest potential is chosen.
Revised: These points are ranked by Fisher potential after a single minibatch of training data, and the network with the highest potential is chosen.

Original: The structure we use is that of a WideResNet (Zagoruyko & Komodakis, 2017), since we can use them to construct compact, high performance networks that are broadly representative of the most commonly used neural architectures.
Revised: The structure we use is that of a WideResNet (Zagoruyko & Komodakis, 2017), since we can use them to construct compact, high performance networks that are broadly representative of the most commonly used neural architectures.

Original: A WideResNet with depth 40, and width multiplier 2—WRN40-2—is trained and used as a teacher.
Revised: A WideResNet with depth 40, and width multiplier 2—WRN40-2—is trained and used as a teacher.

Original: It consists of 18 blocks and has 2.2 million parameters.
Revised: It consists of 18 blocks and has 2.2 million parameters.

Original: A student network is generated and is trained from scratch using attention transfer with the teacher.
Revised: A student network is generated and is trained from scratch using attention transfer with the teacher.

Original: Our BlockSwap students are WRN-40-2 nets where each of its 18 blocks are determined using our method with the blocks outlined in Section 4.
Revised: Our BlockSwap students are WRN-40-2 nets where each of its 18 blocks are determined using our method with the blocks outlined in Section 4.

Original: We compare against the following students:
Revised: We compare against the following students:

Original: Reduced width/depth versions of the teacher: WRN-16-1,WRN-16-2, WRN-402.
Revised: Reduced width/depth versions of the teacher: WRN-16-1,WRN-16-2, WRN-402.

Original: Single block-type teacher reductions: every block in the teacher is swapped out for (i) the MBConv6 block from MobileNet v2 (Sandler et al., 2018) and (ii) the  block discovered in DARTS (Liu et al., 2019a).
Revised: Single block-type teacher reductions: every block in the teacher is swapped out for (i) the MBConv6 block from MobileNet v2 (Sandler et al., 2018) and (ii) the (normal cell) block discovered in DARTS (Liu et al., 2019a).

Original: Pruned teacher reductions: we compare against (i) the magnitude-based pruning methodology from Han et al.
Revised: Pruned teacher reductions: we compare against (i) the magnitude-based pruning methodology from Han et al.

Original: (2016) and (ii) SNIP
Revised: (2016) and (ii) SNIP

Original: (Lee et al., 2019) versions of the teacher.
Revised: (Lee et al., 2019) versions of the teacher.

Original: We additionally compare against CondenseNet-86 (Huang et al., 2018), distilled with the Born-Again strategy described by Furlanello et al.
Revised: We additionally compare against CondenseNet-86 (Huang et al., 2018), distilled with the Born-Again strategy described by Furlanello et al.

Original: First, we train three teacher networks independently.
Revised: First, we train three teacher networks independently.

Original: These are used to train all of our students; each student network is trained three times, once with each of these teachers.
Revised: These are used to train all of our students; each student network is trained three times, once with each of these teachers.

Original: Figure 3 shows the mean test errors of BlockSwap students at various parameter counts, compared to the alternatives listed above.
Revised: Figure 3 shows the mean test errors of BlockSwap students at various parameter counts, compared to the alternatives listed above.

Original: Full results with standard deviations are listed in Table 2, along with the number of
Revised: Full results with standard deviations are listed in Table 2, along with the number of

Original: Multiply-Accumulate (MAC) operations each network uses.
Revised: Multiply-Accumulate (MAC) operations each network uses.

Original: Our results show that block cheapening is more effective than simple downscaling schemes (reducing width/depth); not only are the reduced models (WRN-40-1, WRN-16-2, WRN-16-1) inﬂexible in parameter range, they perform signiﬁcantly worse than our models at the parameter budgets they can map to.
Revised: Our results show that block cheapening is more effective than simple downscaling schemes (reducing width/depth); not only are the reduced models (WRN-40-1, WRN-16-2, WRN-16-1) inﬂexible in parameter range, they perform signiﬁcantly worse than our models at the parameter budgets they can map to.

Original: We also show that BlockSwap ﬁnds more accurate networks than the other top-down approaches ( (cid:96) 1 -pruning and SNIP) across the full parameter spectrum.
Revised: We also show that BlockSwap ﬁnds more accurate networks than the other top-down approaches ( (cid:96) 1 -pruning and SNIP) across the full parameter spectrum.

Original: Note that as (cid:96) 1 -pruning and
Revised: Note that as (cid:96) 1 -pruning and

Original: [Table]
Revised: SNIP introduce unstructured sparsity, the parameter counts provided are the number of non-zero parameters (the weight tensors remain the same size as the original teacher). The mixed block architectures that BlockSwap generates are more accurate at these reduced budgets than all of the single blocktype alternatives we considered.

Original: [Figure]
Revised: While the performance of DARTS is very similar to BlockSwap, it is worth noting that the DARTS architecture search required 24 GPU hours. By comparison, a BlockSwap search for an 800K parameter network took less than 5 minutes using a single Titan X Pascal GPU. Given the strength of random baselines in architecture search settings (Liu et al., 2019a; Yu et al., 2020; Li & Talwalkar, 2019), we also compare BlockSwap against randomly generated mixedblocktype conﬁgurations in Appendix C. BlockSwap consistently outperforms these, demonstrating that our Fisher potential metric is effective at selecting potent block structures.

Original: [Figure]
Revised: [Table]

Original:  BlockSwap [Figure]
Revised: 40-2 SNIP 162. - 6.00 ± 0. 40-2 BlockSwap 811.4 132.5 3.

Original: [Figure]
Revised: [Figure]

Original:  BlockSwap 556.
Revised: 40-2 BlockSwap 556.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 4.17 ± 0.

Original:  BlockSwap 404.
Revised: 40-2 BlockSwap 404.

Original: [Figure]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 4.21 ± 0.

Original:  BlockSwap 289.
Revised: 40-2 BlockSwap 289.

Original: [Figure] [Figure] [Figure] [Figure]
Revised: 4.45 ± 0.

Original:  BlockSwap [Figure] [Figure] [Figure]
Revised: 40-2 BlockSwap 217.

Original: [Figure]
Revised: [Figure]

Original:  BlockSwap 162.
Revised: 40-2 BlockSwap 162.

Original: [Figure]
Revised: [Figure]

Original:  also outperform single blocktype networks (
Revised: [Figure] [Figure] They also outperform single blocktype networks (MBConv, DARTS, CondenseNet-86).

Original: SNIP introduce unstructured sparsity, the parameter counts provided are the number of non-zero parameters (the weight tensors remain the same size as the original teacher). The mixed block architectures thatBlockSwap generates are more accurate at these reduced budgets than all of the single blocktype alternatives we considered. While the performance of DARTS is very similar to BlockSwap, it is worth noting that the DARTS architecture search required 24 GPU hours, while BlockSwap needs only 8 minutes on a single CPU.
Revised: Note that all networks have been trained using attention transfer or have been born

Original: Given the strength of random baselines in architecture search settings (Liu etal., 2019a; Sciuto et al., 2019; Li & Talwalkar, 2019), we also compare BlockSwap against randomly generated block conﬁgurations in Appendix C. BlockSwap consistently outperforms these, proving that our aggregated Fisher potential metric is an effective means for choosing optimal block structures.
Revised: (Furlanello et in the case of CondenseNet

Original: Implementation Details: Networks are trained for 200 epochs using SGD with momentum 0.9.
Revised: Implementation Details: Networks are trained for 200 epochs using SGD with momentum 0.9.

Original: The initial learning rate of 0.1 is reduced by a factor of 5 every 60 epochs.
Revised: The initial learning rate of 0.1 is cosine annealed (Loshchilov & Hutter, 2017) to zero across the training run.

Original: Minibatches of size 128 are used with standard crop + ﬂip data augmentation and Cutout (DeVries & Taylor, 2017).
Revised: Minibatches of size 128 are used with standard crop + ﬂip data augmentation and Cutout (DeVries & Taylor, 2017).

Original: The weight decay factor is set to 0.0005.
Revised: The weight decay factor is set to 0.0005.

Original: For attention transfer β is set to 1000.
Revised: For attention transfer β is set to 1000.

Original:  I MAGE N ET C LASSIFICATION
Revised:  I MAGE N ET C LASSIFICATION

Original: Here, we demonstrate that students chosen by BlockSwap succeed on the more challenging ImageNet dataset (Russakovsky et al., 2015).
Revised: Here, we demonstrate that students chosen by BlockSwap succeed on the more challenging ImageNet dataset (Russakovsky et al., 2015).

Original: We use a pretrained ResNet-34 (16 blocks) as a teacher, and compare students at two parameter budgets (3M and 8M).
Revised: We use a pretrained ResNet-34 (16 blocks, 21.8M parameters) as a teacher, and compare students at two parameter budgets (3M and 8M).

Original: We train a Blockswap student at each of these budgets and compare their validation errors to those of a reduced depth/width student (ResNet and ResNet-18-0.5—a ResNet-18 where the channel width in the last 3 sections has been halved) and a single-blocktype student (ResNet-34 with G(4) and G(N) blocks).
Revised: We train a BlockSwap student at each of these budgets and compare their validation errors to those of a reduced depth/width student (ResNet18 and ResNet-18-0.5—a ResNet-18 where the channel width in the last 3 sections has been halved) and a single-blocktype student (ResNet-34 with G(4) and G(N) blocks).

Original: The student networks found by BlockSwap for these two budgets are illustrated in Appendix D. Top-1 and top-5 validation errors are presented in Table 3.
Revised: The student networks found by BlockSwap for these two budgets are illustrated in Appendix D. Top-1 and top-5 validation errors are presented in Table 3.

Original: At both budgets, BlockSwap chooses networks that outperform its comparators.
Revised: At both budgets, BlockSwap chooses networks that outperform its comparators.

Original: At 8M parameters it even surpasses the teacher by quite a margin.
Revised: At 8M parameters it even surpasses the teacher by quite a margin.

Original: Speciﬁcally, it beats the teacher by 0.49% in top-1 error and 0.82% in top-5 error despite using almost 3 × fewer parameters.
Revised: Speciﬁcally, it beats the teacher by 0.49% in top-1 error and 0.82% in top-5 error despite using almost 3 × fewer parameters.

Original: Implementation Details: Networks are trained with a cross-entropy loss for 100 epochs using SGD with momentum 0.9.
Revised: Implementation Details: Networks are trained with a cross-entropy loss for 100 epochs using SGD with momentum 0.9.

Original: The initial learning rate of 0.1 is reduced by 10 × every 30 epochs.
Revised: The initial learning rate of 0.1 is reduced by 10 × every 30 epochs.

Original: Minibatches of size 256 are used with standard crop + ﬂip augmentation.
Revised: Minibatches of size 256—split across 4 GPUs—are used with standard crop + ﬂip augmentation.

Original: The weight decay factor is set to 0.0001.
Revised: The weight decay factor is set to 0.0001.

Original: For attention transfer β is set to 750 using the output of each of the four sections of network.
Revised: For attention transfer β is set to 750 using the output of each of the four sections of network.

Original: [Table]
Revised: [Table]

Original:  COCO D ETECTION
Revised:  COCO D ETECTION

Original: Thus far [Table] we have used BlockSwap for image classiﬁcation problems.
Revised: Thus far, we have used BlockSwap for image classiﬁcation problems.

Original: Here we observe whether it extends to object detection on the COCO dataset (Lin et al., 2014)—speciﬁcally, training ontrain
Revised: Here we observe whether it extends to object detection on the COCO dataset (Lin et al., 2014)—speciﬁcally, training ontrain

Original: [Table]
Revised: [Table]

Original: and evaluating on 2017 val.
Revised: and evaluating on 2017 val.

Original: We consider a Mask R-CNN (He et al., 2017) with a ResNetbackbone
Revised: We consider a Mask R-CNN (He et al., 2017) with a ResNetbackbone

Original: [Table]
Revised: [Table]

Original: and apply BlockSwap using COCO images to obtain a mixed-blocktype backbone with 3M parameters.
Revised: and apply BlockSwap using COCO images to obtain a mixed-blocktype backbone with 3M parameters.

Original: We compare this to a single-blocktype ResNet-34-G(N) backbone which uses the same number of parameters.
Revised: We compare this to a single-blocktype ResNet-34-G(N) backbone which uses the same number of parameters.

Original: To avoid conﬂation with ImageNet, we train everything from scratch .
Revised: To avoid conﬂation with ImageNet, we train everything from scratch .

Original: The results can be found in Table 4.
Revised: The results can be found in Table 4.

Original: We can see that the BlockSwap backbone again outperforms its single-blocktype counterpart.
Revised: We can see that the BlockSwap backbone again outperforms its single-blocktype counterpart.

Original: Implementation Details: Networks are trained using the default Mask R-CNN settings in Torchvision.
Revised: Implementation Details: Networks are trained using the default Mask R-CNN settings in Torchvision.

Original: We use a batch-size of 16 split across 8 GPUs.
Revised: We use a batch-size of 16 split across 8 GPUs.

Original: All models are trained from scratch, and we forgo distillation due to memory constraints.
Revised: All models are trained from scratch, and we forgo distillation due to memory constraints.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We have developed BlockSwap: a fast, simple method for reducing large neural networks to ﬂexible parameter targets based on block substitution.
Revised: We have developed BlockSwap: a fast, simple method for reducing large neural networks to ﬂexible parameter targets based on block substitution.

Original: We have veriﬁed that these reduced networks make for excellent students, and have performed a comprehensive ablation study.
Revised: We have veriﬁed that these reduced networks make for excellent students, and have performed a comprehensive ablation study.

Original: Future work could use
Revised: Future work could use

Original: BlockSwap to choose networks based on inference time, or energy cost instead of parameter count.
Revised: BlockSwap to choose networks based on inference time, or energy cost instead of parameter count.

Original: A B LOCK R EPLACEMENTS
Revised: A CKNOWLEDGMENTS

Revised: This work was supported in part by the EPSRC Centre for Doctoral Training in Pervasive Parallelism and a Huawei DDMPLab Innovation Research Grant, as well as funding from the European Union’s

Revised: Horizon 2020 research and innovation programme under grant agreement No.732204 (Bonseyes).

Revised: This work is supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 16.0159.

Revised: The opinions expressed and arguments employed herein do not necessarily reﬂect the ofﬁcial views of these funding bodies.

Revised: The authors are grateful to David

Revised: Sterratt for his L A TEX prowess and to the BayesWatch team and anonymous reviewers for their helpful comments.

Revised: A B LOCK R EPLACEMENTS

Original: Where applicable, g is the number of groups and b is the bottleneck contraction convolutional and batch-norm (BN) costs are given, although the latter is signiﬁcantly smaller.
Revised: Where applicable, g is the number of groups and b is the bottleneck contraction. The convolutional and batch-norm (BN) costs are given, although the latter is signiﬁcantly smaller.

Original: BN ReLU precedes
Revised: BN+ReLU precedes

Revised: WideResNets and convolution standard ResNets

Original: [Table]
Revised: [Table]

Original: B U NDERSTANDING B LOCK P LACEMENT
Revised: B U NDERSTANDING B LOCK P LACEMENT

Original: To understand what makes for “good” block placement
Revised: To understand what makes for “good” block placement

Original: [Table]
Revised: [Table]

Original: we begin with the more tractable question: if we were only to cheapen one block, which one should we choose?
Revised: we begin with the more tractable question: if we were only to cheapen one block, which one should we choose?

Original: In Figure 4 we show what happens when we substitute a single standard block with G(4) for each of the possible blocks for our
Revised: In Figure 4 we show what happens when we substitute a single standard block with G(4) for each of the possible blocks for our

Original: WideResNet.
Revised: WideResNet.

Original: We plot the ﬁnal test error after training to convergence, as well as the total parameter count after the substitution (the higher the bar, the higher the error, therefore the more sensitive the block).
Revised: We plot the ﬁnal test error after training to convergence, as well as the total parameter count after the substitution (the higher the bar, the higher the error, therefore the more sensitive the block).

Original: Notice that in Figure 4 (a), substituting the start and end of each stage appears to have a very detrimental effect on accuracy, whereas mid-stage blocks can be substituted without hurting performance.
Revised: Notice that in Figure 4(a), at substituting the start and end of each stage appears to have a very detrimental effect on accuracy, whereas mid-stage blocks can be substituted without hurting performance.

Original: For example, block 12 has few parameters, as shown in Figure 4(b), but substituting it for a cheaper alternative results in a drastic drop in accuracy.
Revised: For example, block 12 has few parameters, as shown in Figure 4(b), but substituting it for a cheaper alternative results in a drastic drop in accuracy.

Original: This implies that it is safest to substitute blocks in the middle of a stage but that blocks at the start and end of each stage should have their representational capacity preserved as much as possible.
Revised: This implies that it is safest to substitute blocks in the middle of a stage but that blocks at the start and end of each stage should have their representational capacity preserved as much as possible.

Original: We used this insight to analyse the block conﬁgurations of “good” and “bad” amongst the random architectures at each parameter budget.
Revised: We used this insight to analyse the block conﬁgurations of “good” and “bad” amongst the random architectures at each parameter budget.

Original: With several blocktypes, however, we need a method to rank the capacity of blocktypes; we make the simplifying assumption that training the blocktype on its own is a good indicator of this (though this is not always true, since blocks give varying performance with different input sizes).
Revised: With several blocktypes, however, we need a method to rank the capacity of blocktypes; we make the simplifying assumption that training the blocktype on its own is a good indicator of this (though this is not always true, since blocks give varying performance with different input sizes).

Original: An example of a good and bad network judged this way can be seen in
Revised: An example of a good and bad network judged this way can be seen in

Original: Figure 5; while they do roughly follow our observations, there are clearly more complex interactionsat play (perhaps unsurprising given the random nature of their construction).
Revised: Figure 5; while they do roughly follow our observations, there are clearly more complex interactions

Original: We found that overall, the models in our ablation set adhere to the pattern described above (for each stage: high capacity blocks at the start and end, increasing capacity throughout), with some variance.

Original: [Figure]
Revised: [Figure]

Original: ﬁnal error.

Original: [Figure]
Revised: [Figure]

Original: (a)
Revised: (a)

Original: C a p ac it y S
Revised: C a p ac it y S

Original: c
Revised: c

Original: o r e
Revised: o r e

Original: Bad model (5.25% error) {+
+}
Revised: Bad model (5.25% error) {+
+}

Original: [Figure]
Revised: [Figure]

Original:  (b)
Revised: 17 Block (b)

Original: C a p ac it y S
Revised: C a p ac it y S

Original: c
Revised: c

Original: o r e
Revised: o r e

Original: Good model (4.62% error)
Revised: Good model (4.62% error)

Original: C C OMPARING TO RANDOM CONFIGURATIONS 
Revised: C C OMPARING TO RANDOM CONFIGURATIONS AND (cid:96) 1 - PRUNING

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: .68 ± 0.

Original: [Table]
Revised: 40BlockSwap 162. 5.17 ± 0. 40Random 795. 4.26 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: Random 551.

Original: [Table]
Revised: 4.54 ± 0.

Original: [Table]
Revised: 40Random 397.

Original: [Table] [Table]
Revised: 4.91 ± 0. 40Random 285. 4.80 ± 0.

Original: [Table]
Revised: 40Random 217. 5.13 ± 0. 40Random 168. 5.75 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: (cid:96) 1 -pruning 894. - 4.12 ± 0.

Original: [Table]
Revised: [Table]

Revised: (cid:96) 1 -pruning 760.

Original: 32 ± 
Revised: - 4.32 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: (cid:96) 1 -pruning 671. - 4.51 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: (cid:96) 1 -pruning 536.

Original: [Table] [Table] [Table] [Table]
Revised: - 4.98 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: (cid:96) 1 -pruning 447.

Original: [Table] [Table]
Revised: - 5.60 ± 0.

Original: [Table] [Table] [Table] [Table]
Revised: 40- (cid:96) 1 -pruning 313.

Original: [Table]
Revised: - 7.70 ± 0.

Original: [Table]
Revised: 40- (cid:96) 1 -pruning 223.

Original: .93 ± [Table]
Revised: - 11.93 ± 0.

Original: D B LOCK S WAP N ETWORKS FOR I MAGE N ET
Revised: D B LOCK S WAP N ETWORKS FOR I MAGE N ET

Original: [Figure]
Revised: [Figure]