Original: B LOCK S WAP : F ISHER - GUIDED B LOCK S UBSTITUTION FOR N ETWORK C OMPRESSION ON A B UDGET
Revised: B LOCK S WAP : F ISHER - GUIDED B LOCK S UBSTITUTION FOR N ETWORK C OMPRESSION ON A B UDGET

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks.
Revised: The desire to map neural networks to varying-capacity devices has led to the development of a wealth of compression techniques, many of which involve replacing standard convolutional blocks in a large network with cheap alternative blocks.

Original: However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive.
Revised: However, not all blocks are created equally; for a required compute budget there may exist a potent combination of many different cheap blocks, though exhaustively searching for such a combination is prohibitively expensive.

Original: In this work, we develop BlockSwap : a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential .
Revised: In this work, we develop BlockSwap : a fast algorithm for choosing networks with interleaved block types by passing a single minibatch of training data through randomly initialised networks and gauging their Fisher potential .

Original: These networks can then be used as students and distilled with the original large network as a teacher.
Revised: These networks can then be used as students and distilled with the original large network as a teacher.

Original: We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classiﬁcation, and COCO for detection, and provide a comprehensive ablation study of our approach.
Revised: We demonstrate the effectiveness of the chosen networks across CIFAR-10 and ImageNet for classiﬁcation, and COCO for detection, and provide a comprehensive ablation study of our approach.

Original: BlockSwap quickly explores possible block conﬁgurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. 8 minutes on a single CPU for CIFAR-10).
Revised: BlockSwap quickly explores possible block conﬁgurations using a simple architecture ranking system, yielding highly competitive networks in orders of magnitude less time than most architecture search techniques (e.g. 8 minutes on a single CPU for CIFAR-10).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional Neural Networks (CNNs) are extremely popular, and demonstrate strong performance on a variety of challenging tasks.
Revised: Convolutional Neural Networks (CNNs) are extremely popular, and demonstrate strong performance on a variety of challenging tasks.

Original: Because of this, there exists a large range of scenarios in the wild in which practitioners wish to deploy these networks e.g. pedestrian detection in a vehicle’s computer, human activity recognition with wearables (Radu et al., 2018).
Revised: Because of this, there exists a large range of scenarios in the wild in which practitioners wish to deploy these networks e.g. pedestrian detection in a vehicle’s computer, human activity recognition with wearables (Radu et al., 2018).

Original: CNNs are largely over-parameterised; a fact that can be exploited when specialising networks for different resource budgets.
Revised: CNNs are largely over-parameterised; a fact that can be exploited when specialising networks for different resource budgets.

Original: For example, there is a wealth of work demonstrating that it is possible to replace expensive convolutional blocks in a large network with cheap alternatives e.g. those using grouped convolutions (Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018) or bottleneck structures (He et al., 2016; Sandler et al., 2018; Peng et al., 2018).
Revised: For example, there is a wealth of work demonstrating that it is possible to replace expensive convolutional blocks in a large network with cheap alternatives e.g. those using grouped convolutions (Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018) or bottleneck structures (He et al., 2016; Sandler et al., 2018; Peng et al., 2018).

Original: This creates a smaller network which may be used as a student , and trained through distillation (Ba & Caruana, 2014;
Revised: This creates a smaller network which may be used as a student , and trained through distillation (Ba & Caruana, 2014;

Original: Hinton et al., 2015) with the original large network as a teacher to retain performance.
Revised: Hinton et al., 2015) with the original large network as a teacher to retain performance.

Original: Typically, each network block is replaced with a single cheap alternative, producing a single-blocktype network for a given budget.
Revised: Typically, each network block is replaced with a single cheap alternative, producing a single-blocktype network for a given budget.

Original: By cheapening each block equally, one relies on the assumption that each block is of equal importance.
Revised: By cheapening each block equally, one relies on the assumption that each block is of equal importance.

Original: We posit instead that for each budget, there exist more powerful mixed-blocktype networks that assign non-uniform importance to each block by cheapening them to different extents.
Revised: We posit instead that for each budget, there exist more powerful mixed-blocktype networks that assign non-uniform importance to each block by cheapening them to different extents.

Original: We now have a paradigm of choice; for a given budget, it is not obvious which cheap alternatives to use for our student network, nor where to place them.
Revised: We now have a paradigm of choice; for a given budget, it is not obvious which cheap alternatives to use for our student network, nor where to place them.

Original: Let’s assume we have a large network that consists of B expensive convolutional blocks, and a candidate pool of C cheap blocks that we could substitute each of these out for, and we are given a limit on the number of parameters (and therefore memory) that the network can use.
Revised: Let’s assume we have a large network that consists of B expensive convolutional blocks, and a candidate pool of C cheap blocks that we could substitute each of these out for, and we are given a limit on the number of parameters (and therefore memory) that the network can use.

Original: Which of the C B mixed-blocktype networks is appropriate?
Revised: Which of the C B mixed-blocktype networks is appropriate?

Original: It rapidly becomes infeasible to exhaustively consider these networks, even for single digit C .
Revised: It rapidly becomes infeasible to exhaustively consider these networks, even for single digit C .

Original: We could turn to neural architecture search (NAS) techniques (Zoph & Le, 2017; Pham et al., 2018;
Revised: We could turn to neural architecture search (NAS) techniques (Zoph & Le, 2017; Pham et al., 2018;

Original: Tan et al., 2019; Wu et al., 2019; Liu et al., 2019a) but these introduce a gigantic computational overhead; even one of the fastest such approaches (Pham et al., 2018) still requires half a day of search time on a GPU.
Revised: Tan et al., 2019; Wu et al., 2019; Liu et al., 2019a) but these introduce a gigantic computational overhead; even one of the fastest such approaches (Pham et al., 2018) still requires half a day of search time on a GPU.

Original: Moreover, in real-world scenarios, allocated budgets can change quickly; we would like to be able to ﬁnd a good network in a matter of minutes, not hours or days.
Revised: Moreover, in real-world scenarios, allocated budgets can change quickly; we would like to be able to ﬁnd a good network in a matter of minutes, not hours or days.

Original: [Figure]
Revised: [Figure]

Original: Our goal in this paper is, given a desired parameter budget, to quickly identify a suitable mixedblocktype version of the original network that makes for a powerful student.
Revised: Our goal in this paper is, given a desired parameter budget, to quickly identify a suitable mixedblocktype version of the original network that makes for a powerful student.

Original: We present a simple method— BlockSwap —to achieve this.
Revised: We present a simple method— BlockSwap —to achieve this.

Original: First, we randomly sample a collection of candidate mixedblocktype architectures that satisfy a desired parameter budget.
Revised: First, we randomly sample a collection of candidate mixedblocktype architectures that satisfy a desired parameter budget.

Original: A single minibatch is then pushed through each candidate network to calculate its Fisher potential : the sum of the total Fisher information for each of its blocks.
Revised: A single minibatch is then pushed through each candidate network to calculate its Fisher potential : the sum of the total Fisher information for each of its blocks.

Original: Finally, the network with the highest potential is selected as a student and trained through distillation—speciﬁcally, attention transfer (Zagoruyko & Komodakis, 2017)—with the original teacher.
Revised: Finally, the network with the highest potential is selected as a student and trained through the distillation method of attention transfer (Zagoruyko & Komodakis, 2017) with the original teacher.

Original: Our method is illustrated in Figure 1.
Revised: Our method is illustrated in Figure 1.

Original: In Section 3 we describe the block substitutions used in BlockSwap, distillation via attention transfer, and Fisher information.
Revised: In Section 3 we describe the block substitutions used in BlockSwap, distillation via attention transfer, and Fisher information.

Original: We describe our method in Section 4 as well as providing a comprehensive ablation study.
Revised: We describe our method in Section 4 as well as providing a comprehensive ablation study.

Original: Finally, we experimentally verify the potency of BlockSwap on CIFAR-10 (Section 5) as well as ImageNet (Section 6) and COCO (Section 7).
Revised: Finally, we experimentally verify the potency of BlockSwap on CIFAR-10 (Section 5) as well as ImageNet (Section 6) and COCO (Section 7).

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: We introduce BlockSwap, an algorithm for reducing large neural networks by performing block-wise substitution.
Revised: We introduce BlockSwap, an algorithm for reducing large neural networks by performing block-wise substitution.

Original: We show that this outperforms other top-down approaches such as depth/width scaling, parameter pruning, and random substitution.
Revised: We show that this outperforms other top-down approaches such as depth/width scaling, parameter pruning, and random substitution.

Original: We outline a simple method for quickly evaluating candidate models via Fisher information, which matches the performance of bottom-up approaches while reducing search time from days to minutes.
Revised: We outline a simple method for quickly evaluating candidate models via Fisher information, which matches the performance of bottom-up approaches while reducing search time from days to minutes.

Original: We conduct ablation 
Revised: We conduct ablation studies to validate

Original: studies to validate our methodology, highlighting the beneﬁts of block mixing, and conﬁrming that our ranking metric is highly correlated tothe ﬁnal error.
Revised: our ranking metric is highly correlated to ﬁnal error.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural networks tend to be overparameterised: Denil et al.
Revised: Neural networks tend to be overparameterised: Denil et al.

Original: (2013) accurately predict most of the weights in a network from a small subset; Frankle & Carbin (2019) hypothesise that within a large network, there exists a fortuitously initialised subnetwork that drives its performance.
Revised: (2013) accurately predict most of the weights in a network from a small subset; Frankle & Carbin (2019) hypothesise that within a large network, there exists a fortuitously initialised subnetwork that drives its performance.

Original: However, it remains difﬁcult to exploit this overparameterisation without taking a hit in performance.
Revised: However, it remains difﬁcult to exploit this overparameterisation without taking a hit in performance.

Original: One means to combat this is to use a large teacher network to regularise the training of a small student network; a process known as distillation.
Revised: One means to combat this is to use a large teacher network to regularise the training of a small student network; a process known as distillation.

Original: The small network is trained from scratch, but is also forced to match the outputs (Ba & Caruana, 2014; Hinton et al., 2015) or activation statistics (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Kim et al., 2018) of the teacher using an additional loss term.
Revised: The small network is trained from scratch, but is also forced to match the outputs (Ba & Caruana, 2014; Hinton et al., 2015) or activation statistics (Romero et al., 2015; Zagoruyko & Komodakis, 2017; Kim et al., 2018) of the teacher using an additional loss term.

Original: When utilising distillation one must decide how to create a student network.
Revised: When utilising distillation one must decide how to create a student network.

Original: A simple approach would be to reduce the depth of the original large network, although this can prove detrimental (Urban et al., 2017).
Revised: A simple approach would be to reduce the depth of the original large network, although this can prove detrimental (Urban et al., 2017).

Original: An effective strategy is to create a student by replacing all the teacher’s convolutions with grouped alternatives (Crowley et al., 2018).
Revised: An effective strategy is to create a student by replacing all the teacher’s convolutions with grouped alternatives (Crowley et al., 2018).

Original: Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying a much cheaper convolution on each split.
Revised: Grouped convolutions are a popular replacement for standard convolutions as they drastically cut the number of parameters used by splitting the input along the channel dimension and applying a much cheaper convolution on each split.

Original: They were originally used in AlexNet (Krizhevsky et al., 2012) due to GPU memory limitations, and have appeared in several subsequent architectures (Ioffe & Szegedy, 2015; Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018).
Revised: They were originally used in AlexNet (Krizhevsky et al., 2012) due to GPU memory limitations, and have appeared in several subsequent architectures (Ioffe & Szegedy, 2015; Chollet, 2017; Xie et al., 2017; Ioannou et al., 2017; Huang et al., 2018).

Original: However, as the number of groups increases, fewer channels are mixed, which hinders representational capacity.
Revised: However, as the number of groups increases, fewer channels are mixed, which hinders representational capacity.

Original: MobileNet (Howard et al., 2017) compensates for this by following its heavily-grouped depthwise convolutions by a pointwise ( 1 × 1 ) convolution to allow for channel mixing.
Revised: MobileNet (Howard et al., 2017) compensates for this by following its heavily-grouped depthwise convolutions by a pointwise ( 1 × 1 ) convolution to allow for channel mixing.

Original: The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search (NAS).
Revised: The increasing complexity of neural network designs has encouraged the development of methods for automating neural architecture search (NAS).

Original: Zoph & Le (2017) use an RNN to generate network descriptions and ﬁlter the options using reinforcement learning.
Revised: Zoph & Le (2017) use an RNN to generate network descriptions and ﬁlter the options using reinforcement learning.

Original: Storing such a large quantity of possible networks is expensive, and their evaluation strategy utilised 450 GPUs over the course of 3 days .
Revised: Storing such a large quantity of possible networks is expensive, and their evaluation strategy utilised 450 GPUs over the course of 3 days .

Original: To address this, Pham et al.
Revised: To address this, Pham et al.

Original: (2018) propose giving all models access to a shared set of weights, achieving similar performance to Zoph & Le (2017) with a single GPU in less than 24 hours.
Revised: (2018) propose giving all models access to a shared set of weights, achieving similar performance to Zoph & Le (2017) with a single GPU in less than 24 hours.

Original: Subsequent works have made extensive use of this technique (Liu et al., 2019a;
Revised: Subsequent works have made extensive use of this technique (Liu et al., 2019a;

Original: Luo et al., 2018; Chen et al., 2018).
Revised: Luo et al., 2018; Chen et al., 2018b).

Original: However, it has been observed that under the constrained architecture search space of the above methods, random architecture search provides a competitive baseline (Li & Talwalkar, 2019;
Revised: However, it has been observed that under the constrained architecture search space of the above methods, random architecture search provides a competitive baseline (Li & Talwalkar, 2019;

Original: Sciuto et al., 2019).
Revised: Sciuto et al., 2019).

Original: In particular, Sciuto et al.
Revised: In particular, Sciuto et al.

Original: (2019) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to ﬁnd suboptimal architectures.
Revised: (2019) show that weight sharing hampers the ability of candidate networks to learn and causes many NAS techniques to ﬁnd suboptimal architectures.

Original: NAS techniques predominantly take a bottom-up approach; they ﬁnd a powerful building block, and form neural networks using stacks of these blocks.
Revised: NAS techniques predominantly take a bottom-up approach; they ﬁnd a powerful building block, and form neural networks using stacks of these blocks.

Original: Other works have taken a top-down approach by searching for architectures using pruning (Lee et al., 2019; Liu et al., 2019b; Frankle & Carbin, 2019). 
Revised: Other works have taken a top-down approach to ﬁnd architectures using pruning (Chen et al., 2018a; Lee et al., 2019; Liu et al., 2019b; Frankle & Carbin, 2019). Chen et al.

Original:  SNIP (Lee et al., 2019) is a fast and effective means of doing this: one randomly initialises a large network and quantiﬁes the sensitivity of each connection using gradient magnitudes.
Revised: (2018a) take a pre-trained network and apply a principled framework to compress it under operational constraints such as latency. In SNIP (Lee et al., 2019), one randomly initialises a large network and quantiﬁes the sensitivity of each connection using gradient magnitudes.

Original: The lowest sensitivity connections are removed to produce a sparse architecture which is then trained as normal.
Revised: The lowest sensitivity connections are removed to produce a sparse architecture which is then trained as normal.

Original:  P RELIMINARIES 3. S UBSTITUTE B LOCKS
Revised:  P RELIMINARIES 3. S UBSTITUTE B LOCKS

Original: Here, we will brieﬂy elaborate on the block substitutions used for BlockSwap .
Revised: Here, we will brieﬂy elaborate on the block substitutions used for BlockSwap .

Original: A tabular comparison is given in Appendix A.
Revised: A tabular comparison is given in Appendix A.

Original: The block choices are deliberately simplistic.
Revised: The block choices are deliberately simplistic.

Original: We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of a speciﬁc highly-engineered block e.g. one from the NAS literature.
Revised: We can therefore demonstrate that it is the combination of blocks that is important rather than the representational capacity of a speciﬁc highly-engineered block e.g. one from the NAS literature.

Original: Standard Block – S
Revised: Standard Block – S

Original: The blocks considered are variations of the standard block used in residual networks.
Revised: The blocks considered are variations of the standard block used in residual networks.

Original: In the majority of these blocks, the input has the same number of channels as the output, so we describe their parameter cost assuming this is the case.
Revised: In the majority of these blocks, the input has the same number of channels as the output, so we describe their parameter cost assuming this is the case.

Original: This standard block contains two convolutional layers, each using N lots of N × k × k ﬁlters where k is the kernel size.
Revised: This standard block contains two convolutional layers, each using N lots of N × k × k ﬁlters where k is the kernel size.

Original: Assuming the costs of batch-norm (BN) layers and shortcut convolutions (where applicable) are negligible, the block uses a total of 2 N 2 k 2 parameters.
Revised: Assuming the costs of batch-norm (BN) layers and shortcut convolutions (where applicable) are negligible, the block uses a total of 2 N 2 k 2 parameters.

Original: Grouped+Pointwise Block – G(g)
Revised: Grouped+Pointwise Block – G(g)

Original: In a grouped convolution, the N channel input is split along the channel dimension into g groups, each of which has N/g channels.
Revised: In a grouped convolution, the N channel input is split along the channel dimension into g groups, each of which has N/g channels.

Original: Each group goes through its own convolution which outputs N/g channels, and all the outputs are concatenated along the channel dimension.
Revised: Each group goes through its own convolution which outputs N/g channels, and all the outputs are concatenated along the channel dimension.

Original: This uses g × ( N/g ) 2 k 2 =
Revised: This uses g × ( N/g ) 2 k 2 =

Original: ( N 2 k 2 ) /g parameters.
Revised: ( N 2 k 2 ) /g parameters.

Original: To compensate for reduced channel mixing, each convolution is further followed by a pointwise ( 1 × 1 ) convolution, incurring an extra cost of N 2 .
Revised: To compensate for reduced channel mixing, each convolution is further followed by a pointwise ( 1 × 1 ) convolution, incurring an extra cost of N 2 .

Original: For this block, each full convolution has been replaced with a grouped+pointwise convolution, and so the block uses 2(( N 2 k 2 ) /g
Revised: For this block, each full convolution has been replaced with a grouped+pointwise convolution, and so the block uses 2(( N 2 k 2 ) /g

Original: + N 2 ) parameters.
Revised: + N 2 ) parameters.

Original: Bottleneck Block – B(b)
Revised: Bottleneck Block – B(b)

Original: In this block, a pointwise convolution is used to reduce the number of channels of the input by a factor of b before a standard convolution is applied.
Revised: In this block, a pointwise convolution is used to reduce the number of channels of the input by a factor of b before a standard convolution is applied.

Original: Then, another pointwise convolution brings the channel size back up.
Revised: Then, another pointwise convolution brings the channel size back up.

Original: This uses ( N/b ) 2 k 2 + 2 N 2 /b parameters.
Revised: This uses ( N/b ) 2 k 2 + 2 N 2 /b parameters.

Original: Bottleneck Grouped+Pointwise Block – BG(b, g)
Revised: Bottleneck Grouped+Pointwise Block – BG(b, g)

Original: This is identical to the bottleneck block, except the standard convolution is further split into g groups, and so uses ( N/bg ) 2 k 2 + 2 N 2 /b parameters.
Revised: This is identical to the bottleneck block, except the standard convolution is further split into g groups, and so uses ( N/bg ) 2 k 2 + 2 N 2 /b parameters.

Original: 3. D ISTILLATION VIA A TTENTION T RANSFER
Revised: 3. D ISTILLATION VIA A TTENTION T RANSFER

Original: Attention transfer (Zagoruyko & Komodakis, 2017) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points are made to be similar to those produced by a large teacher network.
Revised: Attention transfer (Zagoruyko & Komodakis, 2017) is a distillation technique whereby a student network is trained such that its attention maps at several distinct attention points are made to be similar to those produced by a large teacher network.

Original: A formal deﬁnition follows: Consider a choice of layers i = 1 , 2 , ...,N L in a teacher network, and the corresponding layers in the student network.
Revised: A formal deﬁnition follows: Consider a choice of layers i = 1 , 2 , ..., L in a teacher network with L layers, and the corresponding layers in the student network.

Original: At each chosen layer i of the teacher network, collect the spatial map of the activations for channel j into the vector a tij .
Revised: At each chosen layer i of the teacher network, collect the spatial map of the activations for channel j into the vector a t ij .

Original: Let A ti collect a tij for all j .
Revised: Let A t i collect a t ij for all j .

Original: Likewise for the student network we correspondingly collect into a s ij and A s i .
Revised: Likewise for the student network we correspondingly collect into a sij and A si .

Original: Now given some choice of mapping f
Revised: Now given some choice of mapping f

Original: ( A i ) that maps each collection of the form
Revised: ( A i ) that maps each collection of the form

Original: A i into a vector, attention transfer involves learning the student network by minimising
Revised: A i into a vector, attention transfer involves learning the student network by minimising

Original: [Equation]
Revised: [Equation]

Original: where β is a hyperparameter, and L CE is the standard cross-entropy loss.
Revised: where β is a hyperparameter, and L CE is the standard cross-entropy loss.

Original: In Zagoruyko & Komodakis (2017) the authors use f ( A i ) =
Revised: In Zagoruyko & Komodakis (2017) the authors use f ( A i ) =

Original: (1 /N A i ) (cid:80) N
Revised: (1 /N A i ) (cid:80) N

Original: Ai j =1 a 2 ij , where N A
Revised: Ai j =1 a 2 ij , where N A

Original: i is the number of channels at layer i .
Revised: i is the number of channels at layer i .

Original: 3. F ISHER I NFORMATION
Revised: 3. F ISHER I NFORMATION

Original: Theis et al.
Revised: Theis et al.

Original: (2018) derive a second order approximation of the change in loss that would occur on the removal of a particular channel activation in a neural network.
Revised: (2018) derive a second order approximation of the change in loss that would occur on the removal of a particular channel activation in a neural network.

Original: They use this signal ∆ c to identify the least important activation channels, and remove their corresponding weights while performing channel pruning.
Revised: They use this signal ∆ c to identify the least important activation channels, and remove their corresponding weights while performing channel pruning.

Original: Formally, let us consider a single channel of an activation in a network due to some input minibatch of N examples.
Revised: Formally, let us consider a single channel of an activation in a network due to some input minibatch of N examples.

Original: Let us denote the values for this channel as A : a N × W × H tensor where W and H are the channel’s spatial width and height.
Revised: Let us denote the values for this channel as A : a N × W × H tensor where W and H are the channel’s spatial width and height.

Original: Let us refer to the entry corresponding to example n in the mini-batch at location ( i, j ) as A nij .
Revised: Let us refer to the entry corresponding to example n in the mini-batch at location ( i, j ) as A nij .

Original: If the network has a loss function L , then we can back-propagate to get the gradient of the loss with respect to this activation channel ∂ L ∂A .
Revised: If the network has a loss function L , then we can back-propagate to get the gradient of the loss with respect to this activation channel ∂ L ∂A .

Original: Let us denote this gradient as g and index it as g nij .
Revised: Let us denote this gradient as g and index it as g nij .

Original: ∆ c can then be computed by
Revised: ∆ c can then be computed by

Original: [Equation]
Revised: [Equation]

Original: Using an approximation to the Taylor expansion of the loss around an activation originated in LeCun et al.
Revised: Using an approximation to the Taylor expansion of the loss around an activation originated in LeCun et al.

Original: and has inspired many works in pruning (Hassibi & Stork, 1993; Molchanov et al., 2017;
Revised: and has inspired many works in pruning (Hassibi & Stork, 1993; Molchanov et al., 2017;

Original: Guo et al., 2016; Srinivas & Babu, 2015) and quantisation (Choi et al., 2017; Hou et al., 2017).
Revised: Guo et al., 2016; Srinivas & Babu, 2015) and quantisation (Choi et al., 2017; Hou et al., 2017).

Original:  M ETHOD
Revised:  M ETHOD

Original: Let us denote a large teacher network T composed of B blocks each of type S as T
Revised: Let us denote a large teacher network T composed of B blocks each of type S as T

Original: =
Revised: =

Original: [ S 1 , S 2 , ..., S B ] .
Revised: [ S 1 , S 2 , ..., S B ] .

Original: Each of these may be replaced by a cheap block C r , chosen from a list of candidates C 1 , C 2 , ..., C N of various representational capacities.
Revised: Each of these may be replaced by a cheap block C r , chosen from a list of candidates C 1 , C 2 , ..., C N of various representational capacities.

Original: We wish to construct a smaller model M =
Revised: We wish to construct a smaller model M =

Original: [ C r 1 , C r 2 , ..., C rB ] that is powerful, and within a given parameter budget.
Revised: [ C r 1 , C r 2 , ..., C rB ] that is powerful, and within a given parameter budget.

Original: But, the space of possible block conﬁgurations is very large ( ∼ 10 23 in Section 5).
Revised: But, the space of possible block conﬁgurations is very large ( ∼ 10 23 in Section 5).

Original: Even when using a cheap network evaluation strategy it is not possible to exhaustively search through the available network architectures.
Revised: Even when using a cheap network evaluation strategy it is not possible to exhaustively search through the available network architectures.

Original: What we require is a method to quickly propose and score possible conﬁgurations with as little training as possible.
Revised: What we require is a method to quickly propose and score possible conﬁgurations with as little training as possible.

Original: BlockSwap therefore relies on randomly sampling block conﬁgurations that satisfy a parameter budget, and  ranking the  samples by Fisher potential to decide which architectures to train.
Revised: BlockSwap therefore relies on randomly sampling block conﬁgurations that satisfy a parameter budget using rejection sampling—we generate architecture proposals at random and only save those that satisfy our parameter budget— and then ranking the saved samples by Fisher potential to decide which architectures to train.

Original: We obtain a score for each network as follows: we place probes after the last convolution in each block.
Revised: We obtain a score for each network as follows: we place probes after the last convolution in each block.

Original: When a single minibatch of training data is passed through the network and the resulting cross-entropy loss is backpropagated, the probe measures the Fisher potential of the block by summing ∆ c (Equation 2) for each channel in the layer it is placed after.
Revised: When a single minibatch of training data is passed through the network and the resulting cross-entropy loss is backpropagated, the probe measures the Fisher potential of the block by summing ∆ c (Equation 2) for each channel in the layer it is placed after.

Original: For this step, minibatch size is set equal to the size used during training.
Revised: For this step, minibatch size is set equal to the size used during training.

Original: We then sum the Fisher potential of each block to give us a score for the network.
Revised: We then sum the Fisher potential of each block to give us a score for the network.

Original: The intuition for this metric is as follows: the Fisher information measures the sensitivity of the loss to each parameter at initialisation.
Revised: The intuition for this metric is as follows: the Fisher information measures the sensitivity of the loss to each parameter at initialisation.

Original: Where parameters are insensitive at initialisation, they commonly remain insensitive as the other parameters are adjusted to adapt to the signal.
Revised: Where parameters are insensitive at initialisation, they commonly remain insensitive as the other parameters are adjusted to adapt to the signal.

Original: Hence the accumulated
Revised: Hence the accumulated

Original: Fisher potential measures the parameter wastage: how much of the parameter space is likely to be redundant for learning.
Revised: Fisher potential measures the parameter wastage: how much of the parameter space is likely to be redundant for learning.

Original: We wish to choose architectures which are efﬁcient (low parameter numbers) and have low wastage (use the parameters they have well).
Revised: We wish to choose architectures which are efﬁcient (low parameter numbers) and have low wastage (use the parameters they have well).

Original: Once we have scored each candidate architecture using the Fisher potential, we select the best one and train it using attention transfer.
Revised: Once we have scored each candidate architecture using the Fisher potential, we select the best one and train it using attention transfer.

Original: The training hyperparameters can be mirrored from T .
Revised: The training hyperparameters can be mirrored from T .

Original: We use the following blocks (deﬁned in Section 3.1) as candidate blocks:
Revised: We use the following blocks (deﬁned in Section 3.1) as candidate blocks:

Original: • B ( b ) for b ∈ { 2 , 4 } • G ( g ) for g ∈ { 2 , 4 , 8 , 16 , N/ 16 , N/ 8 , N/ 4 , N/ 2 , N } • BG (2 , g ) for g ∈ { 2 , 4 , 8 , 16 , M/ 16 , M/ 8 , M/ 4 , M/ 2 , M }
Revised: • B ( b ) for b ∈ { 2 , 4 } • G ( g ) for g ∈ { 2 , 4 , 8 , 16 , N/ 16 , N/ 8 , N/ 4 , N/ 2 , N } • BG (2 , g ) for g ∈ { 2 , 4 , 8 , 16 , M/ 16 , M/ 8 , M/ 4 , M/ 2 , M }

Original: where N is the number of channels in a block, and M is the number of channels after a bottleneck— this is N/ 2 when b = 2 .
Revised: where N is the number of channels in a block, and M is the number of channels after a bottleneck— this is N/ 2 when b = 2 .

Original: We also use the standard block S as a candidate choice, so as to not force blocks to reduce capacity where it is imperative.
Revised: We also use the standard block S as a candidate choice, so as to not force blocks to reduce capacity where it is imperative.

Original: 4. W HY MIX BLOCK TYPES ?
Revised: 4. W HY MIX BLOCK TYPES ?

Original: A reasonable question to ask is whether mixed-blocktype architectures will perform better than the use of a single blocktype.
Revised: A reasonable question to ask is whether mixed-blocktype architectures will perform better than the use of a single blocktype.

Original: As a simple check for this, we chose parameter budgets of 200K, 400K, and 800K and at each budget, compared single-blocktype models to one hundred randomly assembled, mixed-blocktype models.
Revised: As a simple check for this, we chose parameter budgets of 200K, 400K, and 800K and at each budget, compared single-blocktype models to one hundred randomly assembled, mixed-blocktype models.

Original: The models are WideResNets (Zagoruyko & Komodakis, 2016) with substituted blocks trained on CIFAR-10.
Revised: The models are WideResNets (Zagoruyko & Komodakis, 2016) with substituted blocks trained on CIFAR-10.

Original: [Figure]
Revised: [Figure]

Revised: 2,4) trains to 5.0% error, while the mean error of randomly constructed networks at the same parameter budget lies at around 5.15% error.

Revised: This implies that random architecture search will not work here, and that a better strategy is needed.

Original: In Figure 2 we plot the ﬁnal test error of the random architectures (blue densities) against their single-blocktype counterparts (the red dotted lines).
Revised: In Figure 2 we plot the ﬁnal test error of the random architectures (blue densities) against their single-blocktype counterparts (the red dotted lines).

Original: We observe that for each of the given parameter budgets there exist combinations of mixed blocks that are more powerful than their single blocktype equivalents.
Revised: We observe that for each of the given parameter budgets there exist combinations of mixed blocks that are more powerful than their single blocktype equivalents.

Original: However, single-blocktype networks typically sit below the mean of the density of the random networks.
Revised: However, single-blocktype networks typically sit below the mean of the density of the random networks.

Original: This suggests that, though random search has been shown to work well in architecture search settings (Sciuto et al., 2019), it will yield sub-optimal structures in this case.
Revised: This suggests that, though random search has been shown to work well in architecture search settings (Sciuto et al., 2019), it will yield sub-optimal structures in this case.

Original: 4. W HY USE A SINGLE MINIBATCH ? H OW DOES YOUR RANKING METRIC COMPARE ?
Revised: 4. W HY USE A SINGLE MINIBATCH ? H OW DOES YOUR RANKING METRIC COMPARE ?

Original: Though there are no common metrics for architecture ranking, neural architecture search methods often use information from early learning stages to evaluate possible candidates.
Revised: Though there are no common metrics for architecture ranking, neural architecture search methods often use information from early learning stages to evaluate possible candidates.

Original: In order to assess the fastest, most accurate indicator of ﬁnal error, we examine our 100 random mixed-blocktype networks for parameter budgets of 200K, 400K, and 800K and record how ﬁnal error correlates with (i) train accuracy, (ii) total (cid:96) 2 norms, (iii) summed absolute gradient norms (Golub et al., 2019), and (iv)
Revised: In order to assess the fastest, most accurate indicator of ﬁnal error, we examine our 100 random mixed-blocktype networks for parameter budgets of 200K, 400K, and 800K and record how ﬁnal error correlates with (i) train accuracy, (ii) total (cid:96) 2 norms, (iii) summed absolute gradient norms (Golub et al., 2019), and (iv)

Original: Fisher potential after 1, 10, and 100 minibatches of training in Table 1.
Revised: Fisher potential after 1, 10, and 100 minibatches of training in Table 1.

Original: We found that Fisher potential was by far the most robust metric, and that there were diminishing returns for adding more than a single minibatch.
Revised: We found that Fisher potential was by far the most robust metric, and that there were diminishing returns for adding more than a single minibatch.

Original: 4. H OW MANY SAMPLES ARE NEEDED ?
Revised: 4. H OW MANY SAMPLES ARE NEEDED ?

Original: Figure 2 suggests that sampling 100 random architectures will reliably yield at least one mixedblocktype network that outperforms a similarly parameterised single blocktype network.
Revised: Figure 2 suggests that sampling 100 random architectures will reliably yield at least one mixedblocktype network that outperforms a similarly parameterised single blocktype network.

Original: However, accepting that there is some noise to our ranking metric, we assume that we will need to take more than 100 samples in order to reliably detect powerful architectures.
Revised: However, accepting that there is some noise to our ranking metric, we assume that we will need to take more than 100 samples in order to reliably detect powerful architectures.

Original: As an illustration, at the budget of 400K parameters a single blocktype alternative has a test error of 4 . 45% , whereas BlockSwap ﬁnds networks with ﬁnal test errors of 4 . 85% . 4 .
Revised: As an illustration, at the budget of 400K parameters a single blocktype alternative has a test error of 4 . 45% , whereas BlockSwap ﬁnds networks with ﬁnal test errors of 4 . 85% . 4 .

Original: 54% , and 4 .
Revised: 54% , and 4 .

Original: 21% after 10, 100, and 1000 samples respectively.
Revised: 21% after 10, 100, and 1000 samples respectively.

Original: We empirically found that 1000 samples was enough to be robust to various parameter budgets on the tasks we considered.
Revised: We empirically found that 1000 samples was enough to be robust to various parameter budgets on the tasks we considered.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: W HAT DO “ GOOD NETWORKS ” LOOK LIKE ?
Revised: W HAT DO “ GOOD NETWORKS ” LOOK LIKE ?

Original: For each of the three parameter budgets (200K, 400K, 800K) we inspected the most common block choices for “good” and “bad” networks, where good and bad are networks with ﬁnal error greater two standard deviations below or above the mean respectively.
Revised: For each of the three parameter budgets (200K, 400K, 800K) we inspected the most common block choices for “good” and “bad” networks, where good and bad are networks with ﬁnal error greater two standard deviations below or above the mean respectively.

Original: We found that overwhelmingly, the types of blocks used in both good and bad networks was very similar, implying that it is the placement of the blocks instead of the types of the blocks that matters .
Revised: We found that overwhelmingly, the types of blocks used in both good and bad networks was very similar, implying that it is the placement of the blocks instead of the types of the blocks that matters .

Original: We examine this further in Appendix B
Revised: We examine this further in Appendix B

Original: [Table]
Revised: [Table]

Original:  CIFAR E XPERIMENTS
Revised:  CIFAR E XPERIMENTS

Original: Here, we evaluate student networks obtained using BlockSwap on the CIFAR-10 image classiﬁcation dataset (Krizhevsky, 2009).
Revised: Here, we evaluate student networks obtained using BlockSwap on the CIFAR-10 image classiﬁcation dataset (Krizhevsky, 2009).

Original: We benchmark these against competing student networks for a range of parameter budgets.
Revised: We benchmark these against competing student networks for a range of parameter budgets.

Original: To recapitulate, the BlockSwap networks are found by taking 1000 random samples from the space of possible block combinations that satisfy our constraint (in this case, parameter budget).
Revised: To recapitulate, the BlockSwap networks are found by taking 1000 random samples from the space of possible block combinations that satisfy our constraint (in this case, parameter budget).

Original: These points are ranked by Fisher potential after a single minibatch of training data, and the network with the highest potential is chosen.
Revised: These points are ranked by Fisher potential after a single minibatch of training data, and the network with the highest potential is chosen.

Original: The structure we use is that of a WideResNet (Zagoruyko & Komodakis, 2017), since we can use them to construct compact, high performance networks that are broadly representative of the most commonly used neural architectures.
Revised: The structure we use is that of a WideResNet (Zagoruyko & Komodakis, 2017), since we can use them to construct compact, high performance networks that are broadly representative of the most commonly used neural architectures.

Original: A WideResNet with depth 40, and width multiplier 2—WRN40-2—is trained and used as a teacher.
Revised: A WideResNet with depth 40, and width multiplier 2—WRN40-2—is trained and used as a teacher.

Original: It consists of 18 blocks and has 2.2 million parameters.
Revised: It consists of 18 blocks and has 2.2 million parameters.

Original: A student network is generated and is trained from scratch using attention transfer with the teacher.
Revised: A student network is generated and is trained from scratch using attention transfer with the teacher.

Original: Our BlockSwap students are WRN-40-2 nets where each of its 18 blocks are determined using our method with the blocks outlined in Section 4.
Revised: Our BlockSwap students are WRN-40-2 nets where each of its 18 blocks are determined using our method with the blocks outlined in Section 4.

Original: We compare against the following students:
Revised: We compare against the following students:

Original: Reduced width/depth versions of the teacher: WRN-16-1,WRN-16-2, WRN-402.
Revised: Reduced width/depth versions of the teacher: WRN-16-1,WRN-16-2, WRN-402.

Original: Single block-type teacher reductions: every block in the teacher is swapped out for (i) the MBConv6 block from MobileNet v2 (Sandler et al., 2018) and (ii) the block discovered in DARTS (Liu et al., 2019a).
Revised: Single block-type teacher reductions: every block in the teacher is swapped out for (i) the MBConv6 block from MobileNet v2 (Sandler et al., 2018) and (ii) the block discovered in DARTS (Liu et al., 2019a).

Original: Pruned teacher reductions: we compare against (i) the magnitude-based pruning methodology from Han et al.
Revised: Pruned teacher reductions: we compare against (i) the magnitude-based pruning methodology from Han et al.

Original: (2016) and (ii) SNIP
Revised: (2016) and (ii) SNIP

Original: (Lee et al., 2019) versions of the teacher.
Revised: (Lee et al., 2019) versions of the teacher.

Original: We additionally compare against CondenseNet-86 (Huang et al., 2018), distilled with the Born-Again strategy
Revised: We additionally compare against CondenseNet-86 (Huang et al., 2018), distilled with the Born-Again strategy

Original: described by
Revised: described by

Original: Furlanello et al.
Revised: Furlanello et al.

Original: First, we train three teacher networks independently.
Revised: First, we train three teacher networks independently.

Original: These are used to train all of our students; each student network is trained three times, once with each of these teachers.
Revised: These are used to train all of our students; each student network is trained three times, once with each of these teachers.

Original: Figure 3 shows the mean test errors of BlockSwap students at various parameter counts, compared to the alternatives listed above.
Revised: Figure 3 shows the mean test errors of BlockSwap students at various parameter counts, compared to the alternatives listed above.

Original: Full results with standard deviations are listed in Table 2, along with the number of
Revised: Full results with standard deviations are listed in Table 2, along with the number of

Original: Multiply-Accumulate (MAC) operations each network uses.
Revised: Multiply-Accumulate (MAC) operations each network uses.

Original: Our results show that block cheapening is more effective than simple downscaling schemes (reducing width/depth); not only are the reduced models (WRN-40-1, WRN-16-2, WRN-16-1) inﬂexible in parameter range, they perform signiﬁcantly worse than our models at the parameter budgets they can map to.
Revised: Our results show that block cheapening is more effective than simple downscaling schemes (reducing width/depth); not only are the reduced models (WRN-40-1, WRN-16-2, WRN-16-1) inﬂexible in parameter range, they perform signiﬁcantly worse than our models at the parameter budgets they can map to.

Original: We also show that BlockSwap ﬁnds more accurate networks than the other top-down approaches ( (cid:96) 1 -pruning and SNIP) across the full parameter spectrum. Note that we are being
Revised: We also show that BlockSwap ﬁnds more accurate networks than the other top-down approaches ( (cid:96) 1 -pruning and SNIP) across the full parameter spectrum. Note that as (cid:96) 1 -pruning and

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: BlockSwap 811.4 132.5 3.79 ± 0.
Revised: BlockSwap [Figure] [Figure]

Original: 40-2 BlockSwap 556.
Revised: BlockSwap 556.

Original: [Figure]
Revised: [Figure]

Original: 4.17 ± 0.
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: BlockSwap 404.
Revised: BlockSwap 404.

Original: 4.21 ± 0.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: BlockSwap 289.
Revised: BlockSwap 289.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2 BlockSwap217.
Revised: BlockSwap

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: BlockSwap 162.
Revised: BlockSwap 162.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: also outperform single blocktype networks (

Original: [Figure] particularly generous to the sparse methods by counting only non-zero parameters (the weight tensors remain the same size as the original teacher).
Revised: SNIP introduce unstructured sparsity, the parameter counts provided are the number of non-zero parameters (the weight tensors remain the same size as the original teacher).

Original: The mixed block architectures that BlockSwap generates are more accurate at these reduced budgets than all of the single blocktype alternatives we considered.
Revised: The mixed block architectures that BlockSwap generates are more accurate at these reduced budgets than all of the single blocktype alternatives we considered.

Original: While the performance of DARTS is very similar to BlockSwap, it is worth noting that the DARTS architecture search required 24 GPU hours, while BlockSwap needs only 8 minutes on a single CPU.
Revised: While the performance of DARTS is very similar to BlockSwap, it is worth noting that the DARTS architecture search required 24 GPU hours, while BlockSwap needs only 8 minutes on a single CPU.

Original: Given the strength of random baselines in architecture search settings (Liu et al., 2019a; Sciuto et al., 2019; Li & Talwalkar, 2019), we also compare BlockSwap against randomly generated block conﬁgurations in Appendix C. BlockSwap consistently outperforms these, proving that our aggregated
Revised: Given the strength of random baselines in architecture search settings (Liu et al., 2019a; Sciuto et al., 2019; Li & Talwalkar, 2019), we also compare BlockSwap against randomly generated block conﬁgurations in Appendix C. BlockSwap consistently outperforms these, proving that our aggregated

Original: Fisher potential metric is an effective means for choosing optimal block structures.
Revised: Fisher potential metric is an effective means for choosing optimal block structures.

Original: Implementation 
Revised: Implementation Details: Networks are trained for 200 epochs using SGD with momentum 0.9. The initial learning rate of 0.1 is reduced by a factor of 5 every 60 epochs. Minibatches of size 128 are used with standard crop + ﬂip data augmentation and Cutout (DeVries & Taylor, 2017). The weight decay factor is set to 0.0005.

Revised: For attention transfer β is set to 1000.

Original:  I MAGE N ET C LASSIFICATION
Revised:  I MAGE N ET C LASSIFICATION

Original: Here, we demonstrate that students chosen by BlockSwap succeed on the more challenging ImageNet dataset (Russakovsky et al., 2015).
Revised: Here, we demonstrate that students chosen by BlockSwap succeed on the more challenging ImageNet dataset (Russakovsky et al., 2015).

Original: We use a pretrained ResNet-34 (16 blocks) as a teacher, and compare students at two parameter budgets (3M and 8M).
Revised: We use a pretrained ResNet-34 (16 blocks) as a teacher, and compare students at two parameter budgets (3M and 8M).

Original: We train a Blockswap student at each of these budgets and compare their validation errors to those of a reduced depth/width student (ResNet and ResNet-18-0.5—a ResNet-18 where the channel width in the last 3 sections has been halved) and a single-blocktype student (ResNet-34 with G(4) and G(N) blocks).
Revised: We train a Blockswap student at each of these budgets and compare their validation errors to those of a reduced depth/width student (ResNet and ResNet-18-0.5—a ResNet-18 where the channel width in the last 3 sections has been halved) and a single-blocktype student (ResNet-34 with G(4) and G(N) blocks).

Original: The student networks found by
Revised: The student networks found by

Original: BlockSwap for these two budgets are illustrated in Appendix D. Top-1 and top-5 validation errors are presented in Table 3.
Revised: BlockSwap for these two budgets are illustrated in Appendix D. Top-1 and top-5 validation errors are presented in Table 3.

Original: At both budgets, BlockSwap chooses networks that outperform its comparators.
Revised: At both budgets, BlockSwap chooses networks that outperform its comparators.

Original: At 8M parameters it even surpasses the teacher by quite a margin.
Revised: At 8M parameters it even surpasses the teacher by quite a margin.

Original: Speciﬁcally, it beats the teacher by 0.49% in top-1 error and 0.82% in top-5 error despite using almost 3 × fewer parameters.
Revised: Speciﬁcally, it beats the teacher by 0.49% in top-1 error and 0.82% in top-5 error despite using almost 3 × fewer parameters.

Original: Implementation Details: Networks are trained with a cross-entropy loss for 100 epochs using SGD with momentum 0.9.
Revised: Implementation Details: Networks are trained with a cross-entropy loss for 100 epochs using SGD with momentum 0.9.

Original: The initial learning rate of 0.1 is reduced by 10 × every 30 epochs.
Revised: The initial learning rate of 0.1 is reduced by 10 × every 30 epochs.

Original: Minibatches of size 256 are used with standard crop + ﬂip augmentation.
Revised: Minibatches of size 256 are used with standard crop + ﬂip augmentation.

Original: The weight decay factor is set to 0.0001.
Revised: The weight decay factor is set to 0.0001.

Original: For attention transfer β is set to 750 using the output of each of the four sections of network.
Revised: For attention transfer β is set to 750 using the output of each of the four sections of network.

Original: [Table]
Revised: [Table]

Original:  COCO D ETECTION
Revised:  COCO D ETECTION

Original: Thus far
Revised: Thus far

Original: [Table]
Revised: [Table]

Original: we have used BlockSwap for image classiﬁcation problems.
Revised: we have used BlockSwap for image classiﬁcation problems.

Original: Here we observe whether it extends to object detection on the COCO dataset (Lin et al., 2014)—speciﬁcally, training ontrain
Revised: Here we observe whether it extends to object detection on the COCO dataset (Lin et al., 2014)—speciﬁcally, training ontrain

Original: [Table]
Revised: [Table]

Original: and evaluating on 2017 val.
Revised: and evaluating on 2017 val.

Original: We consider a Mask R-CNN (He et al., 2017) with a ResNetbackbone
Revised: We consider a Mask R-CNN (He et al., 2017) with a ResNetbackbone

Original: [Table]
Revised: [Table]

Original: and apply BlockSwap using COCO images to obtain a mixed-blocktype backbone with 3M parameters.
Revised: and apply BlockSwap using COCO images to obtain a mixed-blocktype backbone with 3M parameters.

Original: We compare this to a single-blocktype ResNet-34-G(N) backbone which uses the same number of parameters.
Revised: We compare this to a single-blocktype ResNet-34-G(N) backbone which uses the same number of parameters.

Original: To avoid conﬂation with ImageNet, we train everything from scratch .
Revised: To avoid conﬂation with ImageNet, we train everything from scratch .

Original: The results can be found in Table 4.
Revised: The results can be found in Table 4.

Original: We can see that the BlockSwap backbone again outperforms its single-blocktype counterpart.
Revised: We can see that the BlockSwap backbone again outperforms its single-blocktype counterpart.

Original: Implementation Details: Networks are trained using the default Mask R-CNN settings in Torchvision.
Revised: Implementation Details: Networks are trained using the default Mask R-CNN settings in Torchvision.

Original: We use a batch-size of 16 split across 8 GPUs.
Revised: We use a batch-size of 16 split across 8 GPUs.

Original: All models are trained from scratch, and we forgo distillation due to memory constraints.
Revised: All models are trained from scratch, and we forgo distillation due to memory constraints.

Original: [Table]
Revised: [Table]

Original: C ONCLUSION
Revised: C ONCLUSION

Original: We have developed BlockSwap: a fast, simple method for reducing large neural networks to ﬂexible parameter targets based on block substitution.
Revised: We have developed BlockSwap: a fast, simple method for reducing large neural networks to ﬂexible parameter targets based on block substitution.

Original: We have veriﬁed that these reduced networks make for excellent students, and have performed a comprehensive ablation study.
Revised: We have veriﬁed that these reduced networks make for excellent students, and have performed a comprehensive ablation study.

Original: Future work could use
Revised: Future work could use

Original: BlockSwap to choose networks based on inference time, or energy cost instead of parameter count.
Revised: BlockSwap to choose networks based on inference time, or energy cost instead of parameter count.

Original: A B LOCK R EPLACEMENTS
Revised: A B LOCK R EPLACEMENTS

Original: Where applicable, g is the number of groups and b is the bottleneck contraction convolutional and batch-norm (BN) costs are given, although the latter is signiﬁcantly smaller.
Revised: Where applicable, g is the number of groups and b is the bottleneck contraction convolutional and batch-norm (BN) costs are given, although the latter is signiﬁcantly smaller.

Original: BN
Revised: BN

Original: ReLU precedes
Revised: ReLU precedes

Original: [Table]
Revised: [Table]

Original: B U NDERSTANDING B LOCK P LACEMENT
Revised: B U NDERSTANDING B LOCK P LACEMENT

Original: To understand what makes for “good” block placement
Revised: To understand what makes for “good” block placement

Original: [Table]
Revised: [Table]

Original: we begin with the more tractable question: if we were only to cheapen one block, which one should we choose?
Revised: we begin with the more tractable question: if we were only to cheapen one block, which one should we choose?

Original: In Figure 4 we show what happens when we substitute a single standard block with G(4) for each of the possible blocks for our
Revised: In Figure 4 we show what happens when we substitute a single standard block with G(4) for each of the possible blocks for our

Original: WideResNet.
Revised: WideResNet.

Original: We plot the ﬁnal test error after training to convergence, as well as the total parameter count after the substitution (the higher the bar, the higher the error, therefore the more sensitive the block).
Revised: We plot the ﬁnal test error after training to convergence, as well as the total parameter count after the substitution (the higher the bar, the higher the error, therefore the more sensitive the block).

Original: Notice that in Figure 4 (a), substituting the start and end of each stage appears to have a very detrimental effect on accuracy, whereas mid-stage blocks can be substituted without hurting performance.
Revised: Notice that in Figure 4 (a), substituting the start and end of each stage appears to have a very detrimental effect on accuracy, whereas mid-stage blocks can be substituted without hurting performance.

Original: For example, block 12 has few parameters, as shown in Figure 4(b), but substituting it for a cheaper alternative results in a drastic drop in accuracy.
Revised: For example, block 12 has few parameters, as shown in Figure 4(b), but substituting it for a cheaper alternative results in a drastic drop in accuracy.

Original: This implies that it is safest to substitute blocks in the middle of a stage but that blocks at the start and end of each stage should have their representational capacity preserved as much as possible.
Revised: This implies that it is safest to substitute blocks in the middle of a stage but that blocks at the start and end of each stage should have their representational capacity preserved as much as possible.

Original: We used this insight to analyse the block conﬁgurations of “good” and “bad” amongst the random architectures at each parameter budget.
Revised: We used this insight to analyse the block conﬁgurations of “good” and “bad” amongst the random architectures at each parameter budget.

Original: With several blocktypes, however, we need a method to rank the capacity of blocktypes; we make the simplifying assumption that training the blocktype on its own is a good indicator of this (though this is not always true, since blocks give varying performance with different input sizes).
Revised: With several blocktypes, however, we need a method to rank the capacity of blocktypes; we make the simplifying assumption that training the blocktype on its own is a good indicator of this (though this is not always true, since blocks give varying performance with different input sizes).

Original: An example of a good and bad network judged this way can be seen in
Revised: An example of a good and bad network judged this way can be seen in

Original: Figure 5; while they do roughly follow our observations, there are clearly more complex interactions at play (perhaps unsurprising given the random nature of their construction).
Revised: Figure 5; while they do roughly follow our observations, there are clearly more complex interactions at play (perhaps unsurprising given the random nature of their construction).

Original: We found that overall, the models in our ablation set adhere to the pattern described above (for each stage: high capacity blocks at the start and end, increasing capacity throughout), with some variance.
Revised: We found that overall, the models in our ablation set adhere to the pattern described above (for each stage: high capacity blocks at the start and end, increasing capacity throughout), with some variance.

Original: [Figure]
Revised: [Figure]

Original: ﬁnal error.
Revised: ﬁnal error.

Original: [Figure]
Revised: [Figure]

Original: (a)
Revised: (a)

Original: C a p ac it y S
Revised: C a p ac it y S

Original: c
Revised: c

Original: o r e
Revised: o r e

Original: Bad model (5.25% error)
Revised: Bad model (5.25% error)

Original: [Figure]
Revised: [Figure]

Original: (b)
Revised: (b)

Original: C a p ac it y S
Revised: C a p ac it y S

Original: c
Revised: c

Original: o r e
Revised: o r e

Original: Good model (4.62% error)
Revised: Good model (4.62% error)

Original: C C OMPARING TO RANDOM CONFIGURATIONS
Revised: C C OMPARING TO RANDOM CONFIGURATIONS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 32 ±
Revised: 32 ±

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: .93 ±
Revised: .93 ±

Original: [Table]
Revised: [Table]

Original: D B LOCK S WAP N ETWORKS FOR I MAGE N ET
Revised: D B LOCK S WAP N ETWORKS FOR I MAGE N ET

Original: [Figure]
Revised: [Figure]