Original: B ATCH -S HAPING FOR L EARNING C ONDITIONAL C HANNEL G ATED N ETWORKS
Revised: B ATCH -S HAPING FOR L EARNING C ONDITIONAL C HANNEL G ATED N ETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a method that trains large capacity neural networks with signiﬁcantly improved accuracy and lower dynamic computational cost.
Revised: We present a method that trains large capacity neural networks with signiﬁcantly improved accuracy and lower dynamic computational cost.

Original: We achieve this by gating the deep-learning architecture on a ﬁne-grained-level.
Revised: We achieve this by gating the deep-learning architecture on a ﬁne-grained-level.

Original: Individual convolutional maps are turned on/off conditionally on features in the network.
Revised: Individual convolutional maps are turned on/off conditionally on features in the network.

Original: To achieve this, we introduce a new residual block architecture that gates convolutional channels in a ﬁne-grained manner.
Revised: To achieve this, we introduce a new residual block architecture that gates convolutional channels in a ﬁne-grained manner.

Original: We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-speciﬁed prior distribution.
Revised: We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-speciﬁed prior distribution.

Original: We use this novel technique to force gates to be more conditional on the data.
Revised: We use this novel technique to force gates to be more conditional on the data.

Original: We present results on CIFAR-10 and ImageNet datasets for image classiﬁcation, and Cityscapes for semantic segmentation.
Revised: We present results on CIFAR-10 and ImageNet datasets for image classiﬁcation, and Cityscapes for semantic segmentation.

Original: Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.
Revised: Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy.

Original: In particular, on ImageNet, our ResNet50 and ResNet gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.
Revised: In particular, on ImageNet, our ResNet50 and ResNet gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity.

Original: We also show that the resulting networks automatically learn to use more features for difﬁcult examples and fewer features for simple examples.
Revised: We also show that the resulting networks automatically learn to use more features for difﬁcult examples and fewer features for simple examples.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Almost all deep neural networks have a prior that seems suboptimal: All features are calculated all the time.
Revised: Almost all deep neural networks have a prior that seems suboptimal: All features are calculated all the time.

Original: Both from a generalization and an inference-time perspective, this is superﬂuous.
Revised: Both from a generalization and an inference-time perspective, this is superﬂuous.

Original: For example, there is no reason to compute features that help differentiate between several dog breeds, if there is no dog to be seen in the image.
Revised: For example, there is no reason to compute features that help differentiate between several dog breeds, if there is no dog to be seen in the image.

Original: The necessity of speciﬁc features for classiﬁcation performance depends on other features.
Revised: The necessity of speciﬁc features for classiﬁcation performance depends on other features.

Original: We can make use of this natural prior information, to improve our neural networks.
Revised: We can make use of this natural prior information, to improve our neural networks.

Original: We can also exploit this to spend less computational power on simple and more on complicated examples.
Revised: We can also exploit this to spend less computational power on simple and more on complicated examples.

Original: This general idea is commonly encapsulated in the terms conditional computing (Bengio, 2013) or gating architectures (Sigaud et al., 2016).
Revised: This general idea is commonly encapsulated in the terms conditional computing (Bengio, 2013) or gating architectures (Sigaud et al., 2016).

Original: It is known that models with increased capacity, for example increased model depth (He et al., 2016) or width (Zagoruyko & Komodakis, 2016), generally help increase model performance when properly regularized.
Revised: It is known that models with increased capacity, for example increased model depth (He et al., 2016) or width (Zagoruyko & Komodakis, 2016), generally help increase model performance when properly regularized.

Original: However, as models increase in size, so do their training and inference times.
Revised: However, as models increase in size, so do their training and inference times.

Original: This often limits practical applications of deep learning.
Revised: This often limits practical applications of deep learning.

Original: By conditionally turning parts of the architecture on or off we can train networks with very large capacity while keeping the computational overhead small.
Revised: By conditionally turning parts of the architecture on or off we can train networks with very large capacity while keeping the computational overhead small.

Original: The hypothesis is that when training conditionally gated networks, we can train models with a better accuracy/computation cost trade-off than their fully feed-forward counterparts.
Revised: The hypothesis is that when training conditionally gated networks, we can train models with a better accuracy/computation cost trade-off than their fully feed-forward counterparts.

Original: In addition, conditional computation with channel gating can potentially increase the interpretability of models.
Revised: In addition, conditional computation with channel gating can potentially increase the interpretability of models.

Original: For example, gating can allow us to identify input patterns that trigger the response of a single unit in the network.
Revised: For example, gating can allow us to identify input patterns that trigger the response of a single unit in the network.

Original: Several works in recent literature that successfully learn conditional architectures, for example,
Revised: Several works in recent literature that successfully learn conditional architectures, for example,

Original: ConvNet-AIG (Veit & Belongie, 2018) and dynamic channel pruning (Gao et al., 2018).
Revised: ConvNet-AIG (Veit & Belongie, 2018) and dynamic channel pruning (Gao et al., 2018).

Original: However, the conditionality is often very coarse as in ConvNet-AIG
Revised: However, the conditionality is often very coarse as in ConvNet-AIG

Original: (Veit & Belongie, 2018), or the amount of actual conditional features learned is very minimal as in Gaternet (Chen et al., 2018).
Revised: (Veit & Belongie, 2018), or the amount of actual conditional features learned is very minimal as in Gaternet (Chen et al., 2018).

Original: We attempt to solve both.
Revised: We attempt to solve both.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We propose a ﬁne-grained gating architecture that turns individual input and output convolutional maps on or off, leading to features that are individually gated.
Revised: • We propose a ﬁne-grained gating architecture that turns individual input and output convolutional maps on or off, leading to features that are individually gated.

Original: This allows for a better trade-off between computation cost and accuracy than previous work.
Revised: This allows for a better trade-off between computation cost and accuracy than previous work.

Original: • We propose a generally applicable tool named batch-shaping that matches the marginal aggregated posterior of a feature in the network to a speciﬁed prior.
Revised: • We propose a generally applicable tool named batch-shaping that matches the marginal aggregated posterior of a feature in the network to a speciﬁed prior.

Original: Depending on the chosen prior, networks can match activation distributions to e.g. the uniform distribution for better quantization, or the Gaussian to enforce behavior similar to batch-normalization (Ioffe & Szegedy, 2015).
Revised: Depending on the chosen prior, networks can match activation distributions to e.g. the uniform distribution for better quantization, or the Gaussian to enforce behavior similar to batch-normalization (Ioffe & Szegedy, 2015).

Original: Speciﬁcally, in this paper, we apply batch-shaping to help the network learn conditional features.
Revised: Speciﬁcally, in this paper, we apply batch-shaping to help the network learn conditional features.

Original: We show that this helps performance by controlling the gates to be more conditional on the input data at the end of training.
Revised: We show that this helps performance by controlling the gates to be more conditional on the input data at the end of training.

Original: • We show state-of-the-art results compared to other conditional computing architectures such as Convnet-AIG (Veit & Belongie, 2018), SkipNet (Wang et al., 2018a), Dynamic Channel Pruning(Gao et al., 2018), and soft-guided adaptively-dropped neural network
Revised: • We show state-of-the-art results compared to other conditional computing architectures such as Convnet-AIG (Veit & Belongie, 2018), SkipNet (Wang et al., 2018a), Dynamic Channel Pruning(Gao et al., 2018), and soft-guided adaptively-dropped neural network

Original: (Wang et al., 2018b).
Revised: (Wang et al., 2018b).

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Literature on gating connections for deep neural networks dates back to Hinton (1981).
Revised: Literature on gating connections for deep neural networks dates back to Hinton (1981).

Original: Gating can be seen as a tri-way connection in a neural network (Droniou et al., 2015), where one output can only be 0 and 1.
Revised: Gating can be seen as a tri-way connection in a neural network (Droniou et al., 2015), where one output can only be 0 and 1.

Original: These connections have originally been used to learn transformations between images with gated Restricted Boltzmann Machines as in Memisevic & Hinton (2007).
Revised: These connections have originally been used to learn transformations between images with gated Restricted Boltzmann Machines as in Memisevic & Hinton (2007).

Original: One of the earliest works to apply this to create sparse network architectures is that of a Mixture of Experts proposed by
Revised: One of the earliest works to apply this to create sparse network architectures is that of a Mixture of Experts proposed by

Original: Jacobs et al.
Revised: Jacobs et al.

Original: Several compression methods exist that statically reduce model complexity.
Revised: Several compression methods exist that statically reduce model complexity.

Original: Tensor factorization methods (Jaderberg et al., 2014; Zhang et al., 2016) decompose single layers into two more efﬁcient bottleneck layers.
Revised: Tensor factorization methods (Jaderberg et al., 2014; Zhang et al., 2016) decompose single layers into two more efﬁcient bottleneck layers.

Original: Methods such as channel-pruning (He et al., 2017; Molchanov et al., 2017) remove entire input/output channels from the network.
Revised: Methods such as channel-pruning (He et al., 2017; Molchanov et al., 2017) remove entire input/output channels from the network.

Original: Similarly, full channels can be removed during training as in VIBnets (Dai et al., 2018), Bayesian Compression (Louizos et al., 2017) and L0-regularization (Louizos et al., 2018).
Revised: Similarly, full channels can be removed during training as in VIBnets (Dai et al., 2018), Bayesian Compression (Louizos et al., 2017) and L0-regularization (Louizos et al., 2018).

Original: These methods reduce the overall model capacity while keeping the accuracy as high as possible.
Revised: These methods reduce the overall model capacity while keeping the accuracy as high as possible.

Original: Our method allows for higher model capacity while keeping inference times similar to the papers cited above.
Revised: Our method allows for higher model capacity while keeping inference times similar to the papers cited above.

Original: Some networks exploit the complexity of each input example to gain performance.
Revised: Some networks exploit the complexity of each input example to gain performance.

Original: Networks such as Branchynet (Teerapittayanon et al., 2016) and Multi-scale dense net (Huang et al., 2018) have early exiting nodes, where complex examples proceed deeper in the network than simpler ones.
Revised: Networks such as Branchynet (Teerapittayanon et al., 2016) and Multi-scale dense net (Huang et al., 2018) have early exiting nodes, where complex examples proceed deeper in the network than simpler ones.

Original: Our approach also assigns less computation to simpler examples than more complicated examples but has no early-exiting paths.
Revised: Our approach also assigns less computation to simpler examples than more complicated examples but has no early-exiting paths.

Original: Both methods of inducing less computation can work in tandem, which we leave for future work.
Revised: Both methods of inducing less computation can work in tandem, which we leave for future work.

Original: Several works focus on adaptive spatial attention for faster inference.
Revised: Several works focus on adaptive spatial attention for faster inference.

Original: Figurnov et al.
Revised: Figurnov et al.

Original: (2017) proposed a residual network based model that dynamically adjusts the number of executed layers for different regions of the image.
Revised: (2017) proposed a residual network based model that dynamically adjusts the number of executed layers for different regions of the image.

Original: Difﬁculty-aware region convolutions were proposed (Li et al., 2017) as an irregular convolution, which allow operating the convolution on speciﬁc regions of a feature map.
Revised: Difﬁculty-aware region convolutions were proposed (Li et al., 2017) as an irregular convolution, which allow operating the convolution on speciﬁc regions of a feature map.

Original: Similarly, tiling-based sparse convolution algorithm was proposed (Ren et al., 2018) which yields a signiﬁcant speed-up in terms of wall-clock time.
Revised: Similarly, tiling-based sparse convolution algorithm was proposed (Ren et al., 2018) which yields a signiﬁcant speed-up in terms of wall-clock time.

Original: Other works exploit similar conditional sparsity properties of networks as this work.
Revised: Other works exploit similar conditional sparsity properties of networks as this work.

Original: ConvNetAIG
Revised: ConvNetAIG

Original: (Veit & Belongie, 2018) and SkipNet (Wang et al., 2018a) turn full residual blocks on or off, conditionally dependent on the input.
Revised: (Veit & Belongie, 2018) and SkipNet (Wang et al., 2018a) turn full residual blocks on or off, conditionally dependent on the input.

Original: Dynamic Channel Pruning (Gao et al., 2018) turns individual features on or off similar to our approach, but they choose the top-k features instead, akin to Outrageously large neural networks (Shazeer et al., 2017).
Revised: Dynamic Channel Pruning (Gao et al., 2018) turns individual features on or off similar to our approach, but they choose the top-k features instead, akin to Outrageously large neural networks (Shazeer et al., 2017).

Original: This approach loses the beneﬁt of being able to trade-off compute for simple and complex examples.
Revised: This approach loses the beneﬁt of being able to trade-off compute for simple and complex examples.

Original: Gaternet (Chen et al., 2018) trains a completely separate network to gate each individual channel of the main network.
Revised: Gaternet (Chen et al., 2018) trains a completely separate network to gate each individual channel of the main network.

Original: The overhead of this network is not necessary for learning effective gates, and we show better conditionality of the gates than is achieved by this paper, at almost no overhead.
Revised: The overhead of this network is not necessary for learning effective gates, and we show better conditionality of the gates than is achieved by this paper, at almost no overhead.

Original:  B ATCH - SHAPING
Revised:  B ATCH - SHAPING

Original: First, we introduce batch-shaping , a general method to match the marginal aggregated posterior distribution over any feature in the neural network to a pre-speciﬁed prior distribution.
Revised: First, we introduce batch-shaping , a general method to match the marginal aggregated posterior distribution over any feature in the neural network to a pre-speciﬁed prior distribution.

Original: In the next paragraph, we will use this to train gates that ﬁre more conditionally, but we see a large potential value of this tool for many other applications such as training auto-encoders or network quantization.
Revised: In the next paragraph, we will use this to train gates that ﬁre more conditionally, but we see a large potential value of this tool for many other applications such as training auto-encoders or network quantization.

Original: Consider a parameterized feature in a neural network X ( θ ) .

Original: The intention is to have X ( θ ) distributed more according to a chosen probability density function (PDF) f

Original: ( x ) , while still being able

Original: to

Original: F(x i* )-F N (x i* )
Revised: F(x i* )-F N (x i* )

Original: Empirical CDF - F N (X)
Revised: Empirical CDF - F N (X)

Original: Prior Beta CDF - F(X) 0.
Revised: Prior Beta CDF - F(X) 0.

Original: (a) 0.
Revised: (a) 0.

Revised: E po c h (

Original: E po c h (b) differentiate with respect to θ .
Revised: [Figure] Consider a parameterized feature in a neural network X ( θ ) . The intention is to have X ( θ ) distributed more according to a chosen probability density function (PDF) f ( x ) , while still being able to differentiate with respect to θ .

Original: To do this we consider the Cramér-von-Mises criterion by Anderson (1962), which is a statistical distance between the cumulative distribution function F ( x ) and an empirical cumulative distribution function F N ( x ) .
Revised: To do this we consider the Cramér-von-Mises criterion by Anderson (1962), which is a statistical distance between the cumulative distribution function F ( x ) and an empirical cumulative distribution function F N ( x ) .

Original: The Cramér-von-Mises criterion lends itself naturally to this use-case.
Revised: The Cramér-von-Mises criterion lends itself naturally to this use-case.

Original: Other frequently used statistical distance functions, such as KL-divergence, require calculating a histogram of the samples to compare to f ( x ) , which does not allow for gradients to propagate.
Revised: Other frequently used statistical distance functions, such as KL-divergence, require calculating a histogram of the samples to compare to f ( x ) , which does not allow for gradients to propagate.

Original: As we will see, we can derive gradients with respect to each sample x with the proposed loss function.
Revised: As we will see, we can derive gradients with respect to each sample x with the proposed loss function.

Original: The Cramér-von-Mises criterion is given by:
Revised: The Cramér-von-Mises criterion is given by:

Original: [Equation]
Revised: [Equation]

Original: We consider batches of N samples x 1: N drawn from X ( θ ) .
Revised: We consider batches of N samples x 1: N drawn from X ( θ ) .

Original: In order to optimize this we follow
Revised: In order to optimize this we follow

Original: Anderson (1962).
Revised: Anderson (1962).

Original: Sorting sort ( x 1: N ) =
Revised: Sorting sort ( x 1: N ) =

Original: x ∗ 1: N , replacing d F ( x ) with d F N ( x ) and normalizing with
Revised: x ∗ 1: N , replacing d F ( x ) with d F N ( x ) and normalizing with

Original: N gives us the batch-shaping loss to minimize
Revised: N gives us the batch-shaping loss to minimize

Original: [Equation]
Revised: [Equation]

Original: where λ is a parameter that controls the strength of the loss function.
Revised: where λ is a parameter that controls the strength of the loss function.

Original: This approach is shown visually in Figure 1a.
Revised: This approach is shown visually in Figure 1a.

Original: We can sum this loss for each considered feature in the network to attain a full network batch-shaping loss.
Revised: We can sum this loss for each considered feature in the network to attain a full network batch-shaping loss.

Original: Note that we can differentiate x ∗ 1: N with respect to θ through the sorting operator by keeping the sorted indices.
Revised: Note that we can differentiate x ∗ 1: N with respect to θ through the sorting operator by keeping the sorted indices.

Original: In the backward pass, if a value with index i was sorted to index j , we put the error from position j in position i .
Revised: In the backward pass, if a value with index i was sorted to index j , we put the error from position j in position i .

Original: This makes the whole loss term differentiable as long as the chosen CDF function is differentiable.
Revised: This makes the whole loss term differentiable as long as the chosen CDF function is differentiable.

Original: The pseudo-code for the implementation of the Batch-Shaping loss is presented in the Appendix A.
Revised: The pseudo-code for the implementation of the Batch-Shaping loss is presented in the Appendix A.

Original: We can use this loss to match the marginal aggregated posterior of a feature in the network to any
Revised: We can use this loss to match the marginal aggregated posterior of a feature in the network to any

Original: PDF.
Revised: PDF.

Original: For example, if we want to encourage our activations to be Gaussian, we could use the CDF of the Gaussian in the loss function.
Revised: For example, if we want to encourage our activations to be Gaussian, we could use the CDF of the Gaussian in the loss function.

Original: This could be useful for purposes similar to batch-normalization.
Revised: This could be useful for purposes similar to batch-normalization.

Original: Alternatively, the CDF can be that of the uniform distribution, which could help with ﬁxed-point quantization Jacob et al.
Revised: Alternatively, the CDF can be that of the uniform distribution, which could help with ﬁxed-point quantization Jacob et al.

Original: We leave this for future work, and only consider a loss function that encourages the conditionality of gates in the next section.
Revised: We leave this for future work, and only consider a loss function that encourages the conditionality of gates in the next section.

Original:  C HANNEL GATED NETWORKS
Revised:  C HANNEL GATED NETWORKS

Original: In this section we introduce our gating network.
Revised: In this section we introduce our gating network.

Original: While the basic structure of our gating network could be any kind of CNN structure, we use ResNet (He et al., 2016) as the basic structure.
Revised: While the basic structure of our gating network could be any kind of CNN structure, we use ResNet (He et al., 2016) as the basic structure.

Original: Figureshows an overview of a channel gated ResNet block.
Revised: Figureshows an overview of a channel gated ResNet block.

Original: Formally, a ResNet building block is deﬁned as:
Revised: Formally, a ResNet building block is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where x l ∈ R c l × w l × h l , and x l +1 ∈ R c l +1 × w l +1 × h l +1 denote the input and output of the residual block, and r is the activation function, in this case a ReLU (Nair & Hinton, 2010) function. The
Revised: C o n v + bn

Original: C o n v + bnC o n v + bn + G 
Revised: C o n v + bn + G 𝐻 ×𝑊 ×𝐶

Original: [Figure]
Revised: [Figure]

Original:  𝐴𝑣𝑒. 𝑝𝑜𝑜𝑙𝑖𝑛𝑔 1×1×𝐶 2𝑙𝑎𝑦𝑒𝑟 𝑀𝐿𝑃 1×1×𝐶′ + 1×1×𝐶′
Revised: map 𝐴𝑣𝑒. 𝑝𝑜𝑜𝑙𝑖𝑛𝑔 1×1×𝐶 2𝑙𝑎𝑦𝑒𝑟 𝑀𝐿𝑃 1×1×𝐶′ + 1×1×𝐶′

Original: Gating [Figure]
Revised: Gating module where x l ∈ R c l × w l × h l , and x l +1 ∈ R c l +1 × w l +1 × h l +1 denote the input and output of the residual block, and r is the activation function, in this case a ReLU (Nair & Hinton, 2010) function.

Original:  residual function F ( x l ) is the residual mapping to be learned and is deﬁned as F = W 2 ∗ r
Revised: The residual function F ( x l ) is the residual mapping to be learned and is deﬁned as F = W 2 ∗ r

Original: ( W 1 ∗ x ) .
Revised: ( W 1 ∗ x ) .

Original: Here ∗ denotes the convolution operator.
Revised: Here ∗ denotes the convolution operator.

Original: W 1 ∈ R c l × c 1 l +1 × k × k is a set of c 1 l +1 ﬁlters, with each ﬁlter of size k × k .
Revised: W 1 ∈ R c l × c 1 l +1 × k × k is a set of c 1 l +1 ﬁlters, with each ﬁlter of size k × k .

Original: Similarly, W 2 ∈ R c 1 l +1 × c l +1 × k × k .
Revised: Similarly, W 2 ∈ R c 1 l +1 × c l +1 × k × k .

Original: After each convolution layer, batch normalization (Ioffe & Szegedy, 2015) is used.
Revised: After each convolution layer, batch normalization (Ioffe & Szegedy, 2015) is used.

Original: Our gated residual block is deﬁned as:
Revised: Our gated residual block is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where G is a gating module and G ( x l )
Revised: where G is a gating module and G ( x l )

Original: =
Revised: =

Original: [ g 1 , g 2 , g 3 , · · · , g c 1 l +1 ] is the output of the gating function, where g c ∈ { 0 , 1 } :
Revised: [ g 1 , g 2 , g [Equation] [Equation]

Original:  denotes skipping the convolution operation for ﬁlter c in W 1 , and 1 denotes computing the convolution.
Revised: 1 ] is the output of the gating function, where g c ∈ { 0 , 1 } : 0 denotes skipping the convolution operation for ﬁlter c in W 1 , and 1 denotes computing the convolution.

Original: Here · refers to channel-wise multiplication between the output feature map r ( W 1 ∗ x ) and the vector G ( x l ) .
Revised: Here · refers to channel-wise multiplication between the output feature map r ( W 1 ∗ x ) and the vector G ( x l ) .

Original: The more sparse the output of G ( x l ) , the more computation we can potentially save.
Revised: The more sparse the output of G ( x l ) , the more computation we can potentially save.

Original: We chose the position of the gate for two speciﬁc reasons.
Revised: We chose the position of the gate for two speciﬁc reasons.

Original: Firstly, the gate is applied after the ReLU activation.
Revised: Firstly, the gate is applied after the ReLU activation.

Original: This prevents the convolution from updating if the gate is off.
Revised: This prevents the convolution from updating if the gate is off.

Original: Placing the gating function before the ReLU caused unstable training behavior.
Revised: Placing the gating function before the ReLU caused unstable training behavior.

Original: Secondly, we only gate the representation between the two layers in the residual block.
Revised: Secondly, we only gate the representation between the two layers in the residual block.

Original: We allow each block to use the full input and update the full output.
Revised: We allow each block to use the full input and update the full output.

Original: The network only gates each feature that determines how to update the incoming representation.
Revised: The network only gates each feature that determines how to update the incoming representation.

Original: We tested applying multiple gating setups, including before and after each convolutional layer.
Revised: We tested applying multiple gating setups, including before and after each convolutional layer.

Original: The proposed setup performed signiﬁcantly better.
Revised: The proposed setup performed signiﬁcantly better.

Original: 4. G ATING MODULE
Revised: 4. G ATING MODULE

Original: To enable a light-weight gating module design, we squeeze global spatial information in x l into a channel descriptor as input to our gating module, similar to ConvNet-AIG and Squeeze-and-excitation nets (Veit & Belongie, 2018;
Revised: To enable a light-weight gating module design, we squeeze global spatial information in x l into a channel descriptor as input to our gating module, similar to ConvNet-AIG and Squeeze-and-excitation nets (Veit & Belongie, 2018;

Original: Hu et al., 2018).
Revised: Hu et al., 2018).

Original: This is achieved via channel-wise global average pooling.
Revised: This is achieved via channel-wise global average pooling.

Original: For our gating module, we use a simple feed-forward design comprising of two fully connected layers, with only 16 neurons in the hidden layer.
Revised: For our gating module, we use a simple feed-forward design comprising of two fully connected layers, with only 16 neurons in the hidden layer.

Original: We apply batch normalization and ReLU on the output of the ﬁrst fully connected layer.
Revised: We apply batch normalization and ReLU on the output of the ﬁrst fully connected layer.

Original: The second fully connected layer linearly projects the features to unnormalized probabilities ˆ π k , k ∈ (cid:8) 1 , 2 , · · · , c 1 l +1 (cid:9) .
Revised: The second fully connected layer linearly projects the features to unnormalized probabilities ˆ π k , k ∈ (cid:8) 1 , 2 , · · · , c 1 l +1 (cid:9) .

Original: We deﬁne the logits ˆ π k
Revised: We deﬁne the logits ˆ π k

Original: = ln
Revised: = ln

Original: ( π k ) .
Revised: ( π k ) .

Original: Our gating module is computationally inexpensive and has an additional overhead that is between 0 .
Revised: Our gating module is computationally inexpensive and has an additional overhead that is between 0 .

Original: 018% − 0 .
Revised: 018% − 0 .

Original: 087% of a ResNet block multiply-accumulate (MAC) usage.
Revised: 087% of a ResNet block multiply-accumulate (MAC) usage.

Original: To dynamically select a subset of ﬁlters relevant for our current input, we need to map the output of our gating module to a binary vector.
Revised: To dynamically select a subset of ﬁlters relevant for our current input, we need to map the output of our gating module to a binary vector.

Original: The task of training binary-valued gates is challenging because we can not directly back-propagate through a non-differentiable gate.
Revised: The task of training binary-valued gates is challenging because we can not directly back-propagate through a non-differentiable gate.

Original: In this paper, we leverage a recently proposed approach called Gumbel-Softmax sampling (Jang et al., 2017; Maddison et al., 2017) to circumvent this problem.
Revised: In this paper, we leverage a recently proposed approach called Gumbel-Softmax sampling (Jang et al., 2017; Maddison et al., 2017) to circumvent this problem.

Original: We consider the binary case of the Gumbel-Max trick, the Binary concrete relaxation BinConcrete ( π, τ ) .
Revised: We consider the binary case of the Gumbel-Max trick, the Binary concrete relaxation BinConcrete ( π, τ ) .

Original: In the forward pass we use the discrete argmax and for the backward pass we use a sigmoid function with temperature: σ τ ( x ) = σ ( xτ ) .
Revised: In the forward pass we use the discrete argmax and for the backward pass we use a sigmoid function with temperature: σ τ ( x ) = σ ( x τ ) .

Original: We use τ = 2 / 3 in all of our experiments as suggested by Maddison et al.
Revised: We use τ = 2 / 3 in all of our experiments as suggested by Maddison et al.

Original: 4. B ATCH - SHAPING BETA DISTRIBUTION PRIOR FOR CONDITIONAL GATES
Revised: 4. B ATCH - SHAPING BETA DISTRIBUTION PRIOR FOR CONDITIONAL GATES

Original: When initially training the channel-wise gating architecture, many features were trained to be only on or off in the ﬁrst few epochs.
Revised: When initially training the channel-wise gating architecture, many features were trained to be only on or off in the ﬁrst few epochs.

Original: Instead, we would like a feature to be sometimes on and sometimes off for different data points, to exploit the potential for conditionality.
Revised: Instead, we would like a feature to be sometimes on and sometimes off for different data points, to exploit the potential for conditionality.

Original: We regulate this by applying the batch-shaping loss, with the CDF of a Beta distribution I x ( a, b ) , as a prior on each of the gates.
Revised: We regulate this by applying the batch-shaping loss, with the CDF of a Beta distribution I x ( a, b ) , as a prior on each of the gates.

Original: We set a = 0 .
Revised: We set a = 0 .

Original: 6 and b = 0 .
Revised: 6 and b = 0 .

Original: 4 in our experiments, initially inducing 40% sparsity.
Revised: 4 in our experiments, initially inducing 40% sparsity.

Original: The Beta distribution will regularize gates towards being sometimes on and sometimes off for different data points, pushing the gates towards the desired batch-wise conditionality.
Revised: The Beta distribution will regularize gates towards being sometimes on and sometimes off for different data points, pushing the gates towards the desired batch-wise conditionality.

Original: We apply this loss with a strong coefﬁcient λ at the start of training to encourage activations to be conditional, and gradually anneal λ as training progresses.
Revised: We apply this loss with a strong coefﬁcient λ at the start of training to encourage activations to be conditional, and gradually anneal λ as training progresses.

Original: This allows the network to learn different levels of sparsity or even undo the effect of batch-shaping if necessary.
Revised: This allows the network to learn different levels of sparsity or even undo the effect of batch-shaping if necessary.

Original: Figure 1b presents the output distribution of four gates during training.
Revised: Figure 1b presents the output distribution of four gates during training.

Original: 4. L 0 - LOSS
Revised: 4. L 0 - LOSS

Original: Our batch-shaping loss encourages the network to learn more conditional features.
Revised: Our batch-shaping loss encourages the network to learn more conditional features.

Original: However, our actual intention is to ﬁnd a model that has the best trade-off between (conditional) sparsity and our task loss (e.g., cross-entropy loss).
Revised: However, our actual intention is to ﬁnd a model that has the best trade-off between (conditional) sparsity and our task loss (e.g., cross-entropy loss).

Original: For the second part of our training procedure, we add a loss that regularizes the complexity of the full network explicitly.
Revised: For the second part of our training procedure, we add a loss that regularizes the complexity of the full network explicitly.

Original: We use a method proposed by Louizos et al.
Revised: We use a method proposed by Louizos et al.

Original: (2018), which deﬁnes a L 0 regularization process for neural network sparsiﬁcation by learning a set of gates with a sparsifying regularizer.
Revised: (2018), which deﬁnes a L 0 regularization process for neural network sparsiﬁcation by learning a set of gates with a sparsifying regularizer.

Original: Our work can be considered as a conditional version of the Lgates introduced in this paper, sans stretching parameters.
Revised: Our work can be considered as a conditional version of the Lgates introduced in this paper, sans stretching parameters.

Original: Hence this loss term is a natural choice for sparsifying the activations.
Revised: Hence this loss term is a natural choice for sparsifying the activations.

Original: We use a modiﬁed version of the L 0 -loss without stretching, deﬁned as:
Revised: We use a modiﬁed version of the L 0 -loss without stretching, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where k is the total number of gates, σ is the sigmoid function, and γ is a parameter that controls the level of sparsiﬁcation we want to achieve.
Revised: where k is the total number of gates, σ is the sigmoid function, and γ is a parameter that controls the level of sparsiﬁcation we want to achieve.

Original: It is essential to mention that introducing this loss too early in training can reduce effective network capacity and potentially hurt performance.
Revised: It is essential to mention that introducing this loss too early in training can reduce effective network capacity and potentially hurt performance.

Original: If the training procedure is not carefully chosen, the procedure often degenerates into training a smaller architecture, as full convolutional maps are turned off prematurely.
Revised: If the training procedure is not carefully chosen, the procedure often degenerates into training a smaller architecture, as full convolutional maps are turned off prematurely.

Original: Thus, in all our experiments, we introduce this L 0 -loss after some delay, and we use a warm-up schedule as described in Sønderby et al.
Revised: Thus, in all our experiments, we introduce this L 0 -loss after some delay, and we use a warm-up schedule as described in Sønderby et al.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate the performance of our method on two image classiﬁcation benchmarks: CIFAR-
Revised: We evaluate the performance of our method on two image classiﬁcation benchmarks: CIFAR-

Original: (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015).
Revised: (Krizhevsky, 2009) and ImageNet (Russakovsky et al., 2015).

Original: We additionally report preliminary results on the Cityscapes semantic segmentation benchmark (Cordts et al., 2016).
Revised: We additionally report preliminary results on the Cityscapes semantic segmentation benchmark (Cordts et al., 2016).

Original: For CIFAR-10, we use ResNet20 and ResNet32 architectures (He et al., 2016) as our base model.
Revised: For CIFAR-10, we use ResNet20 and ResNet32 architectures (He et al., 2016) as our base model.

Original: For ImageNet, we use ResNet18, ResNet34, and ResNet50.
Revised: For ImageNet, we use ResNet18, ResNet34, and ResNet50.

Original: We compare our algorithm with competitive conditional computation methods.
Revised: We compare our algorithm with competitive conditional computation methods.

Original: We additionally perform experiments to understand the learning patterns of the gates and whether they specialize to certain categories.
Revised: We additionally perform experiments to understand the learning patterns of the gates and whether they specialize to certain categories.

Original: For semantic segmentation, we employ the pyramid scene parsing network (PSPNet) (Zhao et al., 2017) with ResNet-50 backbone.
Revised: For semantic segmentation, we employ the pyramid scene parsing network (PSPNet) (Zhao et al., 2017) with ResNet-50 backbone.

Original: The training details and hyperparameters for our gated networks trained on CIFAR10, ImageNet, and
Revised: The training details and hyperparameters for our gated networks trained on CIFAR10, ImageNet, and

Original: Cityscapes are provided in the appendix B. The base networks and the gates were trained together from scratch for all of our models.
Revised: Cityscapes are provided in the appendix B. The base networks and the gates were trained together from scratch for all of our models.

Original: CIFAR-10:
Revised: CIFAR-10:

Original: We applied the batch-shaping loss with beta distribution prior from the start of training with a coefﬁcient of λ = 0 .
Revised: We applied the batch-shaping loss with beta distribution prior from the start of training with a coefﬁcient of λ = 0 .

Original: 75 and linearly annealed it to zero until epoch 100.
Revised: 75 and linearly annealed it to zero until epoch 100.

Original: Next, the L 0 -loss was applied to the output of the gates starting from epoch 100, and the coefﬁcient was linearly increased until epoch 300 where it was kept ﬁxed for the rest of the training.
Revised: Next, the L 0 -loss was applied to the output of the gates starting from epoch 100, and the coefﬁcient was linearly increased until epoch 300 where it was kept ﬁxed for the rest of the training.

Original: For the L 0 -loss we used γ values of { 0 , 1 , 2 , 5 , 10 , 15 , 20 } · 10 − 2 to generate different trade-off points.
Revised: For the L 0 -loss we used γ values of { 0 , 1 , 2 , 5 , 10 , 15 , 20 } · 10 − 2 to generate different trade-off points.

Original: We also experimented with only using the batch-shaping loss (no L 0 -loss) with a ﬁxed λ = 0 .
Revised: We also experimented with only using the batch-shaping loss (no L 0 -loss) with a ﬁxed λ = 0 .

Original: 75 for the entire training.
Revised: 75 for the entire training.

Original: We compare our batch-shaped channel gated ResNet20 and ResNet32 models, hereafter referred to as ResNet20-BAS and ResNet32-BAS, with other adaptive computation methods: ConvNetAIG (Veit & Belongie, 2018), SkipNet (Wang et al., 2018a), and SGAD (Wang et al., 2018b).
Revised: We compare our batch-shaped channel gated ResNet20 and ResNet32 models, hereafter referred to as ResNet20-BAS and ResNet32-BAS, with other adaptive computation methods: ConvNetAIG (Veit & Belongie, 2018), SkipNet (Wang et al., 2018a), and SGAD (Wang et al., 2018b).

Original: As shown in Figure 3a, ResNet20-BAS and ResNet32-BAS outperform SkipNet38, SGAD-ResNet and ConvNet-AIG variants of ResNet20 and ResNet32 by a large margin.
Revised: As shown in Figure 3a, ResNet20-BAS and ResNet32-BAS outperform SkipNet38, SGAD-ResNet and ConvNet-AIG variants of ResNet20 and ResNet32 by a large margin.

Original: Our results show that given a deep network such as ResNet32, we can reduce the average computation (conditioned on the input) to a value equal or lower than that of a ResNet20 architecture and still achieve better performance.
Revised: Our results show that given a deep network such as ResNet32, we can reduce the average computation (conditioned on the input) to a value equal or lower than that of a ResNet20 architecture and still achieve better performance.

Original: Ideally, a gated ResNet32 model should outperform a gated ResNet20 model at the  same average computation.
Revised: Ideally, a gated ResNet32 model should outperform a gated ResNet20 model at the [Figure] same average computation.

Original: This property is evident in our results.
Revised: This property is evident in our results.

Original: However, the competing methods show performance equal to or lower than their smaller sized counterparts, indicating one may instead train a smaller model from scratch.
Revised: However, the competing methods show performance equal to or lower than their smaller sized counterparts, indicating one may instead train a smaller model from scratch.

Original: [Figure] The highest accuracy obtained by ResNet20-BAS and ResNet32-BAS in Figure 3a relates to the case where we only used the batch-shaping loss with a ﬁxed λ .
Revised: The highest accuracy obtained by ResNet20-BAS and ResNet32-BAS in Figure 3a relates to the case where we only used the batch-shaping loss with a ﬁxed λ .

Original: Despite the high accuracy, this setting leads to a minor reduction in computation costs.
Revised: Despite the high accuracy, this setting leads to a minor reduction in computation costs.

Original: We also experimented with changing the batch-shaping distribution to be more sparse (e.g. changing prior over time).
Revised: We also experimented with changing the batch-shaping distribution to be more sparse (e.g. changing prior over time).

Original: However, the performance degraded signiﬁcantly.
Revised: However, the performance degraded signiﬁcantly.

Original: The L 0 -loss, in contrast, provided a better trade-off between accuracy and MAC saving.
Revised: The L 0 -loss, in contrast, provided a better trade-off between accuracy and MAC saving.

Original: ImageNet: To evaluate the performance of our models on a larger dataset, we applied our gating networks to ImageNet.
Revised: ImageNet: To evaluate the performance of our models on a larger dataset, we applied our gating networks to ImageNet.

Original: Similar to CIFAR classiﬁcation, we introduce the batch-shaping loss from the start of the training with λ = 0 .
Revised: Similar to CIFAR classiﬁcation, we introduce the batch-shaping loss from the start of the training with λ = 0 .

Original: 75 and linearly annealed it to zero until epoch 20.
Revised: 75 and linearly annealed it to zero until epoch 20.

Original: L 0 -loss was then applied to the output of the gates starting from epoch 30, and the coefﬁcient was linearly increased until epoch 60 where it was kept ﬁxed for the rest of the training.
Revised: L 0 -loss was then applied to the output of the gates starting from epoch 30, and the coefﬁcient was linearly increased until epoch 60 where it was kept ﬁxed for the rest of the training.

Original: We used γ values of { 0 , 1 , 2 , 5 , 10 , 15 , 20 , 30 , 40 }· 10 − 2 to generate different trade-off points.
Revised: We used γ values of { 0 , 1 , 2 , 5 , 10 , 15 , 20 , 30 , 40 }· 10 − 2 to generate different trade-off points.

Original: We also experimented with only using the batch-shaping loss with a ﬁxed λ = 0 .
Revised: We also experimented with only using the batch-shaping loss with a ﬁxed λ = 0 .

Original: 75 for the entire training.
Revised: 75 for the entire training.

Original: Compared to CIFAR-10, the performance difference with the baseline is larger for ImageNet, likely because of the more substantial complexity of the dataset allowing for more conditionality of the features.
Revised: Compared to CIFAR-10, the performance difference with the baseline is larger for ImageNet, likely because of the more substantial complexity of the dataset allowing for more conditionality of the features.

Original: We also see an increased performance for lower ‘compression rates’, similar to what is frequently seen in compression literature because of extra regularization, e.g., as in Frankle & Carbin (2019).
Revised: We also see an increased performance for lower ‘compression rates’, similar to what is frequently seen in compression literature because of extra regularization, e.g., as in Frankle & Carbin (2019).

Original: Figure 3b shows the trade-off between the computation cost and Top-1 accuracy for our gated network,
Revised: Figure 3b shows the trade-off between the computation cost and Top-1 accuracy for our gated network,

Original: ConvNet-AIG, and ConvNet-FBS (Feature Boosting and Suppression)
Revised: ConvNet-AIG, and ConvNet-FBS (Feature Boosting and Suppression)

Original: (Gao et al., 2018).
Revised: (Gao et al., 2018).

Original: The results indicate that our ResNet-BAS models consistently outperform corresponding ConvNet-AIG and
Revised: The results indicate that our ResNet-BAS models consistently outperform corresponding ConvNet-AIG and

Original: ConvNet-FBS models.
Revised: ConvNet-FBS models.

Original: Similar to the observations on CIFAR-10, the performance of ConvNet-AIG degrades to a level lower than that of ConvNet-AIG18, at the same computation cost.
Revised: Similar to the observations on CIFAR-10, the performance of ConvNet-AIG degrades to a level lower than that of ConvNet-AIG18, at the same computation cost.

Original: Our models, in contrast, make better use of dynamic allocation of features by learning more conditional features.
Revised: Our models, in contrast, make better use of dynamic allocation of features by learning more conditional features.

Original: Subsequently, when the average computation cost is on par with ResNet18, our ResNet50-BAS and
Revised: Subsequently, when the average computation cost is on par with ResNet18, our ResNet50-BAS and

Original: ResNet34-BAS gated networks achieved 74.40% and 72.55% top-1 accuracy compared to the 70.57% best accuracy of ResNet18-BAS and 69.76% accuracy of the baseline ResNet18 model.
Revised: ResNet34-BAS gated networks achieved 74.40% and 72.55% top-1 accuracy compared to the 70.57% best accuracy of ResNet18-BAS and 69.76% accuracy of the baseline ResNet18 model.

Original: Similar to CIFAR10 experiments, the highest accuracy obtained by batch-shaped models in Figure 3b relates to the case where we only used the batch-shaping loss with a ﬁxed λ .
Revised: Similar to CIFAR10 experiments, the highest accuracy obtained by batch-shaped models in Figure 3b relates to the case where we only used the batch-shaping loss with a ﬁxed λ .

Original: Semantic segmentation: For this experiment, we only used our batch-shaping loss with a ﬁxed coefﬁcient.
Revised: Semantic segmentation: For this experiment, we only used our batch-shaping loss with a ﬁxed coefﬁcient.

Original: The original PSP network achieves an overall IoU (intersection over union) of 0.
Revised: The original PSP network achieves an overall IoU (intersection over union) of 0.

Original: with a pixel-level accuracy of 0.929 on the validation set.
Revised: with a pixel-level accuracy of 0.929 on the validation set.

Original: Our gated PSPNet model was able to accomplish an IoU of 0.719 and pixel accuracy of 0.935 while using 76.3% of the MAC count ( λ = 0 . 2 ) of the original PSP model.
Revised: Our gated PSPNet model was able to accomplish an IoU of 0.719 and pixel accuracy of 0.935 while using 76.3% of the MAC count ( λ = 0 . 2 ) of the original PSP model.

Original: We additionally compared the models when starting training using ImageNet-pretrained weights to initialize the ResNet-50 base network.
Revised: We additionally compared the models when starting training using ImageNet-pretrained weights to initialize the ResNet-50 base network.

Original: In this setting, PSPNet achieved an IoU of 0.739 and pixel accuracy of 0.9446.
Revised: In this setting, PSPNet achieved an IoU of 0.739 and pixel accuracy of 0.9446.

Original: Our gated-model, in comparison, obtained an
Revised: Our gated-model, in comparison, obtained an

Original: IoU of 0.744 and pixel accuracy of 0.946 using 76.5% of the PSPNet MAC count ( λ = 0 . 2 ).
Revised: IoU of 0.744 and pixel accuracy of 0.946 using 76.5% of the PSPNet MAC count ( λ = 0 . 2 ).

Original: The performance of our gated network further reaches an IoU of 0.747 and pixel accuracy of 0.948 using 95% of the PSPNet MAC count ( λ = 0 . 05 ).
Revised: The performance of our gated network further reaches an IoU of 0.747 and pixel accuracy of 0.948 using 95% of the PSPNet MAC count ( λ = 0 . 05 ).

Original: Figure 4 shows example images from the cityscapes dataset consuming the highest and lowest MAC counts.
Revised: Figure 4 shows example images from the cityscapes dataset consuming the highest and lowest MAC counts.

Original: 5. E FFECT OF INCREASING WIDTH OF THE NETWORK
Revised: 5. E FFECT OF INCREASING WIDTH OF THE NETWORK

Original: To assess the effect of increasing the network’s width rather than depth [Table] we trained ResNet20 models on CIFAR10 with per-block width increased with factors of 10X and 20X.
Revised: To assess the effect of increasing the network’s width rather than depth, we trained ResNet20 models on CIFAR10 with per-block width increased with factors of 10X and 20X.

Original: The results are shown in Figure 3c.
Revised: The results are shown in Figure 3c.

Original: As can be seen, by strongly increasing the number of parameters of the network and allowing for dynamic selection of such parameters, the network consistently achieves higher accuracy.
Revised: As can be seen, by strongly increasing the number of parameters of the network and allowing for dynamic selection of such parameters, the network consistently achieves higher accuracy.

Original: 5. M ODEL INFERENCE - TIME COMPARISON
Revised: 5. M ODEL INFERENCE - TIME COMPARISON

Original: We compared the inference time of our models to Convnet-AIG
Revised: We compared the inference time of our models to Convnet-AIG

Original: (Veit & Belongie, 2018) on CPU in the same setting (see Table 1) for batch-size of one.
Revised: (Veit & Belongie, 2018) on CPU in the same setting (see Table 1) for batch-size of one.

Original: For batch-computing, a custom convolutional kernel could calculate a mask on the ﬂy.
Revised: For batch-computing, a custom convolutional kernel could calculate a mask on the ﬂy.

Original: We simulated this for the GPU in the same table.
Revised: We simulated this for the GPU in the same table.

Original: Custom hardware would give us beneﬁts from all MACs saved, including energy savings.
Revised: Custom hardware would give us beneﬁts from all MACs saved, including energy savings.

Original: See appendix C for further details of our timing setup
Revised: See appendix C for further details of our timing setup.

Original: [Table]
Revised: [Table]

Original: 5. E FFECT OF THE BATCH - SHAPING LOSS
Revised: 5. E FFECT OF THE BATCH - SHAPING LOSS

Original: To validate the effectiveness of our proposed batch-shaping loss
Revised: To validate the effectiveness of our proposed batch-shaping loss

Original: [Table]
Revised: [Table]

Original: we compare the performance of our
Revised: we compare the performance of our

Original: ResNet-BAS networks in two settings: 1) using both the batch-shaping loss and the L 0 complexity loss for training the model similar to the experiments above, or 2) only using the L 0 complexity loss. As can be seen in Figure 3c
Revised: ResNet-BAS networks in two settings: 1) using both the batch-shaping loss and the L 0 complexity loss for training the model similar to the experiments above

Original: [Table]
Revised: [Table]

Original:  the models that additionally use the batch-shaping loss consistently outperform the ones using only the L 0 complexity loss.
Revised: or 2) only using the L 0 complexity loss. As can be seen in Figure 3c, the models that additionally use the batch-shaping loss consistently outperform the ones using only the L 0 complexity loss.

Original: ResNet20-L0 and ResNet32-L0 trained on CIFAR10 appear to have more rapid accuracy degradation than the models trained using the batch-shaping loss.
Revised: ResNet20-L0 and ResNet32-L0 trained on CIFAR10 appear to have more rapid accuracy degradation than the models trained using the batch-shaping loss.

Original: However, our L 0 -gated models still outperform ConvNet-AIG and SkipNet architectures.
Revised: However, our L 0 -gated models still outperform ConvNet-AIG and SkipNet architectures.

Original: This can be attributed to our speciﬁc channel gated network architecture which allows for ﬁne-grained dynamic selection of channels or ﬁlters in a layer, as compared to the models which are designed to skip whole layers.
Revised: This can be attributed to our speciﬁc channel gated network architecture which allows for ﬁne-grained dynamic selection of channels or ﬁlters in a layer, as compared to the models which are designed to skip whole layers.

Original: Very importantly, as shown in Figure 3c, by gating the larger non-BAS ResNet32/38 models, we do not see any improvement over the smaller ResNet20 model  at a similar computation cost.
Revised: Very importantly, as shown in Figure 3c, by gating the larger non-BAS ResNet32/38 models, we do not see any improvement over the smaller ResNet20 model [Figure] [Figure] at a similar computation cost.

Original: This is in sharp contrast to our ResNet34-BAS model.
Revised: This is in sharp contrast to our ResNet34-BAS model.

Original: We observe a similar pattern in ImageNet results (See Figure 8 in appendix D).
Revised: We observe a similar pattern in ImageNet results (See Figure 8 in appendix D)

Original: 5. G ATE DISTRIBUTION
Revised: 5. G ATE DISTRIBUTION

Original: Analyzing the distribution of the learned gates gives us a better insight into the characteristics of the learned features of the network.
Revised: Analyzing the distribution of the learned gates gives us a better insight into the characteristics of the learned features of the network.

Original: Ideally, we expect three main gate distributions to appear in[Figure] [Figure] a network with dynamic computations: 1) Gates that are always on.
Revised: Ideally, we expect three main gate distributions to appear in a network with dynamic computations: 1) Gates that are always on.

Original: We expect certain ﬁlters in a network to be of key importance for all types of inputs.
Revised: We expect certain ﬁlters in a network to be of key importance for all types of inputs.

Original: 2) Gates that ﬁre conditionally on/off based on the input.
Revised: 2) Gates that ﬁre conditionally on/off based on the input.

Original: The ﬁlters that are more input dependent and hence are more specialized for certain categories can be dynamically selected to be executed based on the input.
Revised: The ﬁlters that are more input dependent and hence are more specialized for certain categories can be dynamically selected to be executed based on the input.

Original: These types of ﬁlters are desirable as they contribute to saving computation at the inference phase and formation of conditional expert sub-networks inside our network.
Revised: These types of ﬁlters are desirable as they contribute to saving computation at the inference phase and formation of conditional expert sub-networks inside our network.

Original: Therefore, we would like to maximize the learning of such ﬁlters. 3)
Revised: Therefore, we would like to maximize the learning of such ﬁlters. 3)

Original: Gates that are always off.
Revised: Gates that are always off.

Original: This introduces complete sparsity.
Revised: This introduces complete sparsity.

Original: Figure 5 shows the distribution of gates on the ImageNet validation set for our ResNet34-BAS and ResNet34-L0 models.
Revised: Figure 5 shows the distribution of gates on the ImageNet validation set for our ResNet34-BAS and ResNet34-L0 models.

Original: We observe that the majority of the gates activate conditionally for our ResNet34-BAS model.
Revised: We observe that the majority of the gates activate conditionally for our ResNet34-BAS model.

Original: This model prefers conditional sparsity over fully turning gates off.
Revised: This model prefers conditional sparsity over fully turning gates off.

Original: ResNet34-L0 model achieves the same sparsity level by fully turning off a large number of gates.
Revised: ResNet34-L0 model achieves the same sparsity level by fully turning off a large number of gates.

Original:  Figure 6 presents an illustration of the ﬁne-grained execution of ﬁlters in our gated networks for different image categories.
Revised: [Figure] Figure 6 presents an illustration of the ﬁne-grained execution of ﬁlters in our gated networks for different image categories.

Original: As can be seen, the histograms show small differences in gate execution patterns between similar classes, and large differences between distinct classes.
Revised: As can be seen, the histograms show small differences in gate execution patterns between similar classes, and large differences between distinct classes.

Original: Figure 7 shows categories that trigger individual gates in different layers of ResNet34-BAS network.
Revised: Figure 7 shows categories that trigger individual gates in different layers of ResNet34-BAS network.

Original: Overall, we observe that there are gates at different layers of the network that are activated for a more[Figure] speciﬁc subset of categories.
Revised: Overall, we observe that there are gates at different layers of the network that are activated for a more speciﬁc subset of categories.

Original: For example, Figure 7a shows that the activation of a gate in the ﬁfth
Revised: For example, Figure 7a shows that the activation of a gate in the ﬁfth

Original: ResNet Block depends on the appearance of large creatures in the water.
Revised: ResNet Block depends on the appearance of large creatures in the water.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented a ﬁne-grained gating architecture that enables conditional computation in deep networks.
Revised: In this paper, we presented a ﬁne-grained gating architecture that enables conditional computation in deep networks.

Original: Our gating network achieves state-of-the-art accuracy among competing conditional computation architectures on CIFAR10 and ImageNet datasets.
Revised: Our gating network achieves state-of-the-art accuracy among competing conditional computation architectures on CIFAR10 and ImageNet datasets.

Original: In both datasets, given a model with large capacity, our gating method was the only approach that could reduce the inference computation to a value equal or lower than that of a lower capacity base network, while obtaining a higher accuracy.
Revised: In both datasets, given a model with large capacity, our gating method was the only approach that could reduce the inference computation to a value equal or lower than that of a lower capacity base network, while obtaining a higher accuracy.

Original: On ImageNet, our ResNet50-BAS and ResNet34-BAS improve the accuracy by more than 4.8% and 2.8% over a ResNet18 model at the same computation cost.
Revised: On ImageNet, our ResNet50-BAS and ResNet34-BAS improve the accuracy by more than 4.8% and 2.8% over a ResNet18 model at the same computation cost.

Original: We also proposed a novel batch-shaping loss that can match the marginal aggregated posterior of a feature in the network to any prior PDF.
Revised: We also proposed a novel batch-shaping loss that can match the marginal aggregated posterior of a feature in the network to any prior PDF.

Original: We use it to enforce each gate in the network to be more conditionally activated at the start of training and improve performance signiﬁcantly.
Revised: We use it to enforce each gate in the network to be more conditionally activated at the start of training and improve performance signiﬁcantly.

Original: We look forward to seeing many novel applications for this loss in the future, for e.g., autoencoders, better quantized models, and as an alternative to batch-normalization.
Revised: We look forward to seeing many novel applications for this loss in the future, for e.g., autoencoders, better quantized models, and as an alternative to batch-normalization.

Original: Another important future research direction that can beneﬁt from our ﬁne-grained gating architecture is continual learning.
Revised: Another important future research direction that can beneﬁt from our ﬁne-grained gating architecture is continual learning.

Original: Designing gating mechanisms that can dynamically decide to allow or prevent the ﬂow of gradients through certain parts of the network could potentially mitigate catastrophic forgetting.
Revised: Designing gating mechanisms that can dynamically decide to allow or prevent the ﬂow of gradients through certain parts of the network could potentially mitigate catastrophic forgetting.

Original: Finally, with our gating setup, we could distill smaller sub-networks that work on only a subset of the trained classes. 
Revised: Finally, with our gating setup, we could distill smaller sub-networks that work on only a subset of the trained classes. A CKNOWLEDGMENTS

Revised: The authors would like to thank Markus Nagel, Pim de Haan and jakub tomczak for their valuable discussions and feedback.

Original:  A PPENDIX A P SEUDO CODE FOR THE IMPLEMENTATION OF THE BATCH - SHAPING LOSS
Revised:  A PPENDIX A P SEUDO CODE FOR THE IMPLEMENTATION OF THE BATCH - SHAPING LOSS

Original: The pseudo code for the forward and backward pass of the batch-shaping loss.
Revised: The pseudo code for the forward and backward pass of the batch-shaping loss.

Original: Note that the terms idx sort , (cid:126)p pdf , (cid:126)p cdf , and (cid:126)x sort can be computed and stored in the forward pass and retrieved in the backward pass. 
Revised: Note that the terms idx sort , (cid:126)p pdf , (cid:126)p cdf , and (cid:126)x sort can be computed and stored in the forward pass and retrieved in the backward pass. Batch-Shaping loss input : Gating [Table] [Table]

Original: [Table]
Revised: [Table]

Original: endreturn
Revised: γ , Beta distribution parameters [Table] [Table] output:

Original: [Table]
Revised: [Table]

Revised: for that gating module loss = 0 for i ← 1 to m do (cid:126)x sort = sort ( X i ) (cid:126)p cdf =

Revised: [Table]

Revised: [Table]

Original:  loss 
Revised: [Table] , β ) loss = [Table] + sum (

Original: [Table]
Revised: [Table]

Revised: )endreturn γ · loss

Original: end
Revised: Backward pass of the Batch-Shaping loss input : Sorted indices idx [Table] [Table] pdf , (cid:126)p

Revised: [Table]

Revised: , and (cid:126)x

Original: [Table]
Revised: [Table]

Revised: : The gradient of the loss with respect to the

Revised: [Table]

Revised: for i

Revised: [Table]

Revised: 1 to m do

Revised: [Table]

Revised: pdf ( (cid:126)x

Revised: [Table]

Revised: ) (cid:126)

Revised: [Table]

Revised: = undo

Revised: [Table]

Revised: [Table]

Revised: ) end return (cid:126) grad

Original: B T RAINING DETAILS B. CIFAR 
Revised: B T RAINING DETAILS B. CIFAR

Original: We trained all models using Nesterov’s accelerated gradient descent Nesterov (1983) with a momentum of 0.9 and weight decay factor of 5 e − 4 .
Revised: We trained all models using Nesterov’s accelerated gradient descent Nesterov (1983) with a momentum of 0.9 and weight decay factor of 5 e − 4 .

Original: No weight decay was applied on the parameters of the gating modules.
Revised: No weight decay was applied on the parameters of the gating modules.

Original: We used a standard data-augmentation scheme by randomly cropping and horizontally ﬂipping the images Lin et al.
Revised: We used a standard data-augmentation scheme by randomly cropping and horizontally ﬂipping the images Lin et al.

Original: We trained the models for 500 epochs with a mini-batch of 256.
Revised: We trained the models for 500 epochs with a mini-batch of 256.

Original: The initial learning rate was 0.

Revised: The initial learning rate was 0.

Original: and it was divided by 10 at epoch 300, 375, and 450.
Revised: and it was divided by 10 at epoch 300, 375, and 450.

Original: B. I MAGE N ET
Revised: B. I MAGE N ET

Original: We used similar optimization settings to CIFAR-10 with a weight decay factor of 1 e − 4 .
Revised: We used similar optimization settings to CIFAR-10 with a weight decay factor of 1 e − 4 .

Original: We used a standard data-augmentation scheme adopted from He et al.
Revised: We used a standard data-augmentation scheme adopted from He et al.

Original: and trained the model forepochs with a mini-batch size of 256.
Revised: and trained the model forepochs with a mini-batch size of 256.

Original: All models were trained using a single GPU.
Revised: All models were trained using a single GPU.

Original: The initial learning rate of 0.1 was divided by 10 at epoch 60, 90, and 120.
Revised: The initial learning rate of 0.1 was divided by 10 at epoch 60, 90, and 120.

Original: B. C ITYSCAPES
Revised: B. C ITYSCAPES

Original: Cityscapes Cordts et al.
Revised: Cityscapes Cordts et al.

Original: (2016) is a dataset for semantic urban scene understanding including 5, images with high quality pixel-level annotations and 20,000 additional images with coarse annotations.
Revised: (2016) is a dataset for semantic urban scene understanding including 5, images with high quality pixel-level annotations and 20,000 additional images with coarse annotations.

Original: There are 19 semantic classes for evaluation of semantic segmentation models in this benchmark.
Revised: There are 19 semantic classes for evaluation of semantic segmentation models in this benchmark.

Original: For data augmentation, we adopt random mirror and resize with a factor between 0.5 and 2 and use a crop-size of 448 × 672 for training.
Revised: For data augmentation, we adopt random mirror and resize with a factor between 0.5 and 2 and use a crop-size of 448 × 672 for training.

Original: We train and test with only single-scale input and run inference on the whole image.
Revised: We train and test with only single-scale input and run inference on the whole image.

Original: The PSPNet network with the ResNet-50 back-end was trained from scratch with a mini-batch size of 6 for 150k iterations.
Revised: The PSPNet network with the ResNet-50 back-end was trained from scratch with a mini-batch size of 6 for 150k iterations.

Original: Momentum and weight decay are set to 0.9 and 1 e − 4 respectively.
Revised: Momentum and weight decay are set to 0.9 and 1 e − 4 respectively.

Original: For the learning rate we use similar policy to Chen et al.
Revised: For the learning rate we use similar policy to Chen et al.

Original: (2017) where the initial learning rate of 2 e − 2 is multiplied by (1 − iter current / iter max ) 0 .
Revised: (2017) where the initial learning rate of 2 e − 2 is multiplied by (1 − iter current / iter max ) 0 .

Original: We used the same settings for training our gated PSPNet.
Revised: We used the same settings for training our gated PSPNet.

Original: Weight decay for the layers in the gating units was set to 1 e − 6 .
Revised: Weight decay for the layers in the gating units was set to 1 e − 6 .

Original: C D ETAILS FOR MODEL TIMING
Revised: C D ETAILS FOR MODEL TIMING

Original: The output of the gates are computed before the convolutional operations inside a resnet block.
Revised: The output of the gates are computed before the convolutional operations inside a resnet block.

Original: The output is fed to the preceding and following convolutional layer kernels, to not compute the masked inputs/outputs.
Revised: The output is fed to the preceding and following convolutional layer kernels, to not compute the masked inputs/outputs.

Original: In many practical settings, images are fed to a network one by one (e.g. real-time inference).
Revised: In many practical settings, images are fed to a network one by one (e.g. real-time inference).

Original: In this case, the convolutional weights can be loaded from DDR memory to local compute memory conditionally.
Revised: In this case, the convolutional weights can be loaded from DDR memory to local compute memory conditionally.

Original: Computation can be done on sliced tensors, which we implemented in Pytorch.
Revised: Computation can be done on sliced tensors, which we implemented in Pytorch.

Original: For batch-computing, a custom convolutional kernel could use the calculated mask on the ﬂy.
Revised: For batch-computing, a custom convolutional kernel could use the calculated mask on the ﬂy.

Original: We simulated this for the GPU in the same table.
Revised: We simulated this for the GPU in the same table.

Original: Custom hardware could naturally give us beneﬁts from all MACs saved, including energy savings.
Revised: Custom hardware could naturally give us beneﬁts from all MACs saved, including energy savings.

Original: All the reported inference times were measured using a machine equipped with an Intel Xeon E5-1620 v4 CPU and an Nvidia GTX 1080 Ti GPU.
Revised: All the reported inference times were measured using a machine equipped with an Intel Xeon E5-1620 v4 CPU and an Nvidia GTX 1080 Ti GPU.

Original: C. CPU MEASUREMENTS
Revised: [Figure]

Original: Consider W 1 ∈ R c in 1 × c 1 out × k × k and W 2 ∈ R c in 2 × c 2 out × k × k representing the weight tensors of the ﬁrst and second layers in a ResNet block, where c 1 out = c in 2 .
Revised: Consider W 1 ∈ R c in 1 × c 1 out × k × k and W 2 ∈ R c in 2 × c 2 out × k × k representing the weight tensors of the ﬁrst and second layers in a ResNet block, where c 1 out = c in 2 .

Original: For each ResNet block, we ﬁrst use the output of the gates to generate a mask.
Revised: For each ResNet block, we ﬁrst use the output of the gates to generate a mask.

Original: Using this mask, we slice the original weight tensor of the ﬁrst layer in the block and apply conv2d on the input featuremap using the sliced weight tensor
Revised: Using this mask, we slice the original weight tensor of the ﬁrst layer in the block and apply conv2d on the input featuremap using the sliced weight tensor

Original: W 1 ∈ R c in 1 × c slice × k × k .
Revised: W 1 ∈ R c in 1 × c slice × k × k .

Original: We next apply masking for the ﬁrst batch normalization layer.
Revised: We next apply masking for the ﬁrst batch normalization layer.

Original: The input to the second layer is a featuremap with lower number of channels.
Revised: The input to the second layer is a featuremap with lower number of channels.

Original: Using the same mask, we slice the weight tensor of the second layer W 2 ∈ R c slice × c 2 out × k × k and apply the conv2d layer using this tensor.
Revised: Using the same mask, we slice the weight tensor of the second layer W 2 ∈ R c slice × c 2 out × k × k and apply the conv2d layer using this tensor.

Original: C. GPU MEASUREMENTS
Revised: C. GPU MEASUREMENTS

Original: For the GPU measurements, we ﬁrst recorded the gating patterns of the entire images in the validation set.
Revised: For the GPU measurements, we ﬁrst recorded the gating patterns of the entire images in the validation set.

Original: For each input image, a sparse model (with a fewer number of convolution kernels in each layer) was deﬁned based on the gating pattern.
Revised: For each input image, a sparse model (with a fewer number of convolution kernels in each layer) was deﬁned based on the gating pattern.

Original: The computation time was then reported for the sparse model.
Revised: The computation time was then reported for the sparse model.

Original: The overhead caused by the gating modules is included in the wall-time calculation.
Revised: The overhead caused by the gating modules is included in the wall-time calculation.

Original: D C OMPARISON OF THE EFFECT OF L 0 AND B ATCH - SHAPING LOSS ON I MAGE N ET
Revised: D C OMPARISON OF THE EFFECT OF L 0 AND B ATCH - SHAPING LOSS ON I MAGE N ET

Original: As can be seen in Figure 8, the models that additionally use the batch-shaping loss consistently outperform the ones using only the L 0 complexity loss.
Revised: As can be seen in Figure 8, the models that additionally use the batch-shaping loss consistently outperform the ones using only the L 0 complexity loss.

Original: However, similar to CIFAR-10 observations, our L 0 -gated models still outperform ConvNet-AIG.
Revised: However, similar to CIFAR-10 observations, our L 0 -gated models still outperform ConvNet-AIG.