Original: R EPARAMETERIZATION THROUGH S PATIAL G RADIENT S CALING
Revised: R EPARAMETERIZATION THROUGH S PATIAL G RADIENT S CALING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reparameterization aims to improve the generalization of deep neural networks by transforming a convolution operation into equivalent multi-branched structures during training.
Revised: Reparameterization aims to improve the generalization of deep neural networks by transforming convolutional layers into equivalent multi-branched structures during training.

Original: However, there exists a gap in understanding how reparameterization may change and benefit  learning processes for neural networks.
Revised: However, there exists a gap in understanding how reparameterization may change and benefit the learning process of neural networks.

Original: In this paper, we present a novel spatial gradient scaling method to redistribute learning focus among weights in convolutionalneural networks.
Revised: In this paper, we present a novel spatial gradient scaling method to redistribute learning focus among weights in convolutional networks.

Original: We prove that spatial gradient scaling achieves the same learning dynamics as a branched reparameterization yet without introducing structural changes into the network.
Revised: We prove that spatial gradient scaling achieves the same learning dynamics as a branched reparameterization yet without introducing structural changes into the network.

Original: We further propose an analytical approach that dynamically learns scalings for each convolutional layer based on the spatial characteristics of its input feature map gauged by mutual information.
Revised: We further propose an analytical approach that dynamically learns scalings for each convolutional layer based on the spatial characteristics of its input feature map gauged by mutual information.

Original: Experiments on CIFAR-10, CIFAR-100, and ImageNet show that without searching for reparameterized structures, our proposed scaling method outperforms the state-of-the-art reparameterization methods at a lower computational cost
Revised: Experiments on CIFAR-10, CIFAR-100, and ImageNet show that without searching for reparameterized structures, our proposed scaling method outperforms the state-of-the-art reparameterization strategies at a lower computational cost.

Revised: The code is available at https://github.com/Ascend-Research/Reparameterization.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ever-increasing performance of deep learning is largely attributed to progress made in neural architectural design, with a trend of not only building deeper networks (Krizhevsky et al., 2012;
Revised: The ever-increasing performance of deep learning is largely attributed to progress made in neural architectural design, with a trend of not only building deeper networks (Krizhevsky et al., 2012;

Original: Simonyan & Zisserman, 2014) but also introducing complex blocks through multi-branched structures (Szegedy et al., 2015; 2016; 2017).
Revised: Simonyan & Zisserman, 2014) but also introducing complex blocks through multi-branched structures (Szegedy et al., 2015; 2016; 2017).

Original: Recently, efforts have been devoted to Neural Architecture
Revised: Recently, efforts have been devoted to Neural Architecture

Original: Search, Network Morphism, and Reparametrization, which aim to strike a balance between network expressiveness and computational cost.
Revised: Search, Network Morphism, and Reparametrization, which aim to strike a balance between network expressiveness, performance, and computational cost.

Original: Neural Architecture Search(NAS) (Elsken et al., 2018; Zoph & Le, 2017) searches for network topologies in a predefined search space, which often involves multi-branched micro-structures.
Revised: Neural Architecture Search (NAS) (Elsken et al., 2018; Zoph & Le, 2017) searches for network topologies in a predefined search space, which often involves multi-branched micro-structures.

Original: Examples include  DARTS (Liu et al., 2019) and NB101 (Ying et al., 2019) search spaces that span a large number of cell (block) topologies that could be stacked  to form a neural network.
Revised: Examples include the DARTS (Liu et al., 2019) and NAS-Bench-101 (Ying et al., 2019) search spaces that span a large number of cell (block) topologies which are stacked together to form a neural network.

Original: In Network Morphism (Wei et al., 2016; 2017), a well-trained parent network is morphed into a child network with the goal of adopting it on a downstream application with minimum re-training.
Revised: In Network Morphism (Wei et al., 2016; 2017), a well-trained parent network is morphed into a child network with the goal of adopting it on a downstream application with minimum re-training.

Original: Morphism preserves the parent network’s functions and output while yielding child networks that are deeper and wider.
Revised: Morphism preserves the parent network’s functions and output while yielding child networks that are deeper and wider.

Original: Structural reparameterization (Ding et al., 2021c) attempts to branch and augment certain operations during training into an equivalent buta more complex structure with extra learnable parameters.
Revised: Structural reparameterization (Ding et al., 2021c) attempts to branch and augment certain operations during training into an equivalent but more complex structure with extra learnable parameters.

Original: For example, Asymmetric Convolution Block (ACB) (Ding et al., 2019) augments a regular 3x convolution with both horizontal 1 × 3 and vertical 3 × 1 convolutions, such that training is performed on the reparameterized network which takes advantage of the changed learning dynamics.
Revised: For example, Asymmetric Convolution Block (ACB) (Ding et al., 2019) augments a regular 3x3 convolution with both horizontal 1 × 3 and vertical 3 × 1 convolutions, such that training is performed on the reparameterized network which takes advantage of the changed learning dynamics.

Original: While during inference, the equivalent original network has the boosted performance but still preserves a lower inference time due to the simpler structure.
Revised: During inference, the trained reparameterized network is equivalently transformed back to its base simple structure, preserving the original low inference time, while maintaining the boosted performance of the reparameterized model.

Original: However, there exists a gap regarding the understanding ofwhy and how  the different learning dynamics of a reparameterized model could help its training.
Revised: However, there exists a gap regarding the understanding of how and when the different learning dynamics of a reparameterized model could help its training.

Original: In addition, the search for the optimal reparameterized structure over a discrete space (Huang et al., 2022) inevitably increases the computational cost in deep learning.
Revised: In addition, the search for the optimal reparameterized structure over a discrete space (Huang et al., 2022) inevitably increases the computational cost in deep learning.

Original: In this paper, we propose Spatial Gradient Scaling (SGS), an approach that changes learning dynamics as with reparameterization, yet without introducing structural changes to the neural network.
Revised: In this paper, we propose Spatial Gradient Scaling (SGS), an approach that changes learning dynamics as with reparameterization, yet without introducing structural changes to the neural network.

Original: We examine the question— Can we achieve the same effect of branched reparameterization on convolutional networks without changing the network structure?
Revised: We examine the question— Can we achieve the same effect as branched reparameterization on convolutional networks without changing the network structure?

Original: Our proposed spatial gradient scaling learns a spatially varied scaling for the gradient of convolutional weights, which we prove to yield the same learning dynamics as branched reparameterization, without having to change the network architecture.
Revised: Our proposed spatial gradient scaling learns a spatially varied scaling for the gradient of convolutional weights, which we prove to have the same learning dynamics as branched reparameterization, without modifying network structure.

Original: We  show that scaling gradients by examining the spatial dependence of  pixelsto their neighbors in the input (or intermediate) feature maps can boost neural network learning performance, without searching for the reparameterized form.
Revised: We further show that scaling gradients by examining the spatial dependence of neighboring pixels in the input (or intermediate) feature maps can boost neural network learning performance, without searching for the reparameterized form.

Original: Our main contributions can be summarized as follows
Revised: Our main contributions can be summarized as follows:

Original: : • We investigate a new problem of spatial gradient scaling for weights in the kernels of convolutional neural networks.
Revised: • We investigate a new problem of spatially scaling gradients in the kernels of convolutional neural networks.

Original: This method enhances the performance of existing architectures only by redistributing learning rates spatially, i
Revised: This method enhances the performance of existing architectures only by redistributing learning rates spatially, i

Original: e., by adaptively strengthening or weakening the gradients of convolution weights according to their  position in the kernel.
Revised: e., by adaptively strengthening or weakening the gradients of convolution weights according to their relative position in the kernel.

Original: • We mathematically establish a connection between the proposed SGS and parallel convolutional reparameterization, and show their equivalence in learning.
Revised: • We mathematically establish a connection between the proposed SGS and parallel convolutional reparameterization, and show their equivalence in learning.

Original: This enables an understanding of how the existing multi-branched reparameterization structures helpin changing the gradient in backpropagation and thus improve deep learning.
Revised: This enables an understanding of how the existing multi-branched reparameterization structures help improve feature learning.

Original: This interpretation also enables us to create an architecture-independent  method by directly inducing the effectof reparameterization via scaled gradients while bypassing the need for searching for optimal branching forms, saving on computational costs.
Revised: This interpretation also suggests an architecture-independent reparameterization method by directly inducing the effect via scaled gradients, bypassing the need for complex structure augmentations, and saving on computational costs.

Original: • We propose a lightweight method to compute gradient scaling for a given network andany given dataset based on the spatial dependencies in  feature maps.
Revised: • We propose a lightweight method to compute gradient scalings for a given network and dataset based on the spatial dependencies in the feature maps.

Original: Specifically, we makea novel use of  inherent mutual information between elements in a feature map within the receptive field of a convolution kernel to dynamically determinea gradient scaling for each network layer, with only a minimum overhead to the original training routine.
Revised: Specifically, we make novel use of the inherent mutual information between neighboring pixels of a feature map within the receptive field of a convolution kernel to dynamically determine gradient scalings for each network layer, with only a minimum overhead to the original training routine.

Original: Extensive experiments show that the proposed data-driven spatial gradient scaling approach leads to results that compete or outperform state-of-the-art methods on several image classification models and datasets, yet with almost no extra computational cost and memory consumption in training that multi-branched reparameterization structures require.
Revised: Extensive experiments show that the proposed data-driven spatial gradient scaling approach leads to results that compete or outperform state-of-the-art methods on several image classification models and datasets, yet with almost no extra computational cost and memory consumption during training that multi-branched reparameterization structures require.

Original:  R ELATED W ORK 2. M ULTI - BRANCH S TRUCTURES
Revised:  R ELATED W ORK 2. M ULTI - BRANCH S TRUCTURES

Original: VGG (Simonyan & Zisserman, 2014) is a base model for several computer vision tasks.
Revised: VGG (Simonyan & Zisserman, 2014) is a base model for several computer vision tasks.

Original: Due to its limitations, several new structures have been proposed with multiple branches to achieve higher performance.
Revised: Due to its limitations, several new structures have been proposed with multiple branches to achieve higher performance.

Original: GoogleNet (Szegedy et al., 2015) and Inception (Szegedy et al., 2015; 2016; 2017) architectures deploy multi-branch structures to enrich the  feature space.
Revised: GoogleNet (Szegedy et al., 2015) and Inception (Szegedy et al., 2015; 2016; 2017) architectures deploy multi-branch structures to enrich the learned feature space.

Original: ResNet (He et al., 2016) uses a simplified two-branch structure which adds the input of a layer to its output through residual connections.
Revised: ResNet (He et al., 2016) uses a simplified two-branch structure that adds the input of a layer to its output through residual connections.

Original: The improvements in top-1 accuracy of ImageNet classification using these structures demonstrate the importance of multiple receptive fields (e.g., 1 × 1 , 1 × K , K × 1 , and K × K convolutions), diverse connections of layers and combination of parallel branches.
Revised: The improvements in top-1 accuracy of ImageNet classification using these structures demonstrate the importance of multiple receptive fields (e.g., 1 × 1 , 1 × K , K × 1 , and K × K convolutions), diverse connections of layers and combination of parallel branches.

Original: Efforts in finding architectures with higher performance are going on as more complicated models are proposed.
Revised: These performance improvements often come at a computational cost, as complex model topologies are less hardware friendly, and have increased computational requirements.

Original: Several search spaces in Neural Architecture Search (NAS) benchmark datasets (Ying et al., 2019; Dong & Yang, 2020; Ding et al., 2021a) were designed to cover architectures with multi-branch structures.
Revised: Outside of expertly designed networks, advancements in Neural Architecture Search (NAS) allow for the automation of network design. Several search spaces and discovered high-performing networks (Ying et al., 2019; Dong & Yang, 2020; Ding et al., 2021a) utilize multi-branch structures, which shows their ubiquity in modern convolution architectures.

Original: However, due to the enormous possibilities in model topology, it is  computationally expensive and requires vast computational resources.
Revised: Due to the enormous possibilities of branched model topologies, search is often computationally expensive and requires vast computational resources.

Original: 2. S TRUCTURAL R EPARAMETRIZATION
Revised: 2. S TRUCTURAL R EPARAMETRIZATION

Original: Multi-branch structures enhance the performance of ConvNets.
Revised: Multi-branch structures enhance the performance of ConvNets.

Original: This comes at the cost of higher memory and computational power requirements, which is undesirable for inference-time applications.
Revised: This comes at the cost of higher memory and computational power requirements, which is undesirable for inference-time applica- tions.

Original: RepVGG (Ding et al., 2021c) proposes using structural reparameterization to train with  multi-branch structures but  inference with an equivalent plain architecture. They introduce a family of VGG-like inference models that are trained with ResNet inspired reparameterization blocks.
Revised: Structural reparameterization solves this by training with a complex multi-branch model to improve the learned representations but equivalently transforming back to the original simple base model during inference for decreased computational costs. RepVGG (Ding et al., 2021c) introduces a family of VGG-like inference models that are trained with ResNet-inspired reparameterization blocks.

Original: These reparameterization blocks consist of parallel 3 × 3 and 1 × 1 convolutions along with an identity branch.
Revised: These reparameterization blocks consist of parallel 3 × 3 and 1 × 1 convolutions along with an identity branch.

Original: After training, they consider identity as a degraded 1 × 1 convolution and a 1 ×convolution as a degraded 3 × 3 convolution. Using the additive property of convolutions,parameters of parallel branches are algebraically transformed intoparameters of a single 3 × 3 kernel. These equivalent kernels are then finally deployed in the structure of inference-time RepVGG models that use plain 3 × 3 convolutions instead of training-time branched structure.
Revised: After training, using the linearity of convolutions, parallel branches are equivalently transformed into a single 3 × 3 convolution. The inference VGG-like model has the advantage of both the enhanced learned representations of the complex reparameterized model, and the fast and efficient inference of the simple base model.

Original: Similarly, DBB (Ding et al., 2021b) structurally reparameterize models by replacing all K × K convolution with a multi-branch topology composed of multi-scale and sequential 1 × 1 - K × K convolutions, average pooling, and branch addition in the training phase.
Revised: Similarly, DBB (Ding et al., 2021b) structurally reparameterize models by replacing all K × K convolution with a multi-branch topology composed of multi-scale and sequential 1 × 1 - K × K convolutions and average pooling during training.

Original: After training, DBB blocks are equivalently reparameterized back to K × K convolutions so that there is no added computational cost during inference.
Revised: After training, DBB blocks are equivalently reparameterized back to K × K convolutions for efficient inference.

Original: Instead of structurally reparameterizing all convolutions, DyRep (Huang et al., 2022) aims to selectively reparameterize only important convolutions to improve the training efficiency of the reparameterized model.
Revised: Instead of structurally reparameterizing all convolutions, DyRep (Huang et al., 2022) aims to selectively reparameterize only important convolutions to improve the training efficiency of the reparameterized model.

Original: During training, using various zero-cost metrics they dynamically search for important convolutions which they equivalently replace with multi-branch blocks. To decrease training cost they find and reduce low importance reparameterized blocks back to their original single convolution form. Unlike RepVGG, both DBB and DyRep do not introduce new architectures, but instead an architecture-neutral multi-branch block that can be used to augment the  training  of any convolutional model at no inference cost.
Revised: Our spatial gradient scaling approach has the benefit of a branched reparameterization without the added training cost of an augmented network structure.

Original: ACNet (Ding et al., 2019), through pruning experiments, showed that convolution weights on the central crisscross positions of the 3 × 3 kernels are more important to the models representational capacity than corner weights.
Revised: ACNet (Ding et al., 2019), through pruning experiments, showed that convolution weights on the central crisscross positions of the 3 × 3 kernels are more important to the model’s representational capacity than corner weights.

Original: To further enhance the kernel crisscross’s importance they reparameterize, during training, 3 × 3 convolutions with their Asymmetric Convolution Blocks (ACB).
Revised: To further enhance the kernel crisscross’s importance, they reparameterize, during training, 3 × 3 convolutions with their Asymmetric Convolution Blocks (ACB).

Original: ACB comprises of parallel 3 × 3 , 1 × 3 , and 3 × 1 convolutions.
Revised: ACB comprises of parallel 3 × 3 , 1 × 3 , and 3 × 1 convolutions.

Original: They found that models trained with
Revised: They found that models trained with

Original: ACB reparameterization attain higher performance than the base model.
Revised: ACB reparameterization perform better than the base model.

Original: Similar to ACNet, we also emphasize the importance of  central positions in the kernels.
Revised: Like ACNet, we also emphasize the importance of kernel central positions.

Original: However, instead of using parallel asymmetric convolution blocks, which adds significant computational cost during training, we scale the gradients of convolution weights with a spatially varying gradient scaling.
Revised: However, instead of using ACB blocks, which add significant training cost, we scale the gradients of convolution weights with a spatially varying gradient scaling.

Original: In fact, by using spatial gradient scaling, we can emulate the presence of multi-branch topology without adding to the structure and computational cost of the training model.
Revised: In fact, by using spatial gradient scaling, we can emulate the presence of multi-branch topology without adding to the structure and computational cost of the training model.

Original: 2. F EATURE S ELECTION WITH M UTUAL I NFORMATION
Revised: 2. F EATURE S ELECTION WITH M UTUAL I NFORMATION

Original: Mutual information has extensive applications in  computer vision andthe field of medical imaging (Pluim et al., 2003).
Revised: Mutual information has extensive applications in the domain of computer vision and medical imaging. Mutual information is used by (Pluim et al., 2003).

Original: Viola & Wells III (1997) use mutual information to derive a metric that compares a 3D model of the shape ofthe object and a  video image of it in an alignment problem. Maes et al. (1997) use it as a measure of statistical dependence between image intensities in image registration.
Revised: Viola & Wells III (1997) as a metric for comparing the alignment of a 3D model to video images.

Original: Russakoff et al.
Revised: Russakoff et al.

Original: (2004) use mutual information to measure similarity between images.
Revised: (2004) use mutual information to measure similarity between images.

Original:  Cheng et al.
Revised: In deep learning, Cheng et al.

Original: (2018) utilized mutual information to evaluate deep neural networks in image classification. They infer a model’s power in detecting objects of each class through measuring mutual information between inputs and outputs of adeep neural network, and again between outputs and target labels.
Revised: (2018) used the mutual information between inputs, outputs, and target labels of a neural network to infer its power of distinguishing between classes.

Original: In this paper,however, we use mutual information in a novel way to capturemutual dependencies between neighboring elements within  feature maps.
Revised: In this paper, we use mutual information in a novel way to capture dependencies between neighboring elements within a feature map.

Original: We use this spatial information as a dynamic scale for adjusting the importance of spatial positions in a convolution kernel.
Revised: We use this spatial information as a dynamic scale for adjusting the importance of spatial positions in a convolution kernel.

Original:  M ETHOD
Revised:  M ETHOD

Original: In this section, we first introduce spatial gradient scaling for convolutional neural networks.
Revised: In this section, we first introduce spatial gradient scaling for convolutional neural networks.

Original: We then establish its connection to reparameterization and mathematically show their equivalence.
Revised: We then establish its connection to reparameterization and mathematically show their equivalence.

Original: Finally, we describe our mutual-information-based approach to dynamically determine spatial gradient scaling during the training process at a low computational cost.
Revised: Finally, we describe our mutual-information-based approach to dynamically determine spatial gradient scaling during the training process at a low computational cost.

Original: 3. SPATIAL GRADIENT SCALING
Revised: 3. SPATIAL GRADIENT SCALING

Original: Gradient scaling adjusts backpropagation by strengthening or diminishing gradients of learnable parameters based on the significance of their position in the convolutional kernel.
Revised: Gradient scaling adjusts backpropagation by strengthening or diminishing gradients of learnable parameters based on the significance of their position in the convolutional kernel.

Original: Let W ( t ) l be the learnable weights for the l -th convolutional layer at training iteration t .
Revised: Let W ( t ) l be the learnable weights for the l -th convolutional layer at training iteration t .

Original: The shape of W ( t ) l is denoted by ( c out l , c in l , k x l , k y l ) , where c out l and c in l are sizes of output and input channels, respectively, and ( k x l , k y l ) denotes the kernel size.
Revised: The shape of W ( t ) l is denoted by ( c out l , c in l , k x l , k y l ) , where c out l and c in l are sizes of output and input channels, respectively, and ( k x l , k y l ) denotes the kernel size.

Original: During each iteration of training, we backpropagate the training loss L train to calculate the gradients of the learnable parameters ∂ L /∂W ( t ) l .
Revised: During each iteration of training, we backpropagate the training loss L train to calculate the gradients of the learnable parameters ∂ L /∂W ( t ) l .

Original: Following matrix calculus notations, let ∂ L /∂W denote a derivative tensor whose element at the position indexed by ( m, n, o, p ) is given by ∂ L /∂W m,n,o,p .
Revised: Following matrix calculus notations, let ∂ L /∂W denote a derivative tensor whose element at the position indexed by ( m, n, o, p ) is given by ∂ L /∂W m,n,o,p .

Original: Then, a gradient descent optimization step for the convolutional layer l with weights W l can be represented by
Revised: Then, a gradient descent optimization step for the convolutional layer l with weights W l can be represented by

Original: [Equation]
Revised: [Equation]

Original: where λ ( t ) is learning rate and f ( · ) is an optimizer dependent element-wise function of the current and past gradients and weights.
Revised: where λ ( t ) is learning rate and f ( · ) is an optimizer dependent element-wise function of the current and past gradients and weights.

Original: Our approach scales the gradient by a spatial gradient scaling matrix
Revised: Our approach scales the gradient by a spatial gradient scaling matrix

Original: G ( t ) l , to yield the following gradient descent update rule:
Revised: G ( t ) l , to yield the following gradient descent update rule:

Original: [Equation]
Revised: [Equation]

Original: where G ( t ) l is a matrix of shape ( k x l , k y l ) and ⊙ denotes element-wise multiplication along dimensions c out l and c in l .
Revised: where G ( t ) l is a matrix of shape ( k x l , k y l ) and ⊙ denotes element-wise multiplication along dimensions c out l and c in l .

Original: We additionally constrain G ( t ) l to be strictly positive with a mean of 1 to prevent large changes in the overall gradient direction and magnitude.
Revised: We additionally constrain G ( t ) l to be strictly positive with a mean of 1 to prevent large changes in the overall gradient direction and magnitude.

Original: To account for various optimizers, we scale the gradient before any optimizer-dependent calculations like momentum and weight decay.
Revised: To account for various optimizers, we scale the gradient before any optimizer-dependent calculations like momentum and weight decay.

Original: Elements of G ( t ) l are learned in three steps.
Revised: Elements of G ( t ) l are learned in three steps.

Original: First, we define each element in G ( t ) l by its displacement from the center element.
Revised: First, we define each element in G ( t ) l by its displacement from the center element.

Original: Next, we measure the average spatial relatedness between every two pixels in the input feature map that are particularly displaced apart.
Revised: Next, we measure the average spatial relatedness between every two pixels in the input feature map that are particularly displaced apart.

Original: We denote this as the average spatial dependence and define it mathematically as a function of the feature map and displacement in Section 3.3.
Revised: We denote this as the average spatial dependence and define it mathematically as a function of the feature map and displacement in Section 3.3.

Original: Finally, we assign values to elements in G ( t ) l based on the average spatial dependence of their displacement.
Revised: Finally, we assign values to elements in G ( t ) l based on the average spatial dependence of their displacement.

Original: The overall process is depicted in Figure 1.
Revised: The overall process is depicted in Figure 1.

Original: We assign higher learning priority to elements with higher average spatial dependence.
Revised: We assign higher learning priority to elements with higher average spatial dependence.

Original: Feature maps with a high spatial correlation over large displacements give rise to more uniform spatial gradient scalings.
Revised: Feature maps with a high spatial correlation over large displacements give rise to more uniform spatial gradient scalings.

Original: Feature maps with low spatial correlations yield center concentrated scalings.
Revised: Feature maps with low spatial correlations yield center concentrated scalings.

Original: 3. E QUIVALENCE TO R EPARAMETERIZATION
Revised: 3. E QUIVALENCE TO R EPARAMETERIZATION

Original: We now establish the relationship between the proposed spatial gradient scaling and parallel convolution reparameterization.
Revised: We now establish the relationship between the proposed spatial gradient scaling and parallel convolution reparameterization.

Original: Specifically, we show that backward propagation for a single convolution is different from that of its N -branch reparameterization, where the latter is equivalent to updating the original convolution with certain spatial gradient scaling.
Revised: Specifically, we show that backward propagation for a single convolution is different from that of its N -branch reparameterization, where the latter is equivalent to updating the original convolution with certain spatial gradient scaling.

Original: Consider a k x l × k y l convolutional layer
Revised: Consider a k x l × k y l convolutional layer

Original: l
Revised: l

Original: with trainable weights W ( t ) l∈ R
Revised: with trainable weights W ( t ) l

Original: [Equation]
Revised: [Equation]

Original: k yl , input
Revised: input

Original: X ( t ) l and output Y ( t ) l at timestep t in training.
Revised: X ( t ) l and output Y ( t ) l at timestep t in training.

Original: Following Ding et al.
Revised: Following Ding et al.

Original: we can reparameterize this single convolutional layer into a general N -branch convolutional structure as depicted in Figure 2 (b) with batchnorms after the reparameterization instead of within.
Revised: we can reparameterize this single convolutional layer into a general N -branch convolutional structure as depicted in Figure 2 (b) with batchnorms after the reparameterization instead of within.

Original: Each branch n contains a convolution with a receptive field no larger than ( k x l , k y l ) .
Revised: Each branch n contains a convolution with a receptive field no larger than ( k x l , k y l ) .

Original: To mathematically represent variablesized kernels in different branches, we let M l,n ∈ { 0 , 1 } k xl
Revised: To mathematically represent variablesized kernels in different branches, we let M l,n ∈ { 0 , 1 } k xl

Original: × k yl denote a binary mask, which is a matrix in the shape of the corresponding kernel’s receptive field, as illustrated in Figure.
Revised: × k yl denote a binary mask, which is a matrix in the shape of the corresponding kernel’s receptive field, as illustrated in Figure.

Original: 2 (c).
Revised: 2 (c).

Original: Considering a forward pass, for each original convolutional layer l with weights W ( t ) l , we can find an equivalent N -branch reparameterization, where each branch n contains a convolutional weight tensor W ( t ) l,n and associated binary mask M l,n , such that the reparameterization yields the same mapping as the original convolution, i.e.,
Revised: Considering a forward pass, for each original convolutional layer l with weights W ( t ) l , we can find an equivalent N -branch reparameterization, where each branch n contains a convolutional weight tensor W ( t ) l,n and associated binary mask M l,n , such that the reparameterization yields the same mapping as the original convolution, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where ⊙ denotes element-wise multiplication across each dimension c out l and c in l .
Revised: where ⊙ denotes element-wise multiplication across each dimension c out l and c in l .

Original: The second equality in Eq.
Revised: The second equality in Eq.

Original: 3 decomposes the weight tensor W l , while the third equality achieves branched structural form.
Revised: 3 decomposes the weight tensor W l , while the third equality achieves branched structural form.

Original: The equivalence between a convolutional layer and its N -branch reparameterization with masked representation for each branch is illustrated in Figure 2 (c).
Revised: The equivalence between a convolutional layer and its N -branch reparameterization with masked representation for each branch is illustrated in Figure 2 (c).

Original: However, in backward passes, W l updates differently in the reparameterized network than in its original form.
Revised: However, in backward passes, W l updates differently in the reparameterized network than in its original form.

Original: That is, one step of gradient descent on the reparameterization in Eq.
Revised: That is, one step of gradient descent on the reparameterization in Eq.

Original: 3 has the following form to yield the merged weights W ( t +1) l for the next timestep:
Revised: 3 has the following form to yield the merged weights W ( t +1) l for the next timestep:

Original: [Equation]
Revised: [Equation]

Original: Note that the learning dynamics for the reparameterized network differ from the original convolution in Eq. 1, which explains why it may attain higher generalization although having identical expressivity.
Revised: Note that the learning dynamics for the reparameterized network differ from the original convolution in Eq. 1, which explains why it may attain higher generalization although having identical expressivity.

Original: Despite their different topologies, we show in the following lemma that updates for the original convolution (Eq. 1) and any of its N -branch reparameterization (Eq. 4) differ only by a constant spatial gradient scaling G l .
Revised: Despite their different topologies, we show in the following lemma that updates for the original convolution (Eq. 1) and any of its N -branch reparameterization (Eq. 4) differ only by a constant spatial gradient scaling G l .

Original: Lemma 1.
Revised: Lemma 1.

Original: Assume f ( . ) is a linear function in a gradient descent optimizationalgorithm
Revised: Assume f ( · ) is a linear function in a gradient descent optimization

Original:  (e.g., momentum, weight decay).
Revised: algorithm (e.g., momentum, weight decay).

Original: For any reparameterization of a convolutional layer l that can be represented as a summation of N convolutional branches with weights W ( t ) l,n and binary receptive field mask M l,n , for n = 1 , . . .
Revised: For any reparameterization of a convolutional layer l that can be represented as a summation of N convolutional branches with weights W ( t ) l,n and binary receptive field mask M l,n , for n = 1 , . . .

Original: , N , its gradient descent update Equation 4 is equivalent to
Revised: , N , its gradient descent update Equation 4 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where G l = (cid:80) Nn =1 M l,n is a spatial gradient scaling applied to the original convolution.
Revised: where G l = (cid:80) Nn =1 M l,n is a spatial gradient scaling applied to the original convolution.

Original: The proof in Appendix A follows readily from the equations of gradient descent.
Revised: The proof in Appendix A follows readily from the equations of gradient descent and the linearity of convolutions.

Original: There are multiple implications of Lemma 1.
Revised: Lemma 1 has multiple implications.

Original: First, it provides an understanding on how branched reparameterization helps to change the backpropagation dynamics; it redistributes learning rates spatially to focus on more important weights in a convolutional kernel.
Revised: First, it provides an understanding of how branched reparameterization helps to change the backpropagation dynamics; it redistributes learning rates spatially to focus on more important weights in a convolutional kernel.

Original: Second, Lemma 1 allows us to convert the search for a reparameterization structure into an equivalent numerical gradient scaling search, which is more efficient and lends itself to analytical methods (as we demonstrate in Section 3.3).
Revised: Second, Lemmaallows us to convert the search for a reparameterization structure into an equivalent numerical gradient scaling search, which is more efficient and lends itself to analytical methods (as we demonstrate in Section 3.3).

Original: Our scaling interpretation of structural reparameterization allows us a more flexible search space unconstrained by the computational complexity of the underlying structure.
Revised: Our scaling interpretation of structural reparameterization allows us a more flexible search space unconstrained by the computational complexity of the underlying structure.

Original: Third, we find agreement between our formalized gradient scaling approach and observational rules of thumb in the reparameterization literature.
Revised: Third, we find agreement between our formalized gradient scaling approach and observational rules of thumb in the reparameterization literature.

Original: For example, the current trend of preferring branches with a diverse range of convolutional receptive fields presented by Ding et al.
Revised: For example, the current trend of preferring branches with a diverse range of convolutional receptive fields presented by Ding et al.

Original: (2021b) may stem from the fact that without it, the gradient scaling will become uniform and thus lose spatial emphasis.
Revised: (2021b) may stem from the fact that without it, the gradient scaling is uniform and loses its spatial emphasis.

Original: 3. M UTUAL I NFORMATION B ASED SPATIAL GRADIENT SCALING
Revised: [Figure]

Original: We describe our approach to finding spatial gradient scalings from the spatial dependencies in the input feature map.
Revised: We describe our approach to finding spatial gradient scalings from the spatial dependencies in the input feature map.

Original: We begin by showing how mutual information can be used to quantify the average spatial dependency between ( i, j ) -displaced pixels.
Revised: We begin by showing how mutual information can be used to quantify the average spatial dependency between ( i, j ) -displaced pixels.

Original: We then assign values to elements of our gradient scalings based on their spatial displacement to the center of the kernel and its associated average spatial dependency.
Revised: We then assign values to elements of our gradient scalings based on their spatial displacement to the center of the kernel and its associated average spatial dependency.

Original: Figure 3 shows an overview of the method
Revised: Figure 3 shows an overview of the method

Original: We quantify the spatial dependence of two pixels through the use of probabilistic dependence.
Revised: We quantify the spatial dependence of two pixels through the use of probabilistic dependence.

Original: We define random variables P l and Q l,i,j as pixel values in the l -th layer feature map and their associated ( i, j ) -displaced neighbor value respectively.
Revised: We define random variables P l and Q l,i,j as pixel values in the l -th layer feature map and their associated ( i, j ) -displaced neighbor values, respectively.

Original: The ( i, j ) -displaced neighbor for a pixel is the corresponding pixel ( i, j ) units away.
Revised: The ( i, j ) -displaced neighbor for a pixel is the corresponding pixel ( i, j ) units away.

Original: We express spatial dependence, S l ( i, j ) , as the normalized mutual information (MI), ˜ I [Equation] · ; · ) of the random variables P l and Q l,i,j :
Revised: We express spatial dependence, S l ( i, j ) , as the normalized mutual information (MI), ˜ I ( · ; · ) of the random variables P l and Q l,i,j :

Original: [Equation]
Revised: [Equation]

Original: Displacement ( i, j ) spans the entire ( k x l , k y l ) receptive field of the convolution kernel as demonstrated in Figure 3 (a).
Revised: Displacement ( i, j ) spans the entire ( k x l , k y l ) receptive field of the convolution kernel as demonstrated in Figure 3 (a).

Original: We arrange elements of S l ( i, j ) into a spatial dependence matrix as illustrated in Figure 3 (b).
Revised: We arrange elements of S l ( i, j ) into a spatial dependence matrix as illustrated in Figure 3 (b).

Original: Intuitively, spatial dependency within pixels results in statistical dependencies in pixel values, which can be detected via mutual information.
Revised: Intuitively, spatial dependency within pixels results in statistical dependencies in pixel values, which can be detected via mutual information.

Original: ˜
Revised: ˜

Original: I ( · ) is bounded between 0, when variables are independent, and 1, with complete mutual dependence.
Revised: I ( · ) is bounded between 0, when variables are independent, and 1, with complete mutual dependence.

Original: We calculate the normalized MI from the Shannon entropy H ( · ) of the random variables:
Revised: We calculate the normalized MI from the Shannon entropy H ( · ) of the random variables:

Original: [Equation]
Revised: [Equation]

Original: where H ( P l , Q l,i,j ) is the joint entropy of P and Q i,j .
Revised: where H ( P l , Q l,i,j ) is the joint entropy of P and Q i,j .

Original: We calculate the entropy by estimating the distributions of P l and Q l,i,j through discrete binning.
Revised: We calculate the entropy by estimating the distributions of P l and Q l,i,j through discrete binning.

Original: High-resolution images often contain redundant / spatially repeated neighbor pixels that may cause mutual information to overestimate the amount of useful learnable spatial dependence.
Revised: High-resolution images often contain redundant / spatially repeated neighbor pixels that may cause mutual information to overestimate the amount of useful learnable spatial dependence.

Original: To account for this, on ImageNet, we remove occurrences of P l with Q l,i,j that are near in pixel values when we estimate their distributions.
Revised: To account for this, on ImageNet, we remove occurrences of P l with Q l,i,j that are near in pixel values when we estimate their distributions.

Original: In practice, a single image is not large enough to generate accurate distributions for P l and Q l,i,j , so we aggregate pixel and their ( i, j ) -neighbors values over multiple batches of training data.
Revised: In practice, a single image is not large enough to generate accurate distributions for P l and Q l,i,j , so we aggregate pixel and their ( i, j ) -neighbors values over multiple batches of training data.

Original: Due to the 
Revised: Due to the inexpensive nature of the mutual information computation, we can afford to perform this calculation for each convolutional layer and every couple of epochs.

Original: [Table]
Revised: [Table]

Original: inexpensive nature of the mutual information computation, we can afford to perform this calculation for
Revised: To get the spatial gradient scaling

Original: [Table]
Revised: [Table]

Original: To get the spatial gradient scaling, G l , we transform the spatial dependency matrix S l with an element-wise transform parameterized by a hyperparameter k :
Revised: G l [Table] we transform the spatial dependency matrix S l with an element-wise transform parameterized by a hyperparameter k :

Original: [Equation]
Revised: [Equation]

Original: k converts mutual information values into effective gradient scalings.
Revised: k converts mutual information values into effective gradient scalings.

Original: Finally, we normalize the mean value of G l .
Revised: Finally, we normalize the mean value of G l .

Original: A overview of the SGS framework is given as pseudo code in Appendix A.5.
Revised: An overview of the SGS framework is given as pseudo-code in Appendix A.5, and details can be found in the corresponding open-source code 1 .

Original: In short, our scaling defines how significant weight elements are for feature extraction based on their spatial location within the kernel.
Revised: In short, our scaling defines how significant weight elements are for feature extraction based on their spatial location within the kernel.

Original: In order to determine the scaling, we use mutual information to measure the notion of spatial dependence between pixels a distance apart.
Revised: In order to determine the scaling, we use mutual information to measure the notion of spatial dependence between pixels a distance apart.

Original: We give high learning priority to the elements with a large spatial dependence to the center kernel element.
Revised: We give high learning priority to the elements with a large spatial dependence on the center kernel element.

Original:  E XPERIMENTS AND R ESULTS
Revised:  E XPERIMENTS AND R ESULTS

Original: In this section, we assess the effectiveness of spatial gradient scaling in improving model generalization ability.
Revised: In this section, we assess the effectiveness of spatial gradient scaling in improving model generalization ability.

Original: Following convention (Huang et al.
Revised: Following convention (Huang et al.

Original: (2022), Ding et al.
Revised: (2022), Ding et al.

Original: (2021b))
Revised: (2021b))

Original: , we compare test accuracies for ResNet and VGG models trained under state-of-the-art reparameterization schemes.
Revised: , we compare test accuracies for ResNet and VGG models trained under state-of-the-art reparameterization schemes.

Original: We adopt the model training code and strategy from Huang et al.
Revised: We adopt the model training code and strategy from Huang et al.

Original: 4. CIFAR
Revised: 4. CIFAR

Original: We train VGG-16 on CIFAR- { 10,100 } for 600 epochs with a batch size of 128, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 1 × 10 − 4 .
Revised: We train VGG-16 on CIFAR- { 10,100 } for 600 epochs with a batch size of 128, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 1 × 10 − 4 .

Original: We update our spatial gradient scalings every 30 epochs using 20 random batches from the training set.
Revised: We update our spatial gradient scalings every 30 epochs using 20 random batches from the training set.

Original: We add a 1 epoch warm-up period at the start of training before generating our gradient scalings.
Revised: We add a 1 epoch warm-up period at the start of training before generating our gradient scalings.

Original: Results are shown in Table 3.
Revised: Results are shown in Table 3.

Original: Additional results are available in Appendix A.1.
Revised: Additional results are available in Appendix A.1.

Original: Our framework uses a single hyperparameter k , which defines a functional mapping between mutual information and spatial gradient scaling.
Revised: Our framework uses a single hyperparameter k , which defines a functional mapping between mutual information and spatial gradient scaling.

Original: We search for k on CIFAR100 and use the optimal for experiments on CIFAR10 and ImageNet.
Revised: We search for k on CIFAR100 and use the optimal for experiments on CIFAR10 and ImageNet.

Original: We perform a grid search on CIFAR100 and VGG-16 over k ∈ { 2 , 3 , 4 , 5 , 6 , 7 } using 20% of the training set for validation.
Revised: We perform a grid search on CIFAR100 and VGG-16 over k ∈ { 2 , 3 , 4 , 5 , 6 , 7 } using 20% of the training set for validation.

Original: As k acts to interpret values of mutual information into meaningful gradient scalings, we may expect k to remain constant across models and datasets (we test this claim in Section 4.3). 
Revised: As k acts to interpret values of mutual information into meaningful gradient scalings, we may expect k to remain constant across models and datasets (we test this claim in Section 4.3). [Table]

Original: Our spatial gradient scaling performs equally or better than state-of-the-art reparameterization methods at a fraction of their cost.
Revised: Our spatial gradient scaling performs equally or better than state-of-the-art reparameterization methods at a fraction of their cost.

Original: On CIFAR10, our spatial gradient scaling outperforms DBB and[Table] performs as well as DyRep while only requiring a third of their training time.
Revised: On CIFAR10, our spatial gradient scaling outperforms DBB and performs as well as DyRep while only requiring a third of their training time.

Original: On CIFAR100, we obtain over 1% accuracypoint improvement from DBB and DyRep while taking less than half their GPU hours.
Revised: On CIFAR100, we obtain over 1% accuracy improvement from DBB and DyRep while taking less than half their GPU hours.

Original: We attribute the success of spatial gradient scaling to its enhanced reparameterization space and strategy.
Revised: We attribute the success of spatial gradient scaling to its enhanced reparameterization space and strategy.

Original: Unlike structural reparameterization methods like DBB and DyRep, we are not limited by the computational complexity of our blocks which enables us to explore a much larger space of reparameterizations.
Revised: Unlike structural reparameterization methods like DBB and DyRep, we are not limited by the computational complexity of our blocks which enables us to explore a much larger space of reparameterizations.

Original: Our formalism ofthe spatial gradient scaling as an equivalent to reparameterization also enables us to performthe search in an easily implementable continuous space as opposed to  discrete structural.
Revised: Our formalism of spatial gradient scaling as an equivalent to reparameterization also enables us to perform search in an easily implementable continuous space as opposed to a discrete structural one.

Original: Our mutual information strategy adaptively reparameterizes each convolution throughout the training process efficiently and effectively (as we show in Section 4.3).
Revised: Our mutual information strategy adaptively reparameterizes each convolution throughout the training process efficiently and effectively (as we show in Section 4.3).

Original: 4. I MAGE N ET
Revised: 4. I MAGE N ET

Original: We train the ResNet models for 120 epochs, with a batch size of 256, cosine annealing scheduler with initial lr of 0.1, color jitter augmentation, and SGD with momentum of 0.1 and weight decay 1 × 10 − 4 .
Revised: We train the ResNet models for 120 epochs, with a batch size of 256, cosine annealing scheduler with initial lr of 0.1, color jitter augmentation, and SGD with momentum of 0.1 and weight decay 1 × 10 − 4 .

Original: Scalings update every 5 epochs using two random training batches after a one epoch model warm-up.
Revised: Scalings update every 5 epochs using two random training batches after a one epoch model warm-up.

Original: Hyperparameter k is taken as the optimal found from the CIFAR100 grid search.
Revised: Hyperparameter k is taken as the optimal found from the CIFAR100 grid search.

Original: Results are presented in Table 2.
Revised: Results are presented in Table 2.

Original: As with CIFAR, we see significantly reduced training times with our spatial gradient scaling method for equal or better accuracy compared to the state-of-the-art reparameterization methods.
Revised: As with CIFAR, we see significantly reduced training times with our spatial gradient scaling method for equal or better accuracy compared to the state-of-the-art reparameterization methods.

Original: The benefits of reparameterization are attained without complicating the model structure with expensive reparameterization blocks.
Revised: The benefits of reparameterization are attained without complicating the model structure with expensive reparameterization blocks.

Original: Large convolution kernels, like the 7 x 7 used in ResNet, are difficult to structurally reparameterize.
Revised: Large convolution kernels, like the 7 x 7 used in ResNet, are difficult to structurally reparameterize.

Original: First, these large convolutions are expensive in terms of compute and memory.
Revised: First, these large convolutions are expensive in terms of compute and memory.

Original: The addition of a parallel reparameterization branch only increases its computational cost further.
Revised: The addition of a parallel reparameterization branch only increases its computational cost further.

Original: Second, as convolution size increases, so does the number of possible diverse receptive fields (for example 7 × 7 ’s receptive fields are: 1 × 1 , 1 × 3 , ..., 3 × 5 , ... ).
Revised: Second, as convolution size increases, so does the number of possible diverse receptive fields (for example 7 × 7 ’s receptive fields are: 1 × 1 , 1 × 3 , ..., 3 × 5 , ... ).

Original: DBB and DyRep can only consider a tiny fraction of the possible set ( 7 × 7 , 1 × 7 , 7 × 1 , 1 × 1 ).
Revised: DBB and DyRep can only consider a tiny fraction of the possible set ( 7 × 7 , 1 × 7 , 7 × 1 , 1 × 1 ).

Original: Our spatial gradient scaling can consider all possible receptive fields, even non-standard ones, through our general binary masks.
Revised: Our spatial gradient scaling can consider all possible receptive fields, even non-standard ones, through our general binary masks.

Original: In addition, our method completely avoids the computational pitfalls of structural reparameterization, as our reparameterization happens efficiently on the gradient level. 
Revised: In addition, our method completely avoids the computational pitfalls of structural reparameterization, as our reparameterization happens efficiently on the gradient level. [Table]

Original: 4. A BLATION S TUDIES
Revised: 4. A BLATION S TUDIES

Original: Effectiveness of Mutual Information Approach.
Revised: Effectiveness of Mutual Information Approach.

Original: We demonstrate our mutual information approach’s efficacy in finding high-performance spatial gradient scalings.
Revised: We demonstrate our mutual information approach’s efficacy in finding high-performance spatial gradient scalings.

Original: We compare to autocorrelation, another commonly used dependency measure, as well as a grid search for gradient scalings.
Revised: We compare to autocorrelation, another commonly used dependency measure, as well as a grid search for gradient scalings.

Original: [Table] We train all methods  with VGG-11 for 200 epochs with a batch size of 512, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 5 × 10 − 4 .
Revised: We train all methods on CIFAR100 with VGG-11 for 200 epochs with a batch size of 512, cosine annealing scheduler with an initial learning rate of 0.1, and SGD optimizer with momentum 0.9 and weight decay 5 × 10 − 4 .

Original: For autocorrelation and mutual information, we update our spatial gradient scalings every 10 epochs using 2 random batches from the training set and a warmup of 1 epoch.
Revised: For autocorrelation and mutual information, we update our spatial gradient scalings every 10 epochs using 2 random batches from the training set and a warmup of 1 epoch.

Original: Similar to mutual information, we  measure spatial dependencies using autocorrelation on input feature maps.
Revised: Similar to mutual information, we can measure spatial dependencies using autocorrelation.

Original: Specifically, we calculate the autocorrelation of a feature map with its copy shifted by ( i, j ) , where ( i, j ) indexes into a spatial dependency matrix similar to Figure 3.3.
Revised: Specifically, we calculate the correlation of a feature map with itself shifted by ( i, j ) , where ( i, j ) indexes into a spatial dependency matrix (Figure 3.3).

Original: We use the same k transform to map autocorrelation values intothe spatial gradient scaling.
Revised: We use the same k transform to map autocorrelation values into gradient scalings.

Original: We perform a grid search on over k ∈ { 1 , 2 , 3 , 4 , 5 , 6 } using 20% of the training set for validation.
Revised: We perform a grid search on over k ∈ { 1 , 2 , 3 , 4 , 5 , 6 } using 20% of the training set for validation.

Original: For mutual information, we use  optimal k found in Section 4.
Revised: For mutual information, we use the optimal k found in Section 4.

Original: We take several considerations for grid search over spatial gradient scaling to make the search tractable.
Revised: We take several considerations for grid search over spatial gradient scaling to make the search tractable.

Original: First, we reduce the search space by considering a single 3x3 gradient scaling shared by all convolutional layers and constant for all training epochs.
Revised: First, we reduce the search space by considering a single 3 × 3 gradient scaling shared by all convolutional layers and constant for all training epochs.

Original: We additionally parameterize the scaling matrix by two variables, α , and β , which determine the ratio of the center element scaling to the edges and corners, respectively (shown in Appendix 7).
Revised: We additionally parameterize the scaling matrix by two variables, α , and β , which determine the ratio of the center element scaling to the edges and corners respectively (shown in Appendix 7).

Original: We search over α, β ∈ { 0 . 8 , 1 . 0 , 1 . 25 , 1 . 7 , 5 . 0 , 10 , 100 } .
Revised: We search over α, β ∈ { 0 . 8 , 1 . 0 , 1 . 25 , 1 . 7 , 5 . 0 , 10 , 100 } .

Original: Our mutual information approach (SGS) outperforms both autocorrelation and grid search.
Revised: Our mutual information approach (SGS) outperforms both autocorrelation and grid search.

Original: While grid search is robust, it suffers from high computational complexity, which requires designing a constrained search space.
Revised: While grid search is robust, it suffers from high computational complexity, which requires designing a constrained search space.

Original: Unlike SGS, grid search cannot effectively adapt across convolution depth and time without an exponential blowup in the search space.
Revised: Unlike SGS, grid search cannot effectively adapt across convolution depth and time without an exponential blowup in the search space.

Original: While autocorrelation outperforms grid search, our mutual information method performs better.
Revised: While autocorrelation outperforms grid search, our mutual information method performs better.

Original: We attribute this to the fact that autocorrelation can only measure linear relationships in the feature map, while mutual information can measure both linear and non-linear dependencies.
Revised: We attribute this to the fact that autocorrelation can only measure linear relationships in the feature map, while mutual information can measure both linear and non-linear dependencies.

Original: k-Transformation Search.
Revised: k-Transformation Search.

Original: In this section, we investigate the behavior of the k hyperparameter across models and datasets.
Revised: In this section, we investigate the behavior of the k hyperparameter across models and datasets.

Original: We additionally corroborate our decision in Section 4 to learn k once on CIFAR-100 and transfer to ImageNet.
Revised: We additionally corroborate our decision in Section 4 to learn k once on CIFAR-100 and transfer to ImageNet.

Original: Following the training strategy defined in Section 4, we train and evaluate models over a range of k values and plot the results in Appendix Figure 8.
Revised: Following the training strategy defined in Section 4, we train and evaluate models over a range of k values and plot the results in Appendix Figure 8.

Original: We observe that k curves across models and datasetsand peaks near k = 5 .
Revised: We observe that k curves across models and datasets peaks near k = 5 .

Original: This may imply that k
Revised: This may imply that k

Original: = 5 is a robust default value.
Revised: =is a robust default value.

Original: We also find consistent performance gains over baseline for a large range of k values.
Revised: We also find consistent performance gains over baseline for a large range of k values.

Original: This implies that the conventional uniform update of weights is suboptimal, and lower testing error can be attained via spatial gradient scaling.
Revised: This implies that the conventional uniform update of weights is suboptimal, and lower testing error can be attained via spatial gradient scaling.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present Spatial Gradient Scaling (SGS), an approach that improves the generalizationperformance of neural networks by changing the learning dynamics to focus on spatially important weights.
Revised: In this paper, we present Spatial Gradient Scaling (SGS), an approach that improves the generalization of neural networks by changing the learning dynamics to focus on spatially important weights.

Original: We achieve this by scaling the convolutions gradients adaptively from the spatial dependencies of feature maps.
Revised: We achieve this by scaling the convolutions gradients adaptively from the spatial dependencies of feature maps.

Original: We propose a mutual information-based approach to compute the gradient scaling with minimum overhead to the original training routine.
Revised: We propose a mutual information-based approach to compute the gradient scaling with minimum overhead to the original training routine.

Original: We prove that our SGS is equivalent to convolutional reparameterization under certain conditions.
Revised: We prove that our SGS is equivalent to convolutional reparameterization under certain conditions.

Original: This enables us to take advantage of the benefits of reparameterization without introducing complex branching into model structures.
Revised: This enables us to take advantage of the benefits of reparameterization without introducing complex branching into model structures.

Original: Experiments show that our method outperforms the state-of-the-art structural reparameterization approaches on several image classification models and datasets, at  much lower computational cost.
Revised: Experiments show that our method outperforms the state-of-the-art structural reparameterization approaches on several image classification models and datasets at a much lower computational cost.

Original: A A PPENDIX A. E XPERIMENTS ON CIFAR- 
Revised: A A PPENDIX A. E XPERIMENTS ON CIFAR-

Original: We report in Table 4 results for the CIFAR-100 dataset on a wide range of CNN architectures.
Revised: We report in Table 4 results for the CIFAR-100 dataset on a wide range of CNN architectures.

Original: We train for 200 epochs with a batch size of 128, a cosine annealing scheduler with an initial learning rate of 0.
Revised: We train for 200 epochs with a batch size of 128, a cosine annealing scheduler with an initial learning rate of 0.

Original: [Table]
Revised: [Table]

Original: and SGD optimizer with momentum 0.9 and weight decay 5 × 10 − 4 .
Revised: and SGD optimizer with momentum 0.9 and weight decay 5 × 10 − 4 .

Original: For data augmentation, we use random crop, flip, and cutout.
Revised: For data augmentation, we use random crop, flip [Table] and cutout.

Original: Spatial gradient scalings are updated with mutual information every 5 epochs, using 2 random batches from the training set
Revised: Spatial gradient scalings are updated with mutual information every 5 epochs, using 2 random batches from the training set

Original: [Table]
Revised: [Table]

Original: and with k = 5 .
Revised: and with k = 5 .

Original: At the start of training, we add a warm-up period of 1 epoch before generating our gradient scalings.
Revised: At the start of training, we add a warm-up period of 1 epoch before generating our gradient scalings.

Original: Adding spatial gradient scaling to optimization yields performance gains for several architectures.
Revised: Adding spatial gradient scaling to optimization yields performance gains for several architectures.

Original: [Table]
Revised: [Table]

Original: A. C OMPARISONS TO A DAPTIVE G RADIENT O PTIMIZERS
Revised: A. C OMPARISONS TO A DAPTIVE G RADIENT O PTIMIZERS

Original: In this section, we compare spatial gradient scaling to popular adaptive gradient optimizers.
Revised: In this section, we compare spatial gradient scaling to popular adaptive gradient optimizers.

Original: Unlike adaptive optimizers, which typically optimize based on the model weights and gradients of previous timesteps, our gradient scaling uses the spatial properties within the training data to effectively scale convolution gradients.
Revised: Unlike adaptive optimizers, which typically optimize based on the model weights and gradients of previous timesteps, our gradient scaling uses the spatial properties within the training data to effectively scale convolution gradients.

Original: In Table 5 we present CIFAR-100 results for various optimizers with and without spatial gradient scaling.
Revised: In Table 5 we present CIFAR-100 results for various optimizers with and without spatial gradient scaling.

Original: We tune optimizer hyperparameters with a grid search using 20% of the training data for validation.
Revised: We tune optimizer hyperparameters with a grid search using 20% of the training data for validation.

Original: We focus our search on learning rate and weight decay, leaving other optimizer settings as PyTorch defaults.
Revised: We focus our search on learning rate and weight decay, leaving other optimizer settings as PyTorch defaults.

Original: Training and SGS settings, outside of searched optimizer hyperparameters, are identical to those described in Appendix A.1.
Revised: Training and SGS settings, outside of searched optimizer hyperparameters, are identical to those described in Appendix A.1.

Original: We find that spatial gradient scaling improves the performance of even the highest-performing optimizer, SGD + Momentum.
Revised: We find that spatial gradient scaling improves the performance of even the highest-performing optimizer, SGD + Momentum.

Original: Moreover, all tested optimizers, Adagrad being the only exception, showed benefit from spatial gradient scaling.
Revised: Moreover, all tested optimizers, Adagrad being the only exception, showed benefit from spatial gradient scaling.

Original: Even in Adagrad’s case, we can find performance gain by applying gradient scaling post-optimizer calculations and right before the weight update step (shown as Adagrad*) as opposed to directly to the back-propagated gradient and before optimizer calculations.
Revised: Even in Adagrad’s case, we can find performance gain by applying gradient scaling post-optimizer calculations and right before the weight update step (shown as Adagrad*) as opposed to directly to the back-propagated gradient and before optimizer calculations.

Original: A. S ENSITIVITY TO D IFFERENT T RAINING S ETTINGS
Revised: A. S ENSITIVITY TO D IFFERENT T RAINING S ETTINGS

Original: In this section, we empirically study performance improvement by spatial gradient scaling across training hyperparameters.
Revised: In this section, we empirically study performance improvement by spatial gradient scaling across training hyperparameters.

Original: Results are shown in Table 6.
Revised: Results are shown in Table 6.

Original: Training and SGS settings, outside of those in the ablation study, are identical to Appendix A.1.
Revised: Training and SGS settings, outside of those in the ablation study, are identical to Appendix A.1.

Original: We find performance improvements over the baseline under all tested hyperparameter configurations.
Revised: We find performance improvements over the baseline under all tested hyperparameter configurations.

Original: This suggests that SGS can be used effectively to improve ConvNets training without requiring extensive hyperparameter tweaking.
Revised: This suggests that SGS can be used effectively to improve ConvNets training without requiring extensive hyperparameter tweaking.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. E XPERIMENTS ON S EMANTIC S EGMENTATION AND H UMAN P OSE E STIMATION
Revised: A. E XPERIMENTS ON S EMANTIC S EGMENTATION AND H UMAN P OSE E STIMATION

Original: We consider semantic segmentation and human pose estimation to evaluate our spatial gradient scaling.
Revised: We consider semantic segmentation and human pose estimation to evaluate our spatial gradient scaling.

Original: For semantic segmentation, we adapt our training code and strategy from SemSeg 1 .
Revised: For semantic segmentation, we adapt our training code and strategy from SemSeg 2 .

Original: We train and evaluate the PSPNet models on the Cityscapes dataset.
Revised: We train and evaluate the PSPNet models on the Cityscapes dataset.

Original: PSPNet is composed of a ResNet18/34/50 backbone and an up-sampling head for pixel-level classification.
Revised: PSPNet is composed of a ResNet18/34/50 backbone and an up-sampling head for pixel-level classification.

Original: Using the backbone weights of the ResNet models pre-trained on ImageNet with and without SGS (Section 4.2), we fine-tune with and without SGS on Cityscapes.
Revised: Using the backbone weights of the ResNet models pre-trained on ImageNet with and without SGS (Section 4.2), we fine-tune with and without SGS on Cityscapes.

Original: Results are shown in Table 7.
Revised: Results are shown in Table 7.

Original: [Table]
Revised: [Table]

Original: For human pose estimation, we use the training code and strategy from Pytorch-Pose-HG-3D 2 .
Revised: For human pose estimation, we use the training code and strategy from Pytorch-Pose-HG-3D 3 .

Original: We train and evaluate on MPII dataset with MSRA ResNet.
Revised: We train and evaluate on MPII dataset with MSRA ResNet.

Original: MSRA ResNet has a Resnet-18/34/ backbone which we take pretrained from ImageNet with and without SGS.
Revised: MSRA ResNet has a Resnet-18/34/ backbone which we take pretrained from ImageNet with and without SGS.

Original: We then fine-tune with and without SGS on MPII.
Revised: We then fine-tune with and without SGS on MPII.

Original: Results are shown in Table 8.
Revised: Results are shown in Table 8.

Original: [Table]
Revised: [Table]

Original: the human pose estimation task.
Revised: human pose estimation task.

Original: A.
Revised: A.

Original: [Equation]
Revised: T RAINING P SEUDO C ODE

Original: Input: Training dataset X .
Revised: Input: Training dataset X .

Original: Set of spatial gradient scaling SGS .
Revised: Set of spatial gradient scaling SGS .

Original: Number of epochs E for training.
Revised: Number of epochs E for training.

Original: Number of SGS warmup epochs N e .
Revised: Number of SGS warmup epochs N e .

Original: Number of batches N b for SGS calculation.
Revised: Number of batches N b for SGS calculation.

Original: Set of convolution layers L .
Revised: Set of convolution layers L .

Original: for e = 0 , ..., ( E − 1) do if e divisible by N e then b ← randomly sample N b batches from X
Revised: for e = 0 , ..., ( E − 1) do if e divisible by N e then b ← randomly sample N b batches from X

Original: Forward propagate b and assign the input feature map of convolution layer l to the set X l for l in L do for ( i, j ) in kernel size of l do
Revised: Forward propagate b and assign the input feature map of convolution layer l to the set X l for l in L do for ( i, j ) in kernel size of l do

Original: P, Q ( i,j ) ← List of pixels of X l and their corresponding (i,j)th neighbours
Revised: P, Q ( i,j ) ← List of pixels of X l and their corresponding (i,j)th neighbours

Original: SGS l, ( i,j ) ← Mutual Information between P and ( i, j ) neighbour Q ( i,j ) for ( x, y ) in ( X, Y ) do
Revised: SGS l, ( i,j ) ← Mutual Information between P and ( i, j ) neighbour Q ( i,j ) for ( x, y ) in ( X, Y ) do

Original: Forward and backward propagate ( x, y ) for l in L do for ( i, j ) in kernel size of l do
Revised: Forward and backward propagate ( x, y ) for l in L do for ( i, j ) in kernel size of l do

Original: Scale the l th convolution spatial ( i, j ) gradient element with SGS l, ( i,j )
Revised: Scale the l th convolution spatial ( i, j ) gradient element with SGS l, ( i,j )

Original: Update weights with gradients -pose-hg-3d
Revised: Update weights with gradients https://github.com/xingyizhou/pytorch-pose-hg-3d

Original: A. L EARNED S PATIAL G RADIENT S CALING
Revised: A. L EARNED S PATIAL G RADIENT S CALING

Original: In Figure 4, we plot the spatial gradient scalings for the first, seventh, and last convolutional layers of VGG-16 at the start and the end of its training on CIFAR-100.
Revised: In Figure 4, we plot the spatial gradient scalings for the first, seventh, and last convolutional layers of VGG-16 at the start and the end of its training on CIFAR-100.

Original: We observe more uniformly distributed gradient scaling for beginning layers giving equal relative importance to all kernel weights.
Revised: We observe more uniformly distributed gradient scaling for beginning layers giving equal relative importance to all kernel weights.

Original: Deeper layers have center-focused gradient scalings with most of the importance distributed on the center and edge elements as opposed to the corners.
Revised: Deeper layers have center-focused gradient scalings with most of the importance distributed on the center and edge elements as opposed to the corners.

Original: Interestingly, we observe that after training, spatial gradient scalings for deeper layers become even more center-focused, indicating a decrease in the spatial dependencies of the feature map.
Revised: Interestingly, we observe that after training, spatial gradient scalings for deeper layers become even more center-focused, indicating a decrease in the spatial dependencies of the feature map.

Original: We plot gradient scalings for ResNet18 on ImageNet in Figure 5.
Revised: We plot gradient scalings for ResNet18 on ImageNet in Figure 5.

Original: Like CIFAR, we see uniform spatial gradient scalings in early layers and center-focused scaling in deeper layers.
Revised: Like CIFAR, we see uniform spatial gradient scalings in early layers and center-focused scaling in deeper layers.

Original: Contrary to
Revised: Contrary to

Original: CIFAR, however, we find that gradient scalings “smoothen” over time and become more uniform.
Revised: CIFAR, however, we find that gradient scalings “smoothen” over time and become more uniform.

Original: Discrepancies between the behavior of spatial gradient scaling on CIFAR and ImageNet warrant future investigation.
Revised: Discrepancies between the behavior of spatial gradient scaling on CIFAR and ImageNet warrant future investigation.

Original: A. W EIGHT M AGNITUDES
Revised: A. W EIGHT M AGNITUDES

Original: Similar to Ding et al.
Revised: Similar to Ding et al.

Original: (2019), we investigate the average kernel magnitude matrix of learned weights with and without our spatial gradient scaling.
Revised: (2019), we investigate the average kernel magnitude matrix of learned weights with and without our spatial gradient scaling.

Original: For a convolution weight, the average kernel magnitude matrix is defined as the mean of the absolute value of the weight tensor across the input and output channels (leaving the spatial channels intact).
Revised: For a convolution weight, the average kernel magnitude matrix is defined as the mean of the absolute value of the weight tensor across the input and output channels (leaving the spatial channels intact).

Original: We further normalize the mean of the matrix for meaningful comparisons.
Revised: We further normalize the mean of the matrix for meaningful comparisons.

Original: Figure 6 depicts the average kernel magnitude matrix for the first, seventh, and last convolutional layer of VGG-16 trained on the CIFAR-100 dataset with and without spatial gradient scaling.
Revised: Figure 6 depicts the average kernel magnitude matrix for the first, seventh, and last convolutional layer of VGG-16 trained on the CIFAR-100 dataset with and without spatial gradient scaling.

Original: We additionally show the spatial gradient scaling of the last training epoch of the SGS training scheme.
Revised: We additionally show the spatial gradient scaling of the last training epoch of the SGS training scheme.

Original: Similar to Ding et al.
Revised: Similar to Ding et al.

Original: (2019), we observe that our spatial gradient scaling modifies the normally trained kernel magnitude matrix to focus more on the center and edge elements.
Revised: (2019), we observe that our spatial gradient scaling modifies the normally trained kernel magnitude matrix to focus more on the center and edge elements.

Original: We also qualitatively observe the similarities between the shape of spatial gradient scaling and the final trained weights.
Revised: We also qualitatively observe the similarities between the shape of spatial gradient scaling and the final trained weights.

Original: A. P ROOF FOR EQUIVALENCE TO REPARAMETERIZATION
Revised: A. P ROOF FOR EQUIVALENCE TO REPARAMETERIZATION

Original: In this section we prove  lemma 1 for optimizers that are linear functions of current and past weights and gradients:
Revised: In this section we prove by induction lemma 1 for optimizers that are linear functions of current and past weights and gradients i.e.,

Original: [Equation]
Revised: [Equation]

Revised: where W ( t ) l and ∂ L /∂W ( t ) l are the weights and respective gradients for the l -th convolutional layer at training iteration t and f ( · ) is a linear optimizer parameterized by arbitrary γ ( τ ) and ζ ( τ ) .

Original: We begin by defining two models: a base convolutional layer, and its n-branch reparameterization which at t = 0 has an identical mapping i.e.,
Revised: We begin by defining two models: a base convolutional layer, and its n-branch reparameterization which at t = 0 has an identical mapping:

Original: [Equation]
Revised: [Equation]

Original: Our goal is to find a modified update rule for the single convolution such the mappings are identical throughout training:
Revised: Our goal is to find a modified update rule for the single convolution such the mappings are identical throughout training i.e., ∀ ( t ≥ 0) the following holds:

Original: [Equation]
Revised: [Equation]

Original: Assume equation 9 holds ∀ ( t ≤ t 0 )
Revised: Assume equation 9 holds ∀ ( t ≤ t 0 )

Original: (we know this is for sure the case for t  = 0 ).
Revised: (we know this is for sure the case for t 0 = 0 ).

Original: We wish to find an update rule for the single convolution such that equation 9 holds for t
Revised: We wish to find an update rule for the single convolution such that equation 9 holds for t

Original: = t 0 + 1 and thus ∀ ( t ≤ ( t 0 + 1)) .
Revised: = t 0 + 1 and thus ∀ ( t ≤ ( t 0 + 1)) .

Original: Together with our assumption of identical mapping at t = 0 , we can then ensure equation 9 holds ∀ ( t ≥ 0) .
Revised: Together with our assumption of identical mapping at t = 0 , we can then ensure equation 9 holds ∀ ( t ≥ 0) .

Original: Assuming equation 9 holds ∀ ( t ≤ t 0 )
Revised: Assuming equation 9 holds ∀ ( t ≤ t 0 )

Original: we find the update equation for the merged weights W t 

Revised: we find the update equation for the merged weights W t

Original: +1 l as:
Revised: +1 l as:

Original: [Equation]
Revised: [Equation]

Original: Equation
Revised: Equation

Original: [Equation]
Revised: [Equation]

Original: Equation
Revised: Equation

Original: We then show that when equation 9 holds for t then:
Revised: We then show that when equation 9 holds for t then:

Original: [Equation]
Revised: [Equation]

Original: First we demonstrate that the convolution gradient is only a function of the input tensor, X ( t ) and the output gradient, ∂ L /∂Y ( t )
Revised: First we demonstrate that the convolution gradient is only a function of the input tensor, X ( t ) and the output gradient, ∂ L /∂Y ( t )

Original: [Equation]
Revised: [Equation]

Original: and not of the weight.
Revised: and not of the weight W .

Original: We make use of the tensor index and Einstein summation notation:
Revised: We make use of the tensor index and Einstein summation notation:

Original: [Equation]
Revised: [Equation]

Original:  L 
Revised: w X ( t ) c i ,h,k h ,w,k w ∂ L ∂W ( t ) c

Original: [Equation] h ,k w = (cid:88) k,i,j ∂ L ∂Y ( t ) k,i,j ∂Y ( t ) k,i,j ∂W ( t )
Revised: o ,c i ,k h ,k w = (cid:88) k,i,j ∂ L ∂Y ( t ) k,i,j ∂Y ( t ) k,i,j ∂W ( t )

Original: c o ,c i ,k h ,k w = ∂
Revised: c o ,c i ,k h ,k w = ∂

Original: L ∂Y ( t ) k,i,j ∂Y ( t ) k,i,j ∂W ( t )
Revised: L ∂Y ( t ) k,i,j ∂Y ( t ) k,i,j ∂W ( t )

Original: c o ,c i ,k h ,k w = ∂
Revised: c o ,c i ,k h ,k w = ∂

Original: L ∂Y ( t ) k,i,j ∂W ( t ) k,l,m,n ∂W ( t )
Revised: L ∂Y ( t ) k,i,j ∂W ( t ) k,l,m,n ∂W ( t )

Original: c o ,c i ,k h ,k w
Revised: c o ,c i ,k h ,k w

Original: X ( t ) i,m,j,n = ∂
Revised: X ( t ) l,i,m,j,n = ∂

Original: L ∂Y ( t ) k,i,j δ k,c
Revised: L ∂Y ( t ) k,i,j δ k,c

Original: o δ c i δ m,k h δ n,k w X ( t ) i,m,j,n = ∂
Revised: o δ l,c i δ m,k h δ n,k w X ( t ) l,i,m,j,n = ∂

Original: L ∂Y ( t ) c
Revised: L ∂Y ( t ) c

Original: o ,i,j
Revised: o ,i,j

Original: X ( t ) c
Revised: X ( t ) c

Original: i ,i,k h ,j,k w = (cid:88) i,j ∂ L ∂Y ( t )
Revised: i ,i,k h ,j,k w = (cid:88) i,j ∂ L ∂Y ( t )

Original: c
Revised: c

Original: o ,i,j
Revised: o ,i,j

Original: X ( t ) c
Revised: X ( t ) c

Original: i ,i,k h ,j,k w ⇒ ∂
Revised: i ,i,k h ,j,k w ⇒ ∂

Original: L ∂W ( t ) =
Revised: L ∂W ( t ) =

Original: g (cid:18)
Revised: g (cid:18)

Original: ∂
Revised: ∂

Original: L dY ( t ) , X ( t ) (cid:19)
Revised: L dY ( t ) , X ( t ) (cid:19)

Original: Given equation 9 we can now show:
Revised: Given equation 9 we can now show:

Original: [Equation]
Revised: [Equation]

Original: which implies that:
Revised: which implies that:

Original: [Equation]
Revised: [Equation]

Original: Finally, picking up where we left off:
Revised: Finally [Equation] picking up where we left off:

Original: [Equation]
Revised: [Equation]

Original: We arrive at lemma 1.
Revised: We arrive at lemma 1.

Original: [Figure]
Revised: [Figure]

Original: K

Original: [Figure]

Original: K

Original: [Figure]

Original: (cid:35) : = (cid:34) 1 /β 1 /α 1 /β 1 /α 1 /α 1 /β 1 /α 1 /β (cid:35) × 1 + 4 /a
Revised: := (cid:34) 1 /β 1 /α 1 /β 1 /α 1 /α 1 /β 1 /α 1 /β (cid:35) × 1 + 4 /a

Original: + 4 /b
Revised: + 4 /b