R E - PARAMETERIZING Y OUR O PTIMIZERS RATHER THAN A RCHITECTURES
Anonymous authors
Paper under double-blind review
A BSTRACT
The well-designed structures in neural networks reﬂect the prior knowledge incorporated into the models.
However, though different models have various priors, we are used to training them with model-agnostic optimizers such as SGD.
In this paper, we propose to incorporate model-speciﬁc prior knowledge into optimizers by modifying the gradients according to a set of model-speciﬁc hyper-parameters.
Such a methodology is referred to as Gradient Re-parameterization, and the optimizers are named RepOptimizers.
For the extreme simplicity of model structure, we focus on a VGG-style plain model and showcase that such a simple model trained with a
RepOptimizer, which is referred to as RepOpt-VGG, performs on par with or better than the recent well-designed models.
From a practical perspective, RepOpt-VGG is a favorable base model because of its simple structure, high inference speed and training efﬁciency.
Compared to Structural Re-parameterization, which adds priors into models via constructing extra training-time structures, RepOptimizers require no extra forward/backward computations and solve the problem of quantization.
We hope to spark further research beyond the realms of model structure design.
We will make the code and models publicly available.
I NTRODUCTION
The structural designs of neural networks are prior knowledge 1 incorporated into the model.
For example, modeling the feature transformation as residual-addition ( y = x
+ f ( x ) ) outperforms the plain form ( y =
f
( x ) )
(He et al., 2016).
ResNet incorporated such prior knowledge into the model with the shortcut structures and outperformed the simple models, e.g., VGGNet (Simonyan & Zisserman, 2014).
The recent advancements in structures have demonstrated the high-quality structural priors are vital for neural networks, e.g., EfﬁcientNet (Tan & Le, 2019) obtained a set of structural hyper-parameters via architecture search and compound scaling, which served as the prior knowledge for constructing the model.
Naturally, better structural priors result in higher performance.
Except for structural designs, the optimization methods are also important, which include 1) the ﬁrst order methods such as SGD (Robbins & Monro, 1951) and its variants (Kingma & Ba, 2014; Duchi et al., 2011; Loshchilov & Hutter, 2017), which have been used with ConvNet, Transformer (Dosovitskiy et al., 2020) and MLP (Tolstikhin et al., 2021), 2) the high-order methods (Shanno, 1970;
Hu et al., 2019; Pajarinen et al., 2019) which usually calculate or approximate the Hessian matrix (Dennis & Mor´e, 1977; Roosta-Khorasani & Mahoney, 2019), and 3) the derivative-free methods (Rios &
Sahinidis, 2013; Berahas et al., 2019) for cases that the derivatives may not exist (Sun et al., 2019).
We note that 1) though the advanced optimizers improve the training process in different ways, they have no prior knowledge speciﬁc to the model being optimized; 2) though we keep incorporating our up-to-date understandings into the models by designing advanced structures, we train them with optimizers like SGD (Robbins & Monro, 1951) and AdamW (Loshchilov & Hutter, 2017), which are model-agnostic .
To explore another approach, we make the following two contributions.
A methodology of incorporating the prior knowledge into a model-speciﬁc optimizer.
We focus on non-convex models like deep neural networks, so we only consider ﬁrst-order gradient- based optimizers such as SGD and AdamW.
We propose to incorporate the prior knowledge via modifying the gradients according to a set of model-speciﬁc hyper-parameters before updating the parameters.
We refer to this methodology as Gradient Re-parameterization (GR) and the optimizers as RepOptimizers .
This methodology differs from the other methods that introduce some extra parameters (e.g., adaptive learning rate (Kingma & Ba, 2014; Loshchilov & Hutter, 2017)) into the training process in that we re-parameterize the training dynamics according to some hyper-parameters derived from the model structure, but not the statistics obtained during training (e.g., the moving averages recorded by Momentum SGD and AdamW).
A favorable base model.
To demonstrate the effectiveness of incorporating the prior knowledge into the optimizer, we naturally use a model without careful structural designs.
We choose a VGGstyle plain architecture with only a stack of 3 × 3 conv layers, which is even simpler than the original
VGGNet (as the latter has max-pooling layers) and has long been considered inferior to well-designed models like EfﬁcientNets (Tan & Le, 2019), since the latter have way more abundant structural priors.
Impressively, such a simple model trained with RepOptimizers, which is referred to as RepOpt-VGG , can perform on par with or better than the well-designed models (Table 3).
We highlight the novelty of our work through a comparison to RepVGG (Ding et al., 2021).
We adopt
RepVGG as a baseline because it also produces powerful VGG-style models but with a different methodology.
Speciﬁcally, targeting a plain inference-time architecture, which is referred to as the target structure , RepVGG constructs extra training-time structures and converts them afterwards into the target structure for deployment.
The differences are summarized as follows (Fig. 1).
1) Similar to the regular models like ResNet, RepVGG also adds priors into models with well-designed structures and uses a generic optimizer, but RepOpt-VGG adds priors into the optimizer.
Compared to a
RepOpt-VGG, though the converted RepVGG has the same inference-time structure, the training-time
RepVGG is much more complicated and consumes more time and memory to train.
In other words, a
RepOpt-VGG is a real plain model during training, but a RepVGG is not.
We extend and deepen
Structural Re-parameterization
(Ding et al., 2021), which improves the performance of a model by changing the training dynamics via extra structures.
We show that changing the training dynamics with an optimizer has a similar effect but is more efﬁcient.
Of note is that we design the behavior of the RepOptimizer following RepVGG simply for a fair comparison and other designs may work as well or better; from a broader perspective, we present a
VGG-style model and an SGD-based RepOptimizer as an example, but the idea may generalize to other models and optimization methods.
For example, we also present the result of an MLP model with an AdamW-based RepOptimizer in the Appendix to show the generality of our method.
From a practical standpoint, RepOpt-VGG is also a favorable base model, which features both efﬁcient inference and efﬁcient training .
As an extremely simple architecture, it features low memory consumption, high degree of parallelism (one big operator is more efﬁcient than several small operators with the same FLOPs (Ma et al., 2018)), and greatly beneﬁts from the highly optimized 3 × 3 conv (e.g., Winograd Algorithm (Lavin & Gray, 2016)).
Better still, as the model only comprises one type of operator, we may integrate many 3 × 3 conv units onto a customized chip for even higher efﬁciency (Ding et al., 2021).
Efﬁcient training is of vital importance to the application scenarios where the computing resources are limited or we desire a fast delivery or rapid iteration of models, e.g., we may need to re-train the models every several days with the data recently collected.
Tableshows the training speed of RepOpt-VGG is around 1.8 × as RepVGG.
Similar to the inference, such simple models may be trained more efﬁciently with customized high-throughput training chips than a complicated model trained with general-purpose devices like GPU. 3)
Besides the training efﬁciency,
RepOptimizers overcome a major weakness of Structural Re-parameterization: the problem of quantization.
The inference-time RepVGG is difﬁcult to quantize via Post-Training Quantization (PTQ).
With simple INT8 PTQ, the accuracy of RepVGG on ImageNet (Deng et al., 2009) reduces to 54.55%.
We will show RepOpt-VGG is friendly to quantization and reveal the problem of quantizing
RepVGG results from the structural transformation of the trained model.
We naturally solve this problem with RepOptimizers as RepOpt-VGG undergoes no structural transformations at all .
R ELATED W ORK
Structural Re-parameterization (SR).
RepVGG adopts Structural Re-parameterization (Ding et al., 2021), which converts structures via transforming the parameters.
Targeting a VGG-style inference-
3 ×1 ×+ 3 ×1 ×+ convgradients
BN gradients
BN batch statistics γ, β μ, σ conv parameters
BN parameters
BN average statistics 3 ×3 ×γ, β μ, σ γ, β μ, σ + * 3 ×1 ×3 ×training structure (RepVGG) + regular Optimizer training structure (ResNet) forward backward update forward backward update training structure (real VGG) + RepOptimizer
[Figure]
γ, β μ, σ
Grad Mult
[Figure]
structure inference structure
[Figure]
layer
[Figure]
layer a parameter or gradient directly deploy
[Figure]
Mult .
time architecture, it constructs extra identity mappings and 1 × 1 conv layers for training and converts the whole block into a single 3 × 3 conv. In brief, the BN layers are fused into the preceding conv layers (note the identity mapping can also be viewed as a 1 × 1 conv layer whose kernel is an identity matrix), and the 1 × 1 conv kernels are added onto the central points of the 3 × 3 kernels (Fig. 1 b).
A signiﬁcant drawback of general SR is that the extra training costs cannot be spared.
Compared to
RepOpt-VGG, a RepVGG has the same target structure but complicated training structure.
Classic re-parameterization.
GR extends and deepens SR just like SR extended the classic meaning of re-paramerization, e.g., DiracNet (Zagoruyko & Komodakis, 2017), which encodes a conv kernel in a different form for the ease of optimization.
Speciﬁcally, a 3 × 3 conv kernel (viewed as a matrix) is re-parameterized as ˆ W = diag ( a )I + diag ( b )W norm , where a , b are learned vectors, and W norm is the normalized trainable kernel.
At each iteration, a conv layer ﬁrst normalizes its kernel, compute ˆW with a , b and W norm , then use the resultant kernel for convolution.
Then the optimizer must derive the gradients of all the parameters and update them.
In contrast, RepOptimizers do not change the form of trainable parameters and introduces no extra forward/backward computations.
Other optimizer designs.
Recent advancements in optimizers include low-rank parameterization (Shazeer & Stern, 2018), better second-order approximation (Gupta et al., 2018), etc., which improve have no prior knowledge speciﬁc to the model structure.
In contrast, we propose a novel perspective that some prior knowledge about architectural design may be translated into the design space of optimizer.
Designing the behavior of optimizer is related to meta-learning methods that learn the update function (Vicol et al., 2021; Alber et al., 2018), which may be combined with our method because adding prior knowledge in our way may facilitate the learning process.
R EP O PTIMIZERS
RepOptimizers change the original training dynamics, but it seems difﬁcult to imagine how the training dynamics should be changed to improve a given model.
To design and describe the behavior [Figure] +
[Figure]
3 ×searched constants
Grad Mult
[Figure]
3 ×1 ×+
Hyper-Search structure searchdataset target structure construct train on targetdataset
RepOptimizer
[Figure]
of a RepOptimizer, we adopt a three-step pipeline: 1) we deﬁne the prior knowledge and imagine a complicated structure to reﬂect the knowledge; 2) we ﬁgure out how to implement equivalent training dynamics with a simpler target structure whose gradients are modiﬁed according to some hyper-parameters; 3) we obtain the hyper-parameters to build up the RepOptimizer.
Note the design of RepOptimizer depends on the given model and prior knowledge, and we showcase with RepOpt-VGG.
3. I NCORPORATE K NOWLEDGE INTO S TRUCTURE
The core of a RepOptimizer is the prior knowledge we desire to utilize.
In the case of RepOpt-VGG, the knowledge is that a model’s performance may be improved by adding up the inputs and outputs of several branches weighted by diverse scales .
Such simple knowledge has sparked multiple structural designs.
For examples, ResNet (He et al., 2016) can be viewed as a simple but successful application of such knowledge, which simply combines (adds up) the inputs and outputs of residual blocks;
RepVGG (Ding et al., 2021) shares a similar idea but uses a different implementation.
With such knowledge, we design structures to meet our demands.
In this case, we desire to improve VGG-style models, so we choose the structural design of RepVGG (3 × 3 conv + 1 × 1 conv + identity mapping).
3. S HIFT THE S TRUCTURAL P RIORS INTO AN E QUIVALENT R EP O PTIMIZER
Then we describe how to shift the structural prior into a RepOptimizer.
We note an interesting phenomenon regarding the above-mentioned prior knowledge: in a special case where each branch only contains one linear trainable operator with optional constant scales , the model’s performance still improves if the constant scale values are set properly.
We refer to such a linear block as ConstantScale Linear Addition (CSLA) .
We note that we can replace a CSLA block by a single operator and realize equivalent training dynamics (i.e., they always produce identical outputs after any number of training iterations given the same training data) via multiplying the gradients by some constant multipliers derived from the constant scales.
We refer to such multipliers as Grad Mult (Fig. 1).
Modifying the gradients with Grad Mult can be viewed as a concrete implementation of GR.
In brief,
Proposition:
[Equation]
We present the proof of such equivalency, which is referred to as CSLA =
GR , in Appendix A. For the simplicity, here we only showcase the conclusion with two convolutions and two constant scalars as the scaling factors.
Let α A , α B be two constant scalars, W ( A ) , W ( B ) be two conv kernels of the same shape, X and Y be the input and output, ∗ denote convolution, the computation ﬂow of the
CSLA block is formulated as Y CSLA =
α A (X ∗ W ( A ) ) + α B (X ∗ W ( B ) ) .
For the GR counterpart, we directly train the target structure parameterized by W (cid:48) , so that Y GR =
X ∗ W (cid:48) .
Let i be the number of training iterations, we can ensure Y ( i )
CSLA
=
Y ( i ) GR , ∀ i ≥
as long as we follow two rules.
Rule of Initialization : W (cid:48) should be initialized as W (cid:48) (0) ← α A W ( A )(0) + α B W ( B )(0) .
In other words, the GR counterpart should be initialized with the equivalent parameters (which can be easily obtained by the linearity) as the CSLA counterpart in order to make their initial outputs identical.
Rule of Iteration : while the CSLA counterpart is updated with regular SGD (optionally with momentum) update rule, the gradients of the GR counterpart should be multiplied by ( α 2 A + α 2 B ) .
Formally, let L be the objective function, λ be the learning rate, we should update W (cid:48) by
[Equation]
With CSLA =
GR , we design and describe the behavior of RepOptimizer by ﬁrst designing the CSLA structure.
For RepOpt-VGG, the CSLA structure is instantiated by replacing the Batch Normalization
(BN) following the 3 × 3 and 1 × 1 layers in a RepVGG block by a constant channel-wise scaling and the BN in the identity branch by a trainable channel-wise scaling (Fig. 2), since a CSLA branch has no more than one linear trainable operator.
In such a slightly more complicated case with convolutions of different kernel sizes channel-wise constant scales, the Grad Mult is a tensor and the entries should be respectively computed with the scales on the corresponding positions.
We present the formulation of the Grad Mult for training a single 3 × 3 conv corresponding to such a CSLA block.
Let C be the number of channels, s , t ∈ R C be the constant channel-wise scales after the 3 × 3 and 1 × 1 layers, respectively, the Grad Mult M C × C × 3 × 3 is constructed by
[Equation]
Intuitively, p = 2 and q = 2 means the central point of the 3 × 3 kernel is related to the 1 × 1 branch (just like a RepVGG block that merges the 1 × 1 conv into the central point of the 3 × 3 kernel).
Since the trainable channel-wise scaling can be viewed as a “depthwise 1 × 1 conv“ followed by a constant scaling factor 1 , we add 1 to the Grad Mult at the “diagonal” positions (if the output shape does not match the input shape, the CSLA block will not have such a shortcut, so we simply ignore this term).
Remark Compared to the common SR forms like RepVGG, a CSLA block has no training-time nonlinearity like BN nor sequential trainable operators, and it can also be converted via common SR techniques into an equivalent structure that produce identical inference results.
However, inferencetime equivalency does not imply training-time equivalency, as the converted structure will have different training dynamics, breaking the equivalency after an update.
Moreover, we must highlight that the CSLA structure is imaginary , which is merely an intermediate tool for describing and visualizing a RepOptimizer but we never actually train it, because the results would be mathematically identical to directly training the target structure with GR.
There remains only one question: how to obtain the constant scaling vectors s and t ?
3. O BTAIN THE H YPER -P ARAMETERS OF R EP O PTIMIZER VIA H YPER -S EARCH
Naturally, as the the hyper-parameters of RepOptimizer, Grad Mults affect the performance.
Due to the non-convexity of optimization of deep neural network, the research community still lacks a thorough understanding of the black box, so we do not expect a provably optimal solution.
We propose a novel method to associate the hyper-parameters of an optimizer with the trainable parameters of an auxiliary model and search, which is therefore referred to as Hyper-Search (HS) .
Given a RepOptimizer, we construct the auxiliary Hyper-Search model by replacing the constants in the
RepOptimizer’s corresponding CSLA model with trainable scales and train it on a small search dataset (e.g., CIFAR-100 (Krizhevsky & Hinton, 2009)).
Intuitively, HS is inspired by DARTS (Liu et al., 2018) that the ﬁnal values of trainable parameters are the values that the model expects them to become , so the ﬁnal values of the trainable scales are the expected constant values in the imaginary CSLA model.
By CSLA =
GR , the expected constants in the CSLA models are exactly what we need to construct the expected Grad Mults of the RepOptimizer.
Note the high-level idea of HS is related to DARTS
but we do not adopt the concrete designs of DARTS (e.g., optimizing the trainable and architectural parameters alternatively) since HS is merely an end-to-end training, not a NAS method.
3. T RAIN WITH R EP O PTIMIZER
After Hyper-Search, we use the searched constants to construct the Grad Mults for each operator (each 3 × 3 layer in RepOpt-VGG) and store them in memory.
During the training of the target model on the target dataset, the RepOptimizer element-wisely multiplies the Grad Mults onto the gradients of the corresponding operators after each regular forward/backward computation ( Rule of Iteration ).
We should also follow the Rule of Initialization .
To start training with RepOptimizer, we re-initialize the model’s parameters according to the searched hyper-parameters.
In the case of RepOpt-VGG, just like the simplest example above ( W (cid:48) (0) ← α A W ( A )(0) + α B W ( B )(0) ), we should also initialize each 3 × 3 kernel with the equivalent parameters of a corresponding CSLA block at initialization.
More precisely, we suppose to initialize the 3 × 3 and 1 × 1 kernels in the imaginary CSLA block naturally by MSRA-initialization (He et al., 2015) as a common practice and the trainable channel-wise scaling
[Table]
layer as 1 .
Let W ( s )(0) ∈
R C × C × 3 × 3 and W ( t )(0)
∈ R C
×
C × 1 × 1 be the randomly initialized 3 ×and 1 × 1 kernels, the equivalent kernel is given by (note the channel-wise scaling is a 1 × 1 conv initialized as an identity matrix, so we should add 1 to the diagonal central positions like Eq. 3)
[Equation]
We use W (cid:48) (0) to initialize the 3 × 3 conv layer in the RepOpt-VGG model, so that the training dynamics will be equivalent to training a CSLA block initialized in the same way .
Without either of the two rules, the equivalency would break, so the performance would degrade (Table 4).
E XPERIMENTS 4. R EP O PTIMERS FOR I MAGE N ET C LASSIFICATION
We adopt RepVGG as our primary baseline since it also realizes simple yet powerful ConvNets.
We demonstrate the effectiveness of RepOptimizers by showcasing that RepOpt-VGG closely matches the accuracy of RepVGG with much faster training speed and lower memory consumption, and performs on par with or better than EfﬁcientNets, which are the state-of-the-art well-designed models.
Architectural setup.
For the fair comparison, RepOpt-VGG adopts the same simple architectural designs as RepVGG: apart from the ﬁrst stride-2 3 × 3 conv, we divide multiple 3 × 3 layers into four stages, and the ﬁrst layer of each stage has a stride of 2.
Global average pooling and an FC layer follow the last stage.
The number of layers and channels of each stage are shown in Table 1.
Training setup.
For training RepOpt-VGG and RepVGG on ImageNet, we adopt the identical training settings.
Speciﬁcally, we use 8 GPUs, a batch size of 32 per GPU, input resolution of 224 × 224, and a learning rate schedule with 5-epoch warm-up, initial value of 0.1 and cosine annealing for 120 epochs.
For the data augmentation, we use a pipeline of random cropping, left-right ﬂipping and RandAugment (Cubuk et al., 2020).
We also use a label smoothing coefﬁcient of 0.1.
The regular SGD optimizers for the baseline models and the RepOptimizers for RepOpt-VGG use momentum of 0.9 and weight decay of 4 × 10 − 5 .
We report the accuracy on the validation set.
Hyper-Search (HS) setup.
We use CIFAR-100 for searching the hyper-parameters of RepOptimizers with the same conﬁgurations as ImageNet except 240 epochs and simple data augmentation (only left-right ﬂipping and cropping).
CIFAR-100 has only 50K images and an input resolution of 32, so the computational cost of HS compared to the ImageNet training is roughly only 501281 × 240120 × ( 32224 ) 2 × 3 × 3+1 × 1 3 ×= 0 .
Note that the trainable scales of the HS model are initialized according to the layer’s depth.
Speciﬁcally, let l be the layer’s depth (i.e., the ﬁrst layer in each stage has l = 1 , the next layer has l = 2 ), the trainable scales are initialized as (cid:113) 2 l .
The intuition behind this common trick is to let the deeper layers behave like an identity mapping at initialization to facilitate training (Goyal et al., 2017;
Shao et al., 2020), which is discussed and analyzed in Appendix B.
Comparisons with RepVGG.
First, we test the maximum batch size (MaxBS) per GPU to measure the training memory consumption and compare the training speed.
That is, a batch size of MaxBS+ would cause OOM (Out Of Memory) error on the 2080Ti GPU which has 11GB of memory.
For the fair comparison, the training costs of all the models are tested with the same training script on the same machine with eight 2080Ti GPUs.
The training speed of each model is tested with both the same batch size (32 per GPU) and the respective MaxBS.
We train the models for 120 epochs both with a batch size of 32 and with their respective MaxBS.
We linearly scale up the initial learning rate
[Table] [Table]
when enlarging the batch size.
We repeat training each model three times and report the mean and standard deviation of accuacy.
We have the following observations (Table 2).
1) RepOpt-VGG consumes less memory and trains faster: RepOpt-VGG-B1 trains 1.8 × as fast as
RepVGG-B1 with the respective MaxBS.
With larger batch size, the performance of every model raises by a clear margin, which may be explained by BN’s better stability.
This highlights the memory efﬁciency of RepOptimizers, which allows larger batch sizes.
The accuracy of RepOpt-VGG closely matches RepVGG, showing a clearly better trade-off between training efﬁciency and accuracy.
Comparisons with VGGNet-16 and EfﬁcientNets.
As shown in Table 3, the superiority of RepOptVGG over the original VGGNet-16 trained with the identical settings can be explained by the depth (VGGNet-16 has 13 conv layers while RepOpt-VGG-B2 has 27) and better architectural design (e.g., replacing max-pooling with stride-2 conv).
Then we compare RepOpt-VGG with EfﬁcientNets, which have abundant high-quality structural priors.
As EfﬁcientNet (Tan & Le, 2019) did not report the number of epochs, we ﬁrst train with the identical settings (BS=32) as described before.
The input resolution is larger for EfﬁcientNets because it is considered as an architectural hyper-parameter of
EfﬁcientNets.
Table 3 shows RepOpt-VGG delivers a favorable speed-accuracy trade-off.
For the
EfﬁcientNet-B4/B5 models, we extend the training epochs to 300 to narrow the gap between our implementation and the reported results (Tan & Le, 2019).
Since EfﬁcientNets use SE Blocks (Hu et al., 2018) and higher test resolution, we also increase the test resolution of RepOpt-VGG-L2 toand insert SE Blocks after each 3 × 3 conv.
Due to the limited GPU memory, we reduce the batch size while testing the throughput.
To highlight the training efﬁciency of our model, we still use training resolution of 224 and only train for 200 epochs.
Impressively, RepOpt-VGG outperforms the ofﬁcial
EfﬁcientNet-B4, even though the latter consumed much more training resources.
4. A BLATION S TUDIES
With RepOpt-VGG-B1, we ﬁrst present baseline results with advanced optimizers (Shazeer & Stern, 2018; Loshchilov & Hutter, 2017; Kingma & Ba, 2014; Tieleman & Hinton, 2012), which can be viewed as re-parameterizing the training dynamics but without model-speciﬁc prior knowledge.
Then we let the RepOptimizer not change the initialization or not modify the gradients.
Table 4 shows either ablation degrades the performance because the training dynamics become no longer equivalent to training a CSLA model.
Furthermore, we change the constant scales to 1) all ones, 2) same as the
[Table]
[Table]
initial values ( (cid:113) 2 l ) in the HS model or 3) the value obtained by averaging the scale values across all the channels for each layer.
In all the cases, the performance degrades, suggesting that the diverse constant scales, which instruct the RepOptimizer to change the training dynamics differently for each channel, encode the vital model-speciﬁc knowledge.
Note that our methodology generalizes beyond SGD, which applies to the other update rules like AdamW.
We present the result of an MLP model Ding et al.
(2022) and an AdamW-based RepOptimizer in the Appendix to show the generality.
4. R EP O PTIMIZERS ARE M ODEL -S PECIFIC BUT D ATASET -A GNOSTIC
The hyper-parameters searched on CIFAR-100 are transferrable to ImageNet, suggesting the RepOptimizers may be model-speciﬁc but dataset-agnostic .
We further investigate by also Hyper-Searching on ImageNet and Caltech256 (Grifﬁn et al., 2007) and using Caltech256 as another target dataset.
The training and search settings on Caltech256 are the same as CIFAR-100, and the ImageNet search settings are the same as the ImageNet training settings.
We have the following observations (Table 5).
The hyper-parameters of RepOptimizers searched on the target datasets deliver no better results than those obtained on a different dataset: by searching and training both on ImageNet, the ﬁnal accuracy is 78.43%, which closely matches the outcome of the CIFAR-searched RepOptimizer.
Please note that searching on ImageNet does not mean training the same model twice on ImageNet because the second training only inherits the HS model’s trained scales into the RepOptimizer but not the other trained parameters.
With different hyper-parameter sources, RepOptimizers deliver similar results on the target datasets, suggesting that the RepOptimizers are dataset-agnostic.
Interestingly, the accuracy on the search dataset does not reﬂect the performance on the target dataset (Table 6).
As RepOpt-VGG is designed for ImageNet, it has ﬁve stride-2 conv layers for downsampling.
Training its corresponding HS model on CIFAR-100 seems unreasonable because CIFAR-100 has a resolution of 32 × 32, meaning the 3 × 3 conv layers in the last two stages operate on 2 × 2 and 1 × 1 feature maps.
As expected, the accuracy of the HS model is low on CIFAR (54.53%), but the searched RepOptimizer works well on ImageNet.
As we reduce the downsampling ratio of HS model on CIFAR-100 by re-conﬁguring the ﬁve original downsampling layers to stride-1, the accuracy of HS model improves, but the corresponding RepOpt-VGG degrades.
Such an observation serves as another evidence that the searched constants are model-speciﬁc , hence the RepOptimizers are model-speciﬁc: the original model has a total downsampling ratio of 32 × , but setting a layer’s stride to 1 makes a different HS model with 16 × downsampling, so the constants searched by such an HS model predictably work poorly with the original 32 × -downsampling model.
On downstream tasks including COCO detection (Lin et al., 2014) and Cityscapes (Cordts et al., 2016) semantic segmentation, RepOpt-VGG also performs on par with RepVGG (Table 7).
For
[Table] [Table]
[Figure]
COCO, we use the implementation of Cascade Mask R-CNN (He et al., 2017; Cai & Vasconcelos, 2019) in MMDetection (Chen et al., 2019) and the 1x training schedule.
For Cityscapes, we use
UperNet (Xiao et al., 2018) and the 40K-iteration schedule in MMSegmentation (Contributors, 2020).
4. R EP O PTIMIZERS FOR E ASY Q UANTIZATION
Quantizing a structurally re-parameterized model (i.e., a model converted from a training-time model with extra structures) may result in signiﬁcant accuracy drop.
For example, quantizing a RepVGG-B to INT8 with a simple PTQ (Post-Training Quantization) method reduces the top-1 accuracy to around 54.55%.
Meanwhile, directly quantizing a RepOpt-VGG only results in 2.5% accuracy drop.
We investigate the parameter distribution of the conv kernel of a RepOpt-VGG-B1 and an inference-time RepVGG-B1.
Speciﬁcally, we ﬁrst analyze the 8th 3 × 3 layer in the 16-layer stage of RepOpt-VGGB1 and then its counterpart from RepVGG-B1.
Fig.
3 and Table 8 shows the distributions of the kernel parameters from RepVGG-B1 are obviously different and the standard deviation is almost 4 × as the RepOpt-VGG kernel.
In Appendix C, we reveal that the structural conversion (i.e., BN fusion and branch addition) cause the quantization-unfriendly parameter distribution, and RepOptimizer naturally solves the problem because there is no structural conversion at all.
C ONCLUSIONS AND L IMITATIONS
This paper proposes to shift a model’s priors into an optimizer.
We hope this paper sparks further research beyond the realms of well-designed model structures.
However, though the methodology is empirically validated, we believe further investigations (e.g., a mathematically provable bound) will be useful, which may require a much deeper understanding of the black box of deep neural networks.
Another major limitation is that our implementation presented in this paper relies on the linearity of operations because we desire to make the training dynamics of a single operator equivalent to a complicated block.
However, our methodology of shifting the structural priors into the training process may generalize beyond equivalently mimicking the training dynamics of another structure.
For examples, we may use some information derived from the model structure to guide the training, but do not have to ensure the equivalency to another model.
[Table]
A PPENDIX A: P ROOF OF CSLA = GR
CSLA means each branch only comprises one differentiable linear operator with trainable parameters (e.g., conv, FC, scaling layer) and no training-time nonlinearity like BN or dropout.
We note that training a CSLA block with regular SGD is equivalent to training a single operator with modiﬁed gradients .
We begin with a simple case where the CSLA block has two parallel conv kernels of the same shape each scaled by a constant scalar.
Let α A , α B be the two constant scalars, W ( B ) , W ( B ) be the two conv kernels.
Let X
and Y be the input and output, the computation ﬂow of the CSLA counterpart is formulated as Y CSLA =
α A (X ∗ W A ) + α B (X ∗ W ( B ) ) , where ∗ denotes convolution.
For the GR counterpart, we directly train the target structure parameterized by W (cid:48) , so that Y GR =
X ∗ W (cid:48) .
Assume the objective function is L , the number of training iterations is i , the gradient of a speciﬁc kernel W is ∂L∂ W and F ( ∂L∂ W (cid:48) ) denotes an arbitrary transformation on the gradient of the GR counterpart.
Proposition : there exists a transformation F determined by only α A and α B so that updating W (cid:48) with F ( ∂L∂ W (cid:48) ) will ensure
[Equation]
Proof: By the additivity and homogeneity of conv, we need to ensure
[Equation]
At iteration 0, a correct initialization ensures the equivalency: assume W ( A )(0) , W ( B )(0) are the arbitrarily initialized values, the initial condition is
[Equation]
so that
[Equation]
Then we use mathematical induction to show that with proper transformations on the gradients of
W (cid:48) , the equivalency always holds.
Let λ be the learning rate, the update rule is
[Equation]
After updating the CSLA counterpart [Equation] we have
[Equation]
We use F ( ∂L∂ W (cid:48) ) to update W (cid:48) , which means
[Equation]
Assume the equivalency holds at iteration
i ( i ≥ 0) .
By Eq. 6, 10, 11, we must ensure
[Equation]
Taking the partial derivatives on Eq.
[Equation]
we have
[Equation]
Then we arrive at
[Equation]
14) which is exactly the formula of the transformation.
With Eq.
14, we ensure α A W ( A )( i +1)
+ α B W ( B )( i +1)
=
W (cid:48)
( i +1) , assuming α A W ( A )( i ) + α B W ( B )( i ) =
W (cid:48)
( i ) .
By the initial condition (Eq.
8) and the mathematical induction, the equivalency for any i ≥ 0 is proved.
Q.E.D.
In conclusion, the gradients we use for updating the operator of the GR counterpart should be simply scaled by a constant factor, which is ( α 2 A + α 2 B ) in this case.
By our deﬁnition, this is exactly the formulation we desire to construct the Grad Mult with the constant scales.
More precisely, with constants α A , α B and two independently initialized (e.g., by MSRAinitialization
He et al.
kernels W ( A )(0) , W ( B )(0) , the following two training processes always give mathematically identical outputs, given the same sequences of inputs.
CSLA : construct two conv layers respectively initialized as W ( A )(0) and W ( B )(0) and train the CSLA block ( Y CS = α A (X ∗ W A ) + α B (X ∗ W B ) ).
GR : construct a single conv parameterized by W (cid:48) , initialize W (cid:48) (0) =
W ( A )(0) +W ( B )(0) , and modify the gradients by Eq.
14 before every update.
In this proof, we assume the constant scales are scalars and the two conv kernels are of the same shape for the simplicity, so the Grad Mult is a scalar.
Otherwise, the Grad Mult will be a tensor and the entries should be respectively computed with the scales on the corresponding positions, just like the formulation to construct the Grad Mults for RepOpt-VGG, which is presented in the paper.
Except for conv, this proof generalizes to any linear operator including FC and scaling.
[Table]
A PPENDIX B: D ISCUSSIONS OF THE I NITIALIZATION I NITIALIZATION OF THE HS M ODEL
For Hyper-Search (HS), the trainable scales of the HS model are initialized according to the layer’s depth.
Let l be the layer’s depth (i.e., the ﬁrst block in each stage that has an identity branch is indexed as l = 1 , the next has l = 2 ), the trainable scales (denoted by s , t in the paper) after the conv layers are initialized as (cid:113) 2 l .
The trainable scales in the identity branches are all initialized as 1 .
Such an initialization strategy is motivated by our observation: in a deep model with identity shortcuts (e.g., ResNet, RepVGG and an HS model of RepOpt-VGG), the shortcut’s effect should be more signiﬁcant at deeper layer at initialization.
In other words, the deeper transformations should be more like an identity mapping to facilitate training Shao et al.
Intuitively, in the early stage of training, the shallow layers learn most and the deep layers do not learn much Zeiler & Fergus (2014), because the deep layers cannot learn well without good features from shallow layers.
For a quantitative analysis, we construct a ResNet with a comparable architecture to RepOpt-VGG-B1, which means the ResNet’s ﬁrst three stages have 4, 6, 16 blocks, respectively.
For the blocks within the 16-block stage (except the ﬁrst one because it has no identity path), we record the variance of the identity path together with the variance of the sum, and compute the ratio of the former to the latter.
Fig.
4 shows that the identity variance ratio is larger at deeper layers.
Such a phenomenon is expected as the output of every residual block has a mean of 0 and a variance of 1 (because it goes through a batch normalization layer).
After every shortcut-addition, the variance of the identity path increases, so that it becomes more and more signiﬁcant compared to the parallel residual block.
This discovery is consistent with a prior discovery Goyal et al.
(2017) that initializing the scaling factor of the aforementioned batch normalization (BN) layer as 0 facilitates training.
We did not observe a similar pattern in the HS model.
Predictably, as there is no BN before the addition, the variances of the three branches all become larger as the depth increases so that the identity variance ratio maintains.
Therefore, in order to produce a similar pattern, we initialize the trainable scales after the conv layers in the HS model according to the depth.
Speciﬁcally, by initializing the trainable scales as (cid:113) 2 l , we make the identity branch more signiﬁcant at deeper layers (Fig. 4).
By doing so, the HS model and the RepOpt-VGG model are both improved (Table 9).
Of note is that such an initialization scheme may not be optimal and we do not seek for a better initialization since the performance on the target dataset is not closely related to the accuracy of the
HS model, as shown in Sec. 4.3.
I NITIALIZATION OF THE T ARGET M ODEL
Of note is that the target model on the target dataset is initialized independent of the initial values of the conv kernels in the HS model.
In other words, the only knowledge inherited from the HS model is the trained scales and we do not need to record any other initial information of the HS model.
More precisely, let the initial state (including the values of every randomly initialized conv kernel) of the
HS model and the target model be ˆ Θ and Θ (cid:48) , respectively, we note that though the RepOptimizer’s hyper-parameters are obtained via a training process beginning with ˆΘ on the search dataset, it works well with Θ (cid:48) on the target dataset.
As shown in the paper, we simply initialize a 3 × 3 kernel in the
RepOpt-VGG as the equivalent kernel of the imaginary CSLA block computed with the trained scales inherited from the HS model and two independently initialized conv kernels.
This discovery, again, shows that the RepOptimizers are speciﬁc to model structures , not datasets nor a certain initial state .
   EORFNLQGH[RIVWDJH    
  L G H Q W L W \ 
YD U L D Q FH  U D W L R 5HV1HW +6PRGHOLQLWDV +6PRGHOLQLWE\GHSWK
[Table]
A PPENDIX C: D ISCUSSIONS OF THE Q UANTIZATION Q UANTIZATION R ESULTS
Quantizing a structurally re-parameterized network may result in a signiﬁcant drop in accuracy, which limits its deployment.
For example, directly quantizing a RepVGG-B1 to INT8 with a simple
Post-Training Quantization (PTQ) method reduces the top-1 accuracy to around 54.55%, making the model completely unusable.
Even worse, as a re-parameterized RepVGG has no BN layers anymore (and it is tricky to add BN layers into it again without harming the performance), ﬁnetuning with a Quantization-Aware Training (QAT) method is not straightforward to implement.
Currently, the quantization of structurally re-parameterized models remains an open problem, which may be addressed by a custom quantization strategy.
Compared to RepVGG, a RepOpt-VGG has the same inference-time structure.
In contrast, we naturally solve the problem of quantization by completely eliminating the need for any structural transformations.
For examples, a simple PTQ method provided by the PyTorch ofﬁcial library (torch.quantization) can quantize RepOpt-VGG-B1 to INT8 with only 2.58% drop in the accuracy (78.47% → 75.89%).
And with a simple QAT method 2 , we narrow the gap to only 0.23% (78.47% → 78.24%) after only 10-epoch ﬁnetuning.
I NVESTIGATION INTO P ARAMETER D ISTRIBUTION
To understand why RepVGG is difﬁcult to quantize, we ﬁrst investigate the parameter distribution of conv kernels of a RepOpt-VGG-B1 and an inference-time RepVGG-B1.
Speciﬁcally, we sample the 8th 3 × 3 layer in the 16-layer stage of RepOpt-VGG-B1 together with its counterpart from
RepVGG-B1.
Fig.
3 and Table 8 in the paper show that the distribution of the kernel parameters of
RepVGG-B1 is obviously different from the RepOpt-VGG kernel.
[Table]
    
[Figure]
       
[Table]
   
[Table]

   
 [Table]
[Table]  
[Table]

[Table]

[Table]

[Table]
     .HUQHO9DOXHV 
       (
[Figure]
RepVGG
[Table]
     .HUQHO9DOXHV 
      
[Table]
[Figure]
RepVGG
[Table]
 
[Table]

       
[Table]
[Figure]
RepVGG
[Figure]
positions.
We continue to investigate the sources of such a high-variance parameter distribution of RepVGG.
Since a training-time RepVGG block undergoes two transformations to ﬁnally become a single 3 ×conv, i
[Table]
e., 1) fusing BN and 2) adding up branches, we evaluate the effects of the two transformations separately.
Moreover, since the branch addition involves the identity branch, 1 × 1 conv and only the central points of the 3 × 3 kernels, we show the standard deviation of the central and the surrounding points (i.e., the eight non-central points in a 3 × 3 matrix) separately (Fig. 5, Table 11).
We make the following observations.
In the 3 × 3 kernel sampled from RepVGG, the weights at different locations have similar standard deviation, and fusing BN enlarges the standard deviation.
After branch addition, the standard deviation of the central points increases signiﬁcantly, which is expected.
The central points of the kernel form RepOpt-VGG have a larger standard deviation than the surrounding points (but lower than the converted kernel from RepVGG), which is expected because we use a Grad Mult with larger values at the central positions.
The RepOpt-VGG kernel has much lower standard deviation than the converted RepVGG kernel, which is expected since it undergoes no BN fusion nor branch addition.
4) Interestingly, even before BN fusion, the surrounding points of the 3 × 3 kernel in RepVGG has a larger variance than the counterparts in the RepOpt-VGG kernel.
This can be explained that placing a BN after a conv can be viewed as multiplying a scaling factor to the conv’s kernel and the scaling factor varies signiﬁcantly from batch to batch (because the batch-wise mean and variance may differ signiﬁcantly), leading to high variance of the conv kernel.
In summary, compared to Structural Re-parameterization, a signiﬁcant strength of Gradient Reparameterization is the capability to train a model without transforming the structures, which is not only efﬁcient but also beneﬁcial to the quantization performance.
A PPENDIX D: G ENERALIZING TO A DAM W AND V ISION MLP
We provide more evidence of the potential generality of the method by generalizing to a vision MLP model.
More precisely, for transforming the features, a vision MLP performs the spatial aggregations
[Table] with fully-connected (FC) layers. In other words, the feature maps should be ﬂattened into vectors and fed into an FC layer, then the outputs are reshaped back into feature maps.
In this way, an output pixel is connected to every input pixel.
However, the locality of images (i.e., a pixel is more related to its neighbors than the distant pixels) is an important prior.
As the vision MLPs lack local priors, they may not perform well on vision tasks.
As another showcase, we use RepOptimizers to incorporate the local priors into vision MLPs to improve the performance on ImageNet.
Speciﬁcally, let an FC kernel be W ∈ R ohw × chw , where h and w are the height and width of the feature map, o and c and the numbers output and input channels.
Intuitively, we can strengthen a subset of the entries of W to create the local prior.
This can be implemented by adding a sparse Grad Mult to the original Grad Mult of W , but constructing such a Grad Mult seems difﬁcult.
As described in the paper, we start from considering an imaginary CSLA block to construct the Grad
Mult.
We simply follow the architectural design of RepMLPNet Ding et al.
(2022): we add two conv layers (3 × 3 and 1 × 1) parallel to the FC layer.
The outputs of the three layers respectively go through three constant channel-wise scaling layers and are added up.
Obviously, the locality is implemented by the conv branches.
We use the same Hyper-Search settings as described in the paper to search the scales of the three imaginary scaling layers.
Then we construct the Grad Mult of W , which is similar to the case of
RepOpt-VGG.
In brief, an entry in an FC kernel corresponds to a connection between two pixels, so that an entry in the Grad Mult is related to the constant scales following a conv layer if the entry corresponds to a connection that is covered by the sliding window of the convolution.
Moreover, as the original RepMLPNet was trained with AdamW, we also combine AdamW with RepOptimizer by inserting the Grad Mults into the AdamW update function.
Just like the case of RepOpt-VGG with
SGD, the formulas for constructing the Grad Mults for RepMLPNet with AdamW can be derived in a similar way.
We will make the code publicly available.
We adopt RepMLPNet, which has a complicated training-time structure, as the baseline.
The
ImageNet accuracy of RepMLPNet-T224 is 76.48, while removing the conv and BN layers (i.e., training without the locality) results in an accuracy of 74.33.
We train the MLP model without the conv and BN layers but our RepOptimizer, which is referred to as RepOpt-MLP-T224, so that the structural priors are shifted into the optimizer.
We obtain an accuracy of 76.35, which closely matches that of the RepMLPNet, just like RepOpt-VGG closely matches the accuracy of RepVGG.