Original: RMN ET : E QUIVALENTLY R EMOVING R ESIDUAL C ONNECTION FROM N ETWORKS
Revised: RMN ET : E QUIVALENTLY R EMOVING R ESIDUAL C ONNECTION FROM N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology.
Revised: Although residual connection enables training very deep neural networks, it is not friendly for online inference due to its multi-branch topology.

Original: This encourages many researchers to work on designing DNNs without residual connections at inference.
Revised: This encourages many researchers to work on designing DNNs without residual connections at inference.

Original: For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow.
Revised: For example, RepVGG re-parameterizes multi-branch topology to a VGG-like (single-branch) model when deploying, showing great performance when the network is relatively shallow.

Original: However, RepVGG can not transform
Revised: However, RepVGG can not transform

Original: ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear Blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks.
Revised: ResNet to VGG equivalently because re-parameterizing methods can only be applied to linear Blocks and the non-linear layers (ReLU) have to be put outside of the residual connection which results in limited representation ability, especially for deeper networks.

Original: In this paper, we aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation on ResBlock.
Revised: In this paper, we aim to remedy this problem and propose to remove the residual connection in a vanilla ResNet equivalently by a reserving and merging (RM) operation on ResBlock.

Original: Speciﬁcally, the RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connections without changing the original output.
Revised: Speciﬁcally, the RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connections without changing the original output.

Original: RMNet basically has two advantages: 1) it achieves a better accuracy-speed trade-off compared with ResNet and RepVGG; 2) its implementation makes it naturally friendly for high ratio network pruning.
Revised: RMNet basically has two advantages: 1) it achieves a better accuracy-speed trade-off compared with ResNet and RepVGG; 2) its implementation makes it naturally friendly for high ratio network pruning.

Original: Extensive experiments are performed to verify the effectiveness of
Revised: Extensive experiments are performed to verify the effectiveness of

Original: RMNet.
Revised: RMNet.

Original: We believe the ideology of RMNet can inspire many insights on model design for the community in the future.
Revised: We believe the ideology of RMNet can inspire many insights on model design for the community in the future.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since AlexNet (Krizhevsky et al., 2012) came out, the state-of-the-art CNN architecture has become deeper and deeper.
Revised: Since AlexNet (Krizhevsky et al., 2012) came out, the state-of-the-art CNN architecture has become deeper and deeper.

Original: For example, AlexNet only has 5 convolutional layers, it is soon extended to 19 and 22 layers by VGG network (Simonyan & Zisserman, 2015) and GoogLeNet(Szegedy et al., 2015; 2016; 2017), respectively.
Revised: For example, AlexNet only has 5 convolutional layers, it is soon extended to 19 and 22 layers by VGG network (Simonyan & Zisserman, 2015) and GoogLeNet(Szegedy et al., 2015; 2016; 2017), respectively.

Original: However, deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers, repeated multiplication may make the gradient inﬁnitely small or large.
Revised: However, deep networks that simply stack layers are hard to train because of the gradient vanishing and exploding problem — as the gradient is back-propagated to earlier layers, repeated multiplication may make the gradient inﬁnitely small or large.

Original: This problem has been largely addressed by the normalized initialization (LeCun et al., 2012; Glorot & Bengio, 2010; Saxe et al., 2013;
Revised: This problem has been largely addressed by the normalized initialization (LeCun et al., 2012; Glorot & Bengio, 2010; Saxe et al., 2013;

Original: He et al., 2015) and intermediate normalization layers (Ioffe & Szegedy, 2015), which enable networks with tens of layers converging.
Revised: He et al., 2015) and intermediate normalization layers (Ioffe & Szegedy, 2015), which enable networks with tens of layers converging.

Original: Meanwhile, another degradation problem has been exposed: with the increase of the network depth, accuracy gets saturated and then degrades rapidly.
Revised: Meanwhile, another degradation problem has been exposed: with the increase of the network depth, accuracy gets saturated and then degrades rapidly.

Original: ResNet (He et al., 2016a;b) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output.
Revised: ResNet (He et al., 2016a;b) addresses the degradation problem and achieves 1K+ layers models by adding a residual connection from the input of a block to the output.

Original: Instead of hoping each stacked layer directly ﬁt a desired underlying mapping, ResNet lets these layers ﬁt a residual mapping.
Revised: Instead of hoping each stacked layer directly ﬁt a desired underlying mapping, ResNet lets these layers ﬁt a residual mapping.

Original: When the identity mapping is optimal, it would be easier to push the residual to zero than to ﬁt an identity mapping by a stack of nonlinear layers.
Revised: When the identity mapping is optimal, it would be easier to push the residual to zero than to ﬁt an identity mapping by a stack of nonlinear layers.

Original: As ResNet gains more and more popularity, researchers propose many new architectures, e.g. (Hu et al., 2018; Li et al., 2019; Tan &
Revised: As ResNet gains more and more popularity, researchers propose many new architectures, e.g. (Hu et al., 2018; Li et al., 2019; Tan &

Original: Le, 2019; 2021), based on ResNet and interpret the success from different aspects.
Revised: Le, 2019; 2021), based on ResNet and interpret the success from different aspects.

Original: However, ResDistill (Li et al., 2020) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, which will slow down the inference process.
Revised: However, ResDistill (Li et al., 2020) points that the residual connections in ResNet-50 account for about 40 percent of the entire memory usage on feature maps, which will slow down the inference process.

Original: Besides, residual connections in the network are not friendly for ‘network pruning’
Revised: Besides, residual connections in the network are not friendly for ‘network pruning’

Original: (Ding et al., 2021b).
Revised: (Ding et al., 2021b).

Original: By contrast, the VGG-like model, also called the plain model in this paper, has only one path and is fast, memory economical, and parallelization friendly.
Revised: By contrast, the VGG-like model, also called the plain model in this paper, has only one path and is fast, memory economical, and parallelization friendly.

Original: RepVGG (Ding et al., 2021b) proposes a powerful approach to remove residual connections via re-parametrization at inference.
Revised: RepVGG (Ding et al., 2021b) proposes a powerful approach to remove residual connections via re-parametrization at inference.

Original: Speciﬁcally, RepVGG adds 3 × 3 convolution, 1 × 1 convolution, and identity together for training.
Revised: Speciﬁcally, RepVGG adds 3 × 3 convolution, 1 × 1 convolution, and identity together for training.

Original: The BN layer was added at the end of each branch and the ReLU was added after the addition operation.
Revised: The BN layer was added at the end of each branch and the ReLU was added after the addition operation.

Original: During training, RepVGG only needs to learn the residual mapping, while in the inference stage, re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3 ×convolution layers plus ReLU operation, which has a favorable speed-accuracy trade-off compared to ResNet.
Revised: During training, RepVGG only needs to learn the residual mapping, while in the inference stage, re-parameterization was utilized to transform the basic block of RepVGG into a stack of 3 ×convolution layers plus ReLU operation, which has a favorable speed-accuracy trade-off compared to ResNet.

Original: However, we ﬁnd that the performance of RepVGG suffers severe degradation when the network goes deeper, we provide experiments to verify this observation in Section 4.1.
Revised: However, we ﬁnd that the performance of RepVGG suffers severe degradation when the network goes deeper, we provide experiments to verify this observation in Section 4.1.

Original: In this paper, we introduce a novel approach named RM operation, which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged.
Revised: In this paper, we introduce a novel approach named RM operation, which can remove the residual connection with non-linear layers inside it and keep the result of the model unchanged.

Original: RM operation reserves the input feature maps by the ﬁrst convolution layer, BN Layer and ReLU Layer, then merges them with the output feature maps by the last convolution in the ResBlock.
Revised: RM operation reserves the input feature maps by the ﬁrst convolution layer, BN Layer and ReLU Layer, then merges them with the output feature maps by the last convolution in the ResBlock.

Original: With this method, we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism.
Revised: With this method, we can equivalently convert a pre-trained ResNet or MobileNetV2 to an RMNet model to increase the degree of parallelism.

Original: Besides, the architecture of RMNet makes it friendly for pruning since there are no residual connections.
Revised: Besides, the architecture of RMNet makes it friendly for pruning since there are no residual connections.

Original: We summarized our main contributions as follows:
Revised: We summarized our main contributions as follows:

Original: • We ﬁnd that removing residual connection by re-parameterization methods has its limitation, especially when the depth of the model is
Revised: • We ﬁnd that removing residual connection by re-parameterization methods has its limitation, especially when the depth of the model is

Original: large
Revised: large

Original: It lies in the non-
Revised: It lies in the non-

Original: linear
Revised: linear

Original: operation that can not be put inside of residual connection for re-parameterization.
Revised: operation that can not be put inside of residual connection for re-parameterization.

Original: • We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps.
Revised: • We propose a novel method named RM operation which can remove residual connection across non-linear layers without changing the output by reserving the input feature maps and merging them with the output feature maps.

Original: • With RM operation, we can convert the ResBlocks to a stack of convolutions and ReLUs, which achieves a better accuracy-speed trade-off and make it very friendly for pruning.
Revised: • With RM operation, we can convert the ResBlocks to a stack of convolutions and ReLUs, which achieves a better accuracy-speed trade-off and make it very friendly for pruning.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Advantage of Residual Networks over Plain Model.
Revised: Advantage of Residual Networks over Plain Model.

Original: He et al.
Revised: He et al.

Original: (He et al., 2016a) introduce ResNet for addressing the degradation problem.
Revised: (He et al., 2016a) introduce ResNet for addressing the degradation problem.

Original: In addition, the gradient of layers in PreActResNet
Revised: In addition, the gradient of layers in PreActResNet

Original: (He et al., 2016b) does not vanish even when the weights are arbitrarily small, which leads to nice backward propagation properties.
Revised: (He et al., 2016b) does not vanish even when the weights are arbitrarily small, which leads to nice backward propagation properties.

Original: (Weinan, 2017) shows the connection between dynamical systems and deep learning.
Revised: (Weinan, 2017) shows the connection between dynamical systems and deep learning.

Original: (Lu et al., 2018) regards ResNet as an Euler discretization of Ordinary Differential
Revised: (Lu et al., 2018) regards ResNet as an Euler discretization of Ordinary Differential

Original: Equations (ODE).
Revised: Equations (ODE).

Original: NODEs (Chen et al., 2018) replaces residual block with neural ODEs for better training extremely deep networks.
Revised: NODEs (Chen et al., 2018) replaces residual block with neural ODEs for better training extremely deep networks.

Original: (Balduzzi et al., 2017) identiﬁes shattered gradients problems making the training of DNNs difﬁcult.
Revised: (Balduzzi et al., 2017) identiﬁes shattered gradients problems making the training of DNNs difﬁcult.

Original: They show that the correlation between gradients in the standard feedforward network decays exponentially with depth.
Revised: They show that the correlation between gradients in the standard feedforward network decays exponentially with depth.

Original: In contrast, the gradients in ResNet are far more resistant to shattering, decaying sublinearly.
Revised: In contrast, the gradients in ResNet are far more resistant to shattering, decaying sublinearly.

Original: (Veit et al., 2016) shows that residual networks can be seen as a collection of many paths of differing lengths.
Revised: (Veit et al., 2016) shows that residual networks can be seen as a collection of many paths of differing lengths.

Original: From this view, n -blocks’
Revised: From this view, n -blocks’

Original: ResNet have O (2 n ) implicit paths connecting input and output, and that adding a block doubles the number of paths.
Revised: ResNet have O (2 n ) implicit paths connecting input and output, and that adding a block doubles the number of paths.

Original: DNNs Without Residual Connection.
Revised: DNNs Without Residual Connection.

Original: A recent work(Oyedotun et al., 2020) combined several techniques including Leaky ReLU, max-norm, and careful initialization to train a 30 layer plain
Revised: A recent work(Oyedotun et al., 2020) combined several techniques including Leaky ReLU, max-norm, and careful initialization to train a 30 layer plain

Original: ConvNet, which could reach 74.6% top-1 accuracy, 2% lower than their baseline.
Revised: ConvNet, which could reach 74.6% top-1 accuracy, 2% lower than their baseline.

Original: A mean-ﬁeld theory-based method (Xiao et al., 2018) is proposed to train extremely deep ConvNets without branches.
Revised: A mean-ﬁeld theory-based method (Xiao et al., 2018) is proposed to train extremely deep ConvNets without branches.

Original: However, 1K-layer plain networks only achieve 82% accuracy on CIFAR-10.
Revised: However, 1K-layer plain networks only achieve 82% accuracy on CIFAR-10.

Original: (Balduzzi et al., 2017) use a ”looks linear” initialization method and CReLUs (Shang et al., 2016) to train a 198 layer plain model to 85% accuracy on CIFAR-10.
Revised: (Balduzzi et al., 2017) use a ”looks linear” initialization method and CReLUs (Shang et al., 2016) to train a 198 layer plain model to 85% accuracy on CIFAR-10.

Original: Though the theoretical contributions were insightful, the models were not practical.
Revised: Though the theoretical contributions were insightful, the models were not practical.

Original: One can get DNNs without residual connection in a reparameterization manner: re-parameterization (Ding et al., 2019; 2021a) means using the params of a structure to parameterize another set of params.
Revised: One can get DNNs without residual connection in a reparameterization manner: re-parameterization (Ding et al., 2019; 2021a) means using the params of a structure to parameterize another set of params.

Original: Those approaches ﬁrst train a model with residual connection and remove residual connection by re-parameterization when inference.
Revised: Those approaches ﬁrst train a model with residual connection and remove residual connection by re-parameterization when inference.

Original: DiracNet (Zagoruyko & Komodakis, 2017) uses the addition of an identity matrix and convolutional matrix for forwarding propagation, the parameters of convolution need only to learn the residual function like ResNet.
Revised: DiracNet (Zagoruyko & Komodakis, 2017) uses the addition of an identity matrix and convolutional matrix for forwarding propagation, the parameters of convolution need only to learn the residual function like ResNet.

Original: After training, DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference.
Revised: After training, DiracNet adds the identity matrix to the convolutional matrix and uses the reparameterized model for inference.

Original: However, DiracNet can only train up to 34 layer plain model which has 72.83% accuracy on ImageNet.
Revised: However, DiracNet can only train up to 34 layer plain model which has 72.83% accuracy on ImageNet.

Original: RepVGG (Ding et al., 2021b) deploys residual neural network only at the training time.
Revised: RepVGG (Ding et al., 2021b) deploys residual neural network only at the training time.

Original: At the inference, RepVGG can transform the residual block into a plain module consisting of a stack of 3 × 3 convolution and ReLU via re-parameterization.
Revised: At the inference, RepVGG can transform the residual block into a plain module consisting of a stack of 3 × 3 convolution and ReLU via re-parameterization.

Original: The difference between DiracNet and RepVGG is each block in DiracNet has two branches (identity without BN and 3 × 3 ConvBN) while RepVGG has three branches (identity with BN, 3 ×
Revised: The difference between DiracNet and RepVGG is each block in DiracNet has two branches (identity without BN and 3 × 3 ConvBN) while RepVGG has three branches (identity with BN, 3 ×

Original: ConvBN, and 1 × 1 ConvBN).
Revised: ConvBN, and 1 × 1 ConvBN).

Original: However, those re-parameterization methods that used commutative property can only apply to linear layers, i.e., the non-linear layers have to be outside of the residual connection, which limits the potential of neural networks for large depths.
Revised: However, those re-parameterization methods that used commutative property can only apply to linear layers, i.e., the non-linear layers have to be outside of the residual connection, which limits the potential of neural networks for large depths.

Original: Filter Pruning: Filter pruning is a promising solution to accelerate CNNs.
Revised: Filter Pruning: Filter pruning is a promising solution to accelerate CNNs.

Original: Numerous inspiring works prune the ﬁlters by evaluating their importance.
Revised: Numerous inspiring works prune the ﬁlters by evaluating their importance.

Original: Heuristic metrics are proposed, such as the magnitude of convolution kernels (Li et al., 2017), the average percentage of zero activations (APoZ) (Hu et al., 2016).
Revised: Heuristic metrics are proposed, such as the magnitude of convolution kernels (Li et al., 2017), the average percentage of zero activations (APoZ) (Hu et al., 2016).

Original: There also exists some work to let networks automatically select important ﬁlters.
Revised: There also exists some work to let networks automatically select important ﬁlters.

Original: For example, (Liu et al., 2017a) sparsify the weights in the BN layer to automatically ﬁnd which ﬁlters contribute most to the network performance.
Revised: For example, (Liu et al., 2017a) sparsify the weights in the BN layer to automatically ﬁnd which ﬁlters contribute most to the network performance.

Original: However, For ResNet-based architecture, the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same.
Revised: However, For ResNet-based architecture, the existence of residual connection limits the power of pruning since the dimension of the input and output through the residual connection must keep the same.

Original: Thus pruning ratio of ResNet is not larger than the ratio of the plain model.
Revised: Thus pruning ratio of ResNet is not larger than the ratio of the plain model.

Original: Since RM operation can equivalently transform ResNet to a plain model, the transferred model (RMNet) also has a great advantage on pruning.
Revised: Since RM operation can equivalently transform ResNet to a plain model, the transferred model (RMNet) also has a great advantage on pruning.

Original:  RM OPERATION AND RMN ET : T OWARDS AN E FFICIENT P LAIN N ETWORK 3. RM OPERATION
Revised:  RM OPERATION AND RMN ET : T OWARDS AN E FFICIENT P LAIN N ETWORK 3. RM O PERATION

Original: Figure 1 shows the process of equivalently removing residual connection by RM Operation.
Revised: Figure 1 shows the process of equivalently removing residual connection by RM Operation.

Original: For simplicity, we do not show BN layer in the ﬁgure, and the number of input channels, medium channels, and output channels are the same and are assigned as C .
Revised: For simplicity, we do not show BN layer in the ﬁgure, and the number of input channels, medium channels, and output channels are the same and are assigned as C .

Original: [Figure]
Revised: [Figure]

Original: 0  0 0𝒛 𝒛 ! 𝒙 ! 𝒙 𝑹 𝒙 𝑹𝟏!
Revised: 0 1 0 0 0𝒛 𝒛 ! 𝒙 ! 𝒙 𝑹 𝒙 𝑹𝟏!

Original: 𝒙 ! 𝒙 ! 0 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0 

Revised: 𝒙 ! 𝒙 ! 0 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0 0 00 0 0 0 1 0

Original: [Figure]
Revised: [Figure]

Original: • Reserving:
Revised: • Reserving:

Original: We ﬁrst insert several Dirac initialized ﬁlters (output channels) in Conv 1.
Revised: We ﬁrst insert several Dirac initialized ﬁlters (output channels) in Conv 1.

Original: The number of Dirac initialized ﬁlters is the same as the input channels’ number in the convolution layer.
Revised: The number of Dirac initialized ﬁlters is the same as the input channels’ number in the convolution layer.

Original: The Dirac initialized ﬁlters are deﬁned as 4-dimensional matrix:
Revised: The Dirac initialized ﬁlters are deﬁned as 4-dimensional matrix:

Original: [Equation]
Revised: [Equation]

Original: We can view these ﬁlters in Figure 1(b).
Revised: We can view these ﬁlters in Figure 1(b).

Revised: Every ﬁlter

Original: Every ﬁlter only has one element to be 1, which can reserve the corresponding channel’s information of the input feature map via convolution.
Revised: only has one element to be 1, which can reserve the corresponding channel’s information of the input feature map via convolution.

Original: For the BN layer, to reserve the input feature map, weight w and bias b in BN need to be adjusted to make the BN layer behave like an identity function.
Revised: For the BN layer, to reserve the input feature map, weight w and bias b in BN need to be adjusted to make the BN layer behave like an identity function.

Original: Suppose the running mean and the running variance of the feature maps are µ and σ 2 separately, we set w = √ σ 2 + (cid:15) and bias b = µ .
Revised: Suppose the running mean and the running variance of the feature maps are µ and σ 2 separately, we set w = √ σ 2 + (cid:15) and bias b = µ .

Original: Then for any input x through the BN layer, the output is:
Revised: Then for any input x through the BN layer, the output is:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) =
Revised: where (cid:15) =

Original: 10 − 5 is added to the σ 2 in case of the divisor to be zero.
Revised: 10 − 5 is added to the σ 2 in case of the divisor to be zero.

Original: For the ReLU layer, There are two cases to be considered:
Revised: For the ReLU layer, There are two cases to be considered:

Original: • When the input value through residual connection is non-negative ( ie.e, in ResNet, every ResBlock has a following a ReLU layer, which keeps input values are all non-negative), we can directly use ReLU to reserve the information.
Revised: • When the input value through residual connection is non-negative ( ie.e, in ResNet, every ResBlock has a following a ReLU layer, which keeps input values are all non-negative), we can directly use ReLU to reserve the information.

Original: Since ReLU does not change the non-negative values.
Revised: Since ReLU does not change the non-negative values.

Original: • When the input value through residual connection can be negative (e
Revised: • When the input value through residual connection can be negative (e

Original: g. in MobileNetV2, ReLU is only located in the middle of the ResBlock), we use PReLU instead of ReLU to reserve the information.
Revised: g. in MobileNetV2, ReLU is only located in the middle of the ResBlock), we use PReLU instead of ReLU to reserve the information.

Original: The parameters of PReLU regarding to the additional channels are set to one.
Revised: The parameters of PReLU regarding to the additional channels are set to one.

Original: In this way, PReLU behaves as Identity function.
Revised: In this way, PReLU behaves as Identity function.

Original: From the above analysis, the input feature map can be well reserved by Conv 1, BN, and ReLU in the ResBlock.
Revised: From the above analysis, the input feature map can be well reserved by Conv 1, BN, and ReLU in the ResBlock.

Original: • Merging: We extend input channels in Conv 2 and dirac initialize these channels.
Revised: • Merging: We extend input channels in Conv 2 and dirac initialize these channels.

Original: The value of i -th channel of z is the sum of the original i-th ﬁlters’ output x R 

Revised: The value of i -th channel of z is the sum of the original i-th ﬁlters’ output x R

Original: i and the i-th input feature map x + i , which is equal to y i in the original ResBlock.
Revised: i and the i-th input feature map x + i , which is equal to y i in the original ResBlock.

Original: The whole merging process can be
Revised: The whole merging process can be

Original: formulated
Revised: formulated

Original: as:
Revised: as:

Original: [Equation]
Revised: [Equation]

Original: where [ ∗ , ∗ ] means the concatenation of the two elements.
Revised: where [ ∗ , ∗ ] means the concatenation of the two elements.

Original: Thus, by reserving and merging, we can remove the residual connection without changing the original output of ResBlock.
Revised: Thus, by reserving and merging, we can remove the residual connection without changing the original output of ResBlock.

Original: 3. C ONVERT R ES N ET TO VGG
Revised: 3. C ONVERT R ES N ET TO VGG

Original: In this section, we show how to apply RM operation to convert ResNet into VGG-like RMNet.
Revised: In this section, we show how to apply RM operation to convert ResNet into VGG-like RMNet.

Original: Notice that there exist some DownSample ResBlocks at the ﬁrst block of layer2, layer3, and layer in ResNet.
Revised: Notice that there exist some DownSample ResBlocks at the ﬁrst block of layer2, layer3, and layer in ResNet.

Original: The function of ‘downsample’ is for:
Revised: The function of ‘downsample’ is for:

Original: • doubling the number of channels; • reducing the depth and width of the feature map by half.
Revised: • doubling the number of channels; • reducing the depth and width of the feature map by half.

Original: Even though there are only a few DownSample ResBlocks in ResNet, removing residual connection in such blocks is not as direct as in basic ResBlocks because there exist 1 × 1 convolution in the
Revised: Even though there are only a few DownSample ResBlocks in ResNet, removing residual connection in such blocks is not as direct as in basic ResBlocks because there exist 1 × 1 convolution in the

Original: S= ⨁
Revised: S= ⨁

Original: S= ⨁
Revised: S= ⨁

Original: ConvBN
Revised: ConvBN

Original: ReLU
Revised: ReLU

Original: S= ⨁
Revised: S= ⨁

Original: S= S=
Revised: S= S=

Original: S= S=
Revised: S= S=

Original: S= S=
Revised: S= S=

Original: PReLU (a) DownSample RMBlock (type 1)
Revised: PReLU (a) DownSample RMBlock (type 1)

Original: S= ⨁ S= ⨁
Revised: S= ⨁ S= ⨁

Original: S=
Revised: S=

Original: ConvBN
Revised: ConvBN

Original: ReLU
Revised: ReLU

Original: S= ⨁
Revised: S= ⨁

Original: S= S=
Revised: S= S=

Original: S= S=
Revised: S= S=

Original: S= (b) DownSample RMBlock (type 2) downsample branch and it will change the input value in the path of residual connection.
Revised: S= (b) DownSample RMBlock (type 2) downsample branch and it will change the input value in the path of residual connection.

Original: We propose two solutions in Figure 2 to convert DownSample ResBlock to RMBlock.
Revised: We propose two solutions in Figure 2 to convert DownSample ResBlock to RMBlock.

Original: As shown in Figure 2(a), the original 1 × 1 downsample kernel is padding with zeros to 3 × 3 kernel.
Revised: As shown in Figure 2(a), the original 1 × 1 downsample kernel is padding with zeros to 3 × 3 kernel.

Original: we replace ReLU in the block with PReLU, whose parameters for each channel can control the scale of negative activation, as analyzed in Section 3.1.
Revised: we replace ReLU in the block with PReLU, whose parameters for each channel can control the scale of negative activation, as analyzed in Section 3.1.

Original: For the left (residual) branch, we set the PReLU parameters as zeros, in this way, the function of the PReLU is equal to ReLU.
Revised: For the left (residual) branch, we set the PReLU parameters as zeros, in this way, the function of the PReLU is equal to ReLU.

Original: PReLU is added after the downsample convolution, the parameters of which are initialized by one.
Revised: PReLU is added after the downsample convolution, the parameters of which are initialized by one.

Original: In this way, PReLU is equal to the Identity Mapping function.
Revised: In this way, PReLU is equal to the Identity Mapping function.

Original: New additional channels of the second convolutional layer are Dirac initialized (similar to Figure 1(b)), which keep the feature map after downsampling convolution unchanged.
Revised: New additional channels of the second convolutional layer are Dirac initialized (similar to Figure 1(b)), which keep the feature map after downsampling convolution unchanged.

Original: Then, all the layers in the left and right branches can be merged together into one sequence shown in Figure 2(a).
Revised: Then, all the layers in the left and right branches can be merged together into one sequence shown in Figure 2(a).

Original: In Figure 2(b), we separate the function of downsampling convolution into two convolutions in downsample branch.
Revised: In Figure 2(b), we separate the function of downsampling convolution into two convolutions in downsample branch.

Original: The ﬁrst module is 3 × 3 convolution with Dirac initialized ﬁlters and the stride is 2 for reducing the depth and width of the input feature map by half.
Revised: The ﬁrst module is 3 × 3 convolution with Dirac initialized ﬁlters and the stride is 2 for reducing the depth and width of the input feature map by half.

Original: The feature map through this Conv layer is still non-negative and ReLU can reserve the value unchanged.
Revised: The feature map through this Conv layer is still non-negative and ReLU can reserve the value unchanged.

Original: The second module is 3 × 3 convolution, whose weights are transferred from original 1 × 1 ﬁlters in downsample branch with 0 paddings.
Revised: The second module is 3 × 3 convolution, whose weights are transferred from original 1 × 1 ﬁlters in downsample branch with 0 paddings.

Original: Since the original ﬁlters’ kernel size is 1 × 1, reducing the size of feature maps before doubling the number of channels will not change the receptive ﬁeld in the original downsample branch.
Revised: Since the original ﬁlters’ kernel size is 1 × 1, reducing the size of feature maps before doubling the number of channels will not change the receptive ﬁeld in the original downsample branch.

Original: Both methods can equally convert the DownSample Blocks to RMBlocks.
Revised: Both methods can equally convert the DownSample Blocks to RMBlocks.

Original: The number of parameters of the ﬁrst method is 108 C 2 + 4 C , including Conv 1 ( C × 4 C × 3 × 3 ), PReLU ( 4 C ), Conv( 4 C × 2 C × 3 × 3 ).
Revised: The number of parameters of the ﬁrst method is 108 C 2 + 4 C , including Conv 1 ( C × 4 C × 3 × 3 ), PReLU ( 4 C ), Conv( 4 C × 2 C × 3 × 3 ).

Original: The number of parameters of the second method is 81 C 2 , consisting of Conv( C × 3 C × 3 × 3 ), Conv 2 ( 3 C × 2 C × 3 × 3 ).
Revised: The number of parameters of the second method is 81 C 2 , consisting of Conv( C × 3 C × 3 × 3 ), Conv 2 ( 3 C × 2 C × 3 × 3 ).

Original: We can see the number of parameters of the second method is only 0.75 times of the ﬁrst method.
Revised: We can see the number of parameters of the second method is only 0.75 times of the ﬁrst method.

Original: Thus we use the second method to convert ResNet to
Revised: Thus we use the second method to convert ResNet to

Original: RMNet in our experiment.
Revised: RMNet in our experiment.

Original: 3. C ONVERT M OBILE N ET V2 TO M OBILE N ET V 
Revised: 3. C ONVERT M OBILE N ET V2 TO M OBILE N ET V

Original: Technically, there is no difference between converting ResNet and converting MobileNetV2 by using
Revised: Technically, there is no difference between converting ResNet and converting MobileNetV2 by using

Original: RM operation.
Revised: RM operation.

Original: However, the specialty of the structure of MobileNetV2 allows to further decrease the inference time by using parameter fusion after RM operation.
Revised: However, the specialty of the structure of MobileNetV2 allows to further decrease the inference time by using parameter fusion after RM operation.

Original: From Figure 3, after applying RM operation on MobileNetV2, the residual connections are removed, which leaves two Pointwise-ConvBN layers shown in the dashed box.
Revised: From Figure 3, after applying RM operation on MobileNetV2, the residual connections are removed, which leaves two Pointwise-ConvBN layers shown in the dashed box.

Original: Since convolution and batchnormalization can be both presented by matrix multiplication and there exists no non-linear layer between them, these two Pointwise-ConvBN layers can be fused.
Revised: Since convolution and batchnormalization can be both presented by matrix multiplication and there exists no non-linear layer between them, these two Pointwise-ConvBN layers can be fused.

Original: Mathematically, let X , Y , W ,
Revised: Mathematically, let X , Y , W ,

Original: B be the input feature map, output feature map, weight, and bias of the ﬁrst Pointwise-ConvBN layer in the dashed box and Z , W (cid:48) , B (cid:48) be the output feature map, weight and bias of the second
Revised: B be the input feature map, output feature map, weight, and bias of the ﬁrst Pointwise-ConvBN layer in the dashed box and Z , W (cid:48) , B (cid:48) be the output feature map, weight and bias of the second

Original: Pointwise-ConvBN layer in the dashed box.
Revised: Pointwise-ConvBN layer in the dashed box.

Original: The process of fusing the parameters can be represented as:
Revised: The process of fusing the parameters can be represented as:

Original: P ointwise ConvBN ReLU
Revised: P ointwise ConvBN ReLU

Original: D epthwise ConvBN ReLU
Revised: D epthwise ConvBN ReLU

Original: P ointwise ConvBN
Revised: P ointwise ConvBN

Original: P ointwise ConvBN ReLU
Revised: P ointwise ConvBN ReLU

Original: D epthwise ConvBN ReLU
Revised: D epthwise ConvBN ReLU

Original: P ointwise ConvBN P ointwise ConvBN
Revised: P ointwise ConvBN P ointwise ConvBN

Original: ReLU D epthwise ConvBN ReLU
Revised: ReLU D epthwise ConvBN ReLU

Original: P ointwise ConvBN
Revised: P ointwise ConvBN

Original: P ointwise ConvBN ReLU
Revised: P ointwise ConvBN ReLU

Original: D epthwise ConvBN ReLU
Revised: D epthwise ConvBN ReLU

Original: P ointwise ConvBN
Revised: P ointwise ConvBN

Original: ReLU D epthwise ConvBN ReLU
Revised: ReLU D epthwise ConvBN ReLU

Original: P ointwise ConvBN
Revised: P ointwise ConvBN

Original: RM
Revised: RM

Original: Fuse P ointwise ConvBN ReLU
Revised: Fuse P ointwise ConvBN ReLU

Original: D epthwise ConvBN ReLU
Revised: D epthwise ConvBN ReLU

Original: P ointwise ConvBN
Revised: P ointwise ConvBN

Original: MobileNetV
Revised: MobileNetV

Original: RMNetV
Revised: RMNetV

Original: MobileNetV
Revised: MobileNetV

Original: [Equation]
Revised: [Equation]

Original: The fused weight is W (cid:48) W ∈ R n,c, 1 , 1 and the fused bias is W (cid:48)
Revised: The fused weight is W (cid:48) W ∈ R n,c, 1 , 1 and the fused bias is W (cid:48)

Original: B + B (cid:48)
Revised: B + B (cid:48)

Original: ∈ R n .
Revised: ∈ R n .

Original: Thus, RM operation provides another opportunity to further decrease the inference time by fusing the parameters.
Revised: Thus, RM operation provides another opportunity to further decrease the inference time by fusing the parameters.

Original: After parameter fusion, the architecture of RMNet is identical to MobileNetV1 which is very interesting since the presence of MobileNetV2 is to increases the generalization ability of MobileNetV by utilizing residual connections.
Revised: After parameter fusion, the architecture of RMNet is identical to MobileNetV1 which is very interesting since the presence of MobileNetV2 is to increases the generalization ability of MobileNetV by utilizing residual connections.

Original: However, we show RM operation can inverse this process, i.e., converting MobileNetV2 into MobileNetV1 to make MobileNetV1 great again.
Revised: However, we show RM operation can inverse this process, i.e., converting MobileNetV2 into MobileNetV1 to make MobileNetV1 great again.

Original: We provide experiments in Section 4.3 to verify the effectiveness of this transformation.
Revised: We provide experiments in Section 4.3 to verify the effectiveness of this transformation.

Original: [Figure]
Revised: [Figure]

Original: Since RMNet does not have any residual connections, it is more friendly for ﬁlter pruning.
Revised: Since RMNet does not have any residual connections, it is more friendly for ﬁlter pruning.

Original: In this paper, we adopt Network slimming (Liu et al., 2017b) to prune RMNet because of its simpleness and effectiveness.
Revised: In this paper, we adopt Network slimming (Liu et al., 2017b) to prune RMNet because of its simpleness and effectiveness.

Original: Speciﬁcally, we ﬁrst train ResNet and sparsity the weight in the BN layer.
Revised: Speciﬁcally, we ﬁrst train ResNet and sparsity the weight in the BN layer.

Original: Note an additional BN layer should be added in the residual connection during training since we also need to determine which additional ﬁlters are important after RM operation.
Revised: Note an additional BN layer should be added in the residual connection during training since we also need to determine which additional ﬁlters are important after RM operation.

Original: After training, we convert
Revised: After training, we convert

Original: ResNet into RMNet and prune the ﬁlters according to the weights in BN layers, which is identical to vanilla Network slimming.
Revised: ResNet into RMNet and prune the ﬁlters according to the weights in BN layers, which is identical to vanilla Network slimming.

Original: Figure 4 shows the overall process of pruning.
Revised: Figure 4 shows the overall process of pruning.

Original: Different from the traditional approach, RMNet can serve as a transition to gain a larger pruning ratio.
Revised: Different from the traditional approach, RMNet can serve as a transition to gain a larger pruning ratio.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section aranges as follows: in Section 4.1, we show RMNet has great advantage over RepVGG for deeper networks; in Section 4.2.
Revised: This section aranges as follows: in Section 4.1, we show RMNet has great advantage over RepVGG for deeper networks; in Section 4.2.

Original: we show RMNet can achieve better speed-accuracy tradeoff on CIFAR10, CIFAR100 and ImageNet datasets; in Section 4.3, we show RMNet is applicable for light-weight models; in Section 4.4, we verify the effectivenss of RMNet on network pruning task.
Revised: we show RMNet can achieve better speed-accuracy tradeoff on CIFAR10, CIFAR100 and ImageNet datasets; in Section 4.3, we show RMNet is applicable for light-weight models; in Section 4.4, we verify the effectivenss of RMNet on network pruning task.

Original: The speed of each model is tested on a Tesla V100 GPU with a batch size of 128 by ﬁrst feedingbatches to warm the hardware up, then 50 batches with time usage recorded.
Revised: The speed of each model is tested on a Tesla V100 GPU with a batch size of 128 by ﬁrst feedingbatches to warm the hardware up, then 50 batches with time usage recorded.

Original: 4. R EMAINING HIGH ACCURACY FOR DEEPER NETWORK
Revised: 4. R EMAINING HIGH ACCURACY FOR DEEPER NETWORK

Original: One power of DNN is that the representation ability increases as we raise the network depth.
Revised: One power of DNN is that the representation ability increases as we raise the network depth.

Original: Thus we examine how network depth inﬂuences network performance on RMNet and RepVGG.
Revised: Thus we examine how network depth inﬂuences network performance on RMNet and RepVGG.

Original: Figure 5 shows the network performance on CIFAR-10 and CIFAR-100.
Revised: Figure 5 shows the network performance on CIFAR-10 and CIFAR-100.

Original: For a fair comparison, the same hyper-parameters are deployed for each method: mini-batch size (256), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (200), learning rate decay 0.1 at every 60 epochs.
Revised: For a fair comparison, the same hyper-parameters are deployed for each method: mini-batch size (256), optimizer (SGD), initial learning rate (0.1), momentum (0.9), weight decay (0.0005), number of epochs (200), learning rate decay 0.1 at every 60 epochs.

Original: [Figure]
Revised: [Figure]

Original: From Figure 5, as the depth increases, RMNet can get better accuracy.
Revised: From Figure 5, as the depth increases, RMNet can get better accuracy.

Original: This is because, unlike
Revised: This is because, unlike

Original: RepVGG, RMNet can equivalently convert ResNet into a plain model.
Revised: RepVGG, RMNet can equivalently convert ResNet into a plain model.

Original: Thus the performance will not decrease as the network goes deeper.
Revised: Thus the performance will not decrease as the network goes deeper.

Original: In contrast, the accuracy of RepVGG will decrease (the reasons are analyzed at Appendix A.2 ), e.g. compared to RMNet 133 that achieves 79.57% accuracy on CIFAR-100, RepVGG 133 only has 41.38% accuracy.
Revised: In contrast, the accuracy of RepVGG will decrease (the reasons are analyzed at Appendix A.2 ), e.g. compared to RMNet 133 that achieves 79.57% accuracy on CIFAR-100, RepVGG 133 only has 41.38% accuracy.

Original: 4. B ETTER A CCURACY - SPEED TRADEOFF
Revised: 4. B ETTER A CCURACY - SPEED TRADEOFF

Original: From Section 3, RMNet removes residual connections in the cost of bringing additional parameters.
Revised: From Section 3, RMNet removes residual connections in the cost of bringing additional parameters.

Original: For example, in Figure 1, RM operation doubles the number of parameters of the original ResBlock.
Revised: For example, in Figure 1, RM operation doubles the number of parameters of the original ResBlock.

Original: To alleviate this issue, we use RM operation on the ResNet-based architectures with Inverted Residual Block for designing RMNet.
Revised: To alleviate this issue, we use RM operation on the ResNet-based architectures with Inverted Residual Block for designing RMNet.

Original: Inverted Residual Block (IRB) mainly contains three consecutive steps: point-wise convolution, group-wise convolution, and point-wise convolution.
Revised: Inverted Residual Block (IRB) mainly contains three consecutive steps: point-wise convolution, group-wise convolution, and point-wise convolution.

Original: The ﬁrst ‘pointwise convolution‘ will increase the number of channels of the input feature map by T times and the second ‘point-wise convolution‘ will reduce the number of channels to the original number.
Revised: The ﬁrst ‘pointwise convolution‘ will increase the number of channels of the input feature map by T times and the second ‘point-wise convolution‘ will reduce the number of channels to the original number.

Original: For point-wise convolution, RM operation only increases the number of parameters and FLOPs by 1 T times.
Revised: For point-wise convolution, RM operation only increases the number of parameters and FLOPs by 1 T times.

Original: Suppose the size of the input feature map is H × W × C and the parameters of the ﬁrst pointwise convolutional layer are expressed as T × C × C × 1 × 1 .
Revised: Suppose the size of the input feature map is H × W × C and the parameters of the ﬁrst pointwise convolutional layer are expressed as T × C × C × 1 × 1 .

Original: We only need additional C × C × 1 ×parameters to reserve the input feature map, since RM operation is only to reserve the input feature map.
Revised: We only need additional C × C × 1 ×parameters to reserve the input feature map, since RM operation is only to reserve the input feature map.

Original: ‘Group-wise convolution’ is also parameter-friendly.
Revised: ‘Group-wise convolution’ is also parameter-friendly.

Original: For example, for a normal convolution, we need C × K × K parameters to reserve the information of an input channel.
Revised: For example, for a normal convolution, we need C × K × K parameters to reserve the information of an input channel.

Original: While for group-wise convolution, we only need 1 G times of parameters to reserve the information where G is the group number in group-wise convolution.
Revised: While for group-wise convolution, we only need 1 G times of parameters to reserve the information where G is the group number in group-wise convolution.

Original: Besides, the cost of storage and calculation does not change whether or not to use group-wise convolution.
Revised: Besides, the cost of storage and calculation does not change whether or not to use group-wise convolution.

Original: Based on the advantage of IRB, we design a series of RMNet models.
Revised: Based on the advantage of IRB, we design a series of RMNet models.

Original: We ﬁrst determine the input and output width by the classic width setting of [64, 128, 256, 512].
Revised: We ﬁrst determine the input and output width by the classic width setting of [64, 128, 256, 512].

Original: We replace 7 × 7 Conv and MaxPooling on the head with two sequences of 3 × 3 Conv, BN, and ReLU (the same approach used in RepVGG).
Revised: We replace 7 × 7 Conv and MaxPooling on the head with two sequences of 3 × 3 Conv, BN, and ReLU (the same approach used in RepVGG).

Original: The tail of RMNet is the same as
Revised: The tail of RMNet is the same as

Original: ResNet.
Revised: ResNet.

Original: For RMNet 50x6 32, ‘50’ indicates the number of all the Conv layers; ‘6’ indicates the multiple of classic width setting, for example, the width of the grouped convolution layer in RMNet 50x6 32 is [6 × 64, 6 × 128, 6 × 256, 6 × 512]; ‘32’ indicates the number of channels of each group in
Revised: For RMNet 50x6 32, ‘50’ indicates the number of all the Conv layers; ‘6’ indicates the multiple of classic width setting, for example, the width of the grouped convolution layer in RMNet 50x6 32 is [6 × 64, 6 × 128, 6 × 256, 6 × 512]; ‘32’ indicates the number of channels of each group in

Original: RMBlock.
Revised: RMBlock.

Original: We print out the detailed architecture of RMNet in the Appendix A.4.
Revised: We print out the detailed architecture of RMNet in the Appendix A.4.

Original: Next, we compare
Revised: Next, we compare

Original: RMNet with SOTA models on CIFAR10, CIFAR100 and ImageNet to test the speed and accuracy.
Revised: RMNet with SOTA models on CIFAR10, CIFAR100 and ImageNet to test the speed and accuracy.

Original: For a fair comparison, we train RMNet following the ofﬁcial implementations of RepVGG 1 .
Revised: For a fair comparison, we train RMNet following the ofﬁcial implementations of RepVGG 1 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: ImageNet
Revised: ImageNet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: From Table 1, the accuracy of RMNet 50 × 6 32 is 2.3% higher than ResNet 101 and 0.59% higher than RepVGG B2 with the nearly same speed.
Revised: From Table 1, the accuracy of RMNet 50 × 6 32 is 2.3% higher than ResNet 101 and 0.59% higher than RepVGG B2 with the nearly same speed.

Original: It is worth noting that RMNet 101 × 6 16 reaches over 80% top-1 accuracy without using any tricks, which is the ﬁrst time for a plain model, to the best of our knowledge.
Revised: It is worth noting that RMNet 101 × 6 16 reaches over 80% top-1 accuracy without using any tricks, which is the ﬁrst time for a plain model, to the best of our knowledge.

Original: Also, increasing the depth to 152 still has a beneﬁt to RMNet, which shows the great potential of our method.
Revised: Also, increasing the depth to 152 still has a beneﬁt to RMNet, which shows the great potential of our method.

Original: 4. F RIENDLY FOR LIGHT - WEIGHT MODELS
Revised: 4. F RIENDLY FOR LIGHT - WEIGHT MODELS

Original: We conduct an experiment to verify our analysis in Section 3.3.
Revised: We conduct an experiment to verify our analysis in Section 3.3.

Original: We ﬁrst train a MobileNetV2 from scratch and convert it to RMNetV1 following the process introduced in Section 3.3.
Revised: We ﬁrst train a MobileNetV2 from scratch and convert it to RMNetV1 following the process introduced in Section 3.3.

Original: Note that the initial architecture of MobileNetV2 has to be designed to make sure the transformed RMNetV1 has the same architecture (depth and width) of MobileNetV1 from the original paper (Howard et al., 2017).
Revised: Note that the initial architecture of MobileNetV2 has to be designed to make sure the transformed RMNetV1 has the same architecture (depth and width) of MobileNetV1 from the original paper (Howard et al., 2017).

Original: Thus the MobileNetV2 in Figure 3 is different from vanilla MobileNetV2(Sandler et al.,
Revised: Thus the MobileNetV2 in Figure 3 is different from vanilla MobileNetV2(Sandler et al.,

Original: We also train a MobileNetV1 trained from scratch for comparison.
Revised: We also train a MobileNetV1 trained from scratch for comparison.

Original: The result are show in
Revised: The result are show in

Original: Table 2.
Revised: Table 2.

Original: [Table]
Revised: [Table]

Original: From the experiment in Table
Revised: From the experiment in Table

Original: [Table]
Revised: [Table]

Original: we can see the speed of RMNetV1 (equivalently converted from
Revised: we can see the speed of RMNetV1 (equivalently converted from

Original: MobileNetV2) is faster than vanilla MobileNetV2 and the performance of RMNetV1 is higher than
Revised: MobileNetV2) is faster than vanilla MobileNetV2 and the performance of RMNetV1 is higher than

Original: MobileNetV1, which shows RM operation can be very friendly for light-weight models
Revised: MobileNetV1, which shows RM operation can be very friendly for light-weight models

Original: [Table]
Revised: [Table]

Original: 4. F RIENDLY FOR P RUNING
Revised: 4. F RIENDLY FOR P RUNING

Original: We conduct an experiment in Figure 6 to verify the effectiveness of pruning RMNet.
Revised: We conduct an experiment in Figure 6 to verify the effectiveness of pruning RMNet.

Original: We use Lnorm multiply a certain sparsity factor to punish the weight of BN layers, and regard the channels whose weight is smaller than a certain threshold as invalid.
Revised: We use Lnorm multiply a certain sparsity factor to punish the weight of BN layers, and regard the channels whose weight is smaller than a certain threshold as invalid.

Original: The sparsity factor is selected from 1eto 1e-
Revised: The sparsity factor is selected from 1eto 1e-

Original: [Table]
Revised: [Table]

Original: and the threshold is selected from 5e-4 to 5e-3.
Revised: and the threshold is selected from 5e-4 to 5e-3.

Original: A larger degree of sparsity will lead to a larger pruning ratio.
Revised: A larger degree of sparsity will lead to a larger pruning ratio.

Original: [Figure]
Revised: [Figure]

Original: From the experiments, RMNet retains higher accuracy than ResNet under a larger pruning ratio and when the speed of pruned RMNet is nearly the same with pruned ResNet, because of more reasonable structure.
Revised: From the experiments, RMNet retains higher accuracy than ResNet under a larger pruning ratio and when the speed of pruned RMNet is nearly the same with pruned ResNet, because of more reasonable structure.

Original: Thus RMNet has a better accuracy-speed tradeoff over ResNet architecture on pruning tasks.
Revised: Thus RMNet has a better accuracy-speed tradeoff over ResNet architecture on pruning tasks.

Original:  C ONCLUSION A ND D ISCUSSION
Revised:  C ONCLUSION A ND D ISCUSSION

Original: In this paper, We propose RM operation to remove residual connections and perform RM operation to convert ResNet and MobileNetV2 into plain models (RMNet).
Revised: In this paper, We propose RM operation to remove residual connections and perform RM operation to convert ResNet and MobileNetV2 into plain models (RMNet).

Original: RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connection without changing original output.
Revised: RM operation allows input feature maps to pass through the block while reserving their information and merges all the information at the end of each block, which can remove residual connection without changing original output.

Original: Experiments have shown that RMNet can achieve a better speed-accuracy trade-off and is very friendly for network pruning.
Revised: Experiments have shown that RMNet can achieve a better speed-accuracy trade-off and is very friendly for network pruning.

Original: There are some future directions to be considered: 1) Analyzing the residual connection on the
Revised: There are some future directions to be considered: 1) Analyzing the residual connection on the

Original: Transformer based architectures and removing such connections via RM operation.
Revised: Transformer based architectures and removing such connections via RM operation.

Original: 2) Searching residual networks with less residual connection with NAS and converting them to efﬁcient RMNets.
Revised: 2) Searching residual networks with less residual connection with NAS and converting them to efﬁcient RMNets.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: The appendix is arranged as follows: in Section A.1, we perform ablation studies; in Section A.2, we further analyze why RepVGG does not have high performace for deeper network; in Section A.3, we show ﬁne-tuning RMNet can achieve higher performance; in Section A.4, we show the detailed architecture of RMNet.
Revised: The appendix is arranged as follows: in Section A.1, we perform ablation studies; in Section A.2, we further analyze why RepVGG does not have high performace for deeper network; in Section A.3, we show ﬁne-tuning RMNet can achieve higher performance; in Section A.4, we show the detailed architecture of RMNet.

Original: A. A BLATION S TUDY
Revised: A. A BLATION S TUDY

Original: In this section, we perform an ablation study to show how batch-size and hyper-parameters of network structure affect inference time for RM operation.
Revised: In this section, we perform an ablation study to show how batch-size and hyper-parameters of network structure affect inference time for RM operation.

Original: The results are shown in Figure 7.
Revised: The results are shown in Figure 7.

Original: As shown in the left bottom sub-ﬁgure, the exact inference speed grows linearly with batch size when the batch size is relatively small, indicating the resource are sufﬁcient.
Revised: As shown in the left bottom sub-ﬁgure, the exact inference speed grows linearly with batch size when the batch size is relatively small, indicating the resource are sufﬁcient.

Original: From the experiments, we can ﬁnd that under the condition of sufﬁcient resources, i.e., lower batch-size and lower complexity of network structure, RMNet has a great advantage over ResNet.
Revised: From the experiments, we can ﬁnd that under the condition of sufﬁcient resources, i.e., lower batch-size and lower complexity of network structure, RMNet has a great advantage over ResNet.

Original: This is because RMNet is a plain model without residual connections.
Revised: This is because RMNet is a plain model without residual connections.

Original: The degree of parallelism can be greatly increased when the resources are sufﬁcient.
Revised: The degree of parallelism can be greatly increased when the resources are sufﬁcient.

Original: Thus we believe RMNet has great potential in the future.
Revised: Thus we believe RMNet has great potential in the future.

Original: [Figure]
Revised: [Figure]

Original: A. C OMPARISON OF R EP VGG AND R ES N ET
Revised: A. C OMPARISON OF R EP VGG AND R ES N ET

Original: [Figure]
Revised: [Figure]

Original: We depict the basic blocks used in ResNet and RepVGG in Figure 8.
Revised: We depict the basic blocks used in ResNet and RepVGG in Figure 8.

Original: In ResBlock (Figure 8(a)), two
Revised: In ResBlock (Figure 8(a)), two

Original: ReLUs are inside and outside of the residual connection, respectively.
Revised: ReLUs are inside and outside of the residual connection, respectively.

Original: While in RepBlock (Figure 8(b)), since re-parameterization is based on the commutative law of multiplication, two ReLUs have to be both outside of the residual connection.
Revised: While in RepBlock (Figure 8(b)), since re-parameterization is based on the commutative law of multiplication, two ReLUs have to be both outside of the residual connection.

Original: One can build RepVGG from a basic ResNet architecture by replacing each ResBlock with two RepBlocks.
Revised: One can build RepVGG from a basic ResNet architecture by replacing each ResBlock with two RepBlocks.

Original: Next, We analyze why RepVGG can not be trained very deep like ResNet from both forward and backward passes:
Revised: Next, We analyze why RepVGG can not be trained very deep like ResNet from both forward and backward passes:

Original: • forward path: (Veit et al.,
Revised: • forward path: (Veit et al.,

Original: assumes that the success of ResNet can be contributed to the
Revised: assumes that the success of ResNet can be contributed to the

Original: ”model ensemble”.
Revised: ”model ensemble”.

Original: we can regard ResNet as an ensemble of many paths of different lengths.
Revised: we can regard ResNet as an ensemble of many paths of different lengths.

Original: Thus, n -blocks’ ResNet have O (2 n ) implicit paths connecting input and output.
Revised: Thus, n -blocks’ ResNet have O (2 n ) implicit paths connecting input and output.

Original: However, unlike ResNet that two branches in the block are separable and can not be merged, multi-branches in RepVGG can be represented by one branch, which can be shown as:
Revised: However, unlike ResNet that two branches in the block are separable and can not be merged, multi-branches in RepVGG can be represented by one branch, which can be shown as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where Conv (cid:48) is the merged convolution of each Conv in the branches.
Revised: ) where Conv (cid:48) is the merged convolution of each Conv in the branches.

Original: Thus RepVGG does not have the implicit ”ensemble assumption” of ResNet, and the representation gap between RepVGG and ResNet increases as the number of blocks raises.
Revised: Thus RepVGG does not have the implicit ”ensemble assumption” of ResNet, and the representation gap between RepVGG and ResNet increases as the number of blocks raises.

Original: • backward path: (Balduzzi et al., 2017) analyzes ”shattered gradients problem” in deep neural networks.
Revised: • backward path: (Balduzzi et al., 2017) analyzes ”shattered gradients problem” in deep neural networks.

Original: The correlation of gradients behaves like ‘White Gaussian Noise’ when there are more ReLUs in the backward path.
Revised: The correlation of gradients behaves like ‘White Gaussian Noise’ when there are more ReLUs in the backward path.

Original: Suppose ResNet and RepVGG both have n layers.
Revised: Suppose ResNet and RepVGG both have n layers.

Original: From Figure 8(a), Information in ResNet can pass through the residual without going through inside ReLU in each block.
Revised: From Figure 8(a), Information in ResNet can pass through the residual without going through inside ReLU in each block.

Original: However, each ReLU in RepVGG is located in the main path.
Revised: However, each ReLU in RepVGG is located in the main path.

Original: Thus the number of ReLUs for ResNet
Revised: Thus the number of ReLUs for ResNet

Original: in the backward path is n 2 , while the number of ReLUs for RepVGG in the path is n which suggests that the gradients in ResNet are far more resistant to shattering when the depth is large, leading to better performance than RepVGG.
Revised: in the backward path is n 2 , while the number of ReLUs for RepVGG in the path is n which suggests that the gradients in ResNet are far more resistant to shattering when the depth is large, leading to better performance than RepVGG.

Original: Table 3 shows the performance of RepVGG and ResNet on ImageNet dataset.
Revised: Table 3 shows the performance of RepVGG and ResNet on ImageNet dataset.

Original: The setting of this experiment is followed by the ofﬁcial implementation of RepVGG.
Revised: The setting of this experiment is followed by the ofﬁcial implementation of RepVGG.

Original: We can see under the same network structure, ResNet-18 is 0.5% higher than RepVGG-18, and ResNet-34 is 0.8% higher than
Revised: We can see under the same network structure, ResNet-18 is 0.5% higher than RepVGG-18, and ResNet-34 is 0.8% higher than

Original: RepVGG in terms of top-1 accuracy.
Revised: RepVGG in terms of top-1 accuracy.

Original: Thus, RepVGG increases the speed at the cost of losing representation ability.
Revised: Thus, RepVGG increases the speed at the cost of losing representation ability.

Original: [Table]
Revised: [Table]

Original: Compared to other approaches (Li et al., 2020; Ding et al., 2021b; Zagoruyko & Komodakis, 2017) that remove residual connections, RM operation can equivalently convert ResBlock to a plain module without re-training.
Revised: Compared to other approaches (Li et al., 2020; Ding et al., 2021b; Zagoruyko & Komodakis, 2017) that remove residual connections, RM operation can equivalently convert ResBlock to a plain module without re-training.

Original: However, if we ﬁne-tune the pre-trained network, the performance may slightly improve since RM operation brings additional parameters into the network.
Revised: However, if we ﬁne-tune the pre-trained network, the performance may slightly improve since RM operation brings additional parameters into the network.

Original: It is worth noting that we need to carefully design the BN layer of additional channels (brought by RM operation) during ﬁne-tuning since the running mean and running var in BN may lead to an unstable state of
Revised: It is worth noting that we need to carefully design the BN layer of additional channels (brought by RM operation) during ﬁne-tuning since the running mean and running var in BN may lead to an unstable state of

Original: DNN.
Revised: DNN.

Original: Thus We statistics the mean and variance by inputting the training dataset through the network, which will serve as the initial values of running mean and running var of BN layers for better stableness.
Revised: Thus We statistics the mean and variance by inputting the training dataset through the network, which will serve as the initial values of running mean and running var of BN layers for better stableness.

Original: In addition, the weight and bias in BN are set by Equation 2 to guarantee the equality of the model.
Revised: In addition, the weight and bias in BN are set by Equation 2 to guarantee the equality of the model.

Original: Of course, there exists another solution that we do not add BN layers when ﬁne-tuning, i.e., fusing the BN with convolution before ﬁne-tuning.
Revised: Of course, there exists another solution that we do not add BN layers when ﬁne-tuning, i.e., fusing the BN with convolution before ﬁne-tuning.

Original: However, it will lead to worse performance.
Revised: However, it will lead to worse performance.

Original: Figure 9 shows the accuracy of ResNet 18/34 on CIFAR-10/100 with/without fusing the BN layer when training.
Revised: Figure 9 shows the accuracy of ResNet 18/34 on CIFAR-10/100 with/without fusing the BN layer when training.

Original: We can see the network without fusing BN has better performance and the accuracy is better than the pre-trained model
Revised: We can see the network without fusing BN has better performance and the accuracy is better than the pre-trained model

Original: [Figure]
Revised: [Figure]

Original: A. D ETAILED S TRUCTURE OF RMN ET
Revised: A. D ETAILED S TRUCTURE OF RMN ET

Original: We print out the detailed network structure of RMNet 26 × 3 32 in Table 4.
Revised: We print out the detailed network structure of RMNet 26 × 3 32 in Table 4.

Original: From Table 4, the most components in RMNet are ‘Conv’ and ‘ReLU’, making it very efﬁcient on inference.
Revised: From Table 4, the most components in RMNet are ‘Conv’ and ‘ReLU’, making it very efﬁcient on inference.

Original: [Table]
Revised: [Table]

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: True)
Revised: True)

Original: Conv2d
Revised: Conv2d

Original: 128, 384, kernel size=(1, 1)
Revised: 128, 384, kernel size=(1, 1)

Original: stride=(1,
Revised: stride=(1,

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: True
Revised: True

Original: Conv2d
Revised: Conv2d

Original: 384, 384, kernel
Revised: 384, 384, kernel

Original: size
Revised: size

Original: stride
Revised: stride

Original: padding
Revised: padding

Original: =(
Revised: =(

Original: groups
Revised: groups

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: size
Revised: size

Original: =(1, 1)
Revised: =(1, 1)

Original: stride=(1,
Revised: stride=(1,

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: True
Revised: True

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: size=
Revised: size=

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: size=
Revised: size=

Original: stride
Revised: stride

Original: =
Revised: =

Original: padding
Revised: padding

Original: =
Revised: =

Original: groups
Revised: groups

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: True)
Revised: True)

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: size=(
Revised: size=(

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =True) (32):
Revised: =True) (32):

Original: Conv2d
Revised: Conv2d

Original: , kernel size=(
Revised: , kernel size=(

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: 1)) (33): ReLU(
Revised: 1)) (33): ReLU(

Original: inplace
Revised: inplace

Original: =True) (34):
Revised: =True) (34):

Original: Conv2d
Revised: Conv2d

Original: (768, 768, kernel size=(
Revised: (768, 768, kernel size=(

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: padding
Revised: padding

Original: =(
Revised: =(

Original: groups
Revised: groups

Original: =
Revised: =

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =True) (36):
Revised: =True) (36):

Original: Conv2d
Revised: Conv2d

Original: , 256, kernel size=(
Revised: , 256, kernel size=(

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace

Original: =
Revised: =

Original: True
Revised: True

Original: Conv2d
Revised: Conv2d

Original: kernel
Revised: kernel

Original: size=(
Revised: size=(

Original: stride
Revised: stride

Original: =(
Revised: =(

Original: ReLU
Revised: ReLU

Original: inplace
Revised: inplace