L EARNING D EEPLY S HARED F ILTER B ASES FOR E FFICIENT C ONV N ETS
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, inspired by repetitive block structure of modern ConvNets, such as
ResNets, parameter-sharing among repetitive convolution layers has been proposed to reduce the size of parameters.
However, naive sharing of convolution ﬁlters poses many challenges such as overﬁtting and vanishing/exploding gradients, resulting in worse performance than non-shared counterpart models.
Furthermore, sharing parameters often increases computational complexity due to additional operations for re-parameterization.
In this work, we propose an efﬁcient parametersharing structure and an effective training mechanism for recursive ConvNets.
In the proposed ConvNet architecture, convolution layers are decomposed into a ﬁlter basis, that can be shared recursively, and non-shared layer-speciﬁc parts.
We conjecture that a shared ﬁlter basis combined with a small amount of layerspeciﬁc parameters can retain, or further enhance, the representation power of individual layers, if a proper training method is applied.
We show both theoretically and empirically that potential vanishing/exploding gradients problems can be mitigated by enforcing orthogonality to the shared ﬁlter bases.
Experimental results demonstrate that our scheme effectively reduces redundancy by saving up to 63.8% of parameters while consistently outperforming non-shared counterpart networks even when a ﬁlter basis is shared by up to 10 repetitive convolution layers.
I NTRODUCTION
Modern networks such as ResNets usually have massive identical convolution blocks and recent analytic studies (Jastrzebski et al., 2018) show that these blocks perform similar iterative reﬁnement rather than learning new features.
Inspired by these massive identical block structure of modern networks, recursive ConvNets sharing weights across iterative blocks have been studied as a promising direction to parameter-efﬁcient ConvNets (Jastrzebski et al., 2018; Guo et al., 2019; Savarese &
Maire, 2019).
However, repetitive use of parameters across many convolution layers incurs several challenges that limit the performance of such recursive networks.
First of all, deep sharing of parameters might result in vanishing gradients and exploding gradients problems, which are often found in recurrent neural networks (RNNs) (Pascanu et al., 2013; Jastrzebski et al., 2018).
Another challenge is that overall representation power of the networks might be limited by using same ﬁlters repeatedly for many convolution layers.
To address aforementioned challenges, in this paper, we propose an effective and efﬁcient parametersharing mechanism for modern ConvNets having many repetitive convolution blocks.
In our work, convolution ﬁlters are decomposed into a fundamental and reusable unit, which is called a ﬁlter basis , and a layer-speciﬁc part, which is called coefﬁcients .
By sharing a ﬁlter basis, not whole convolution ﬁlters or a layer, we can impose two desirable properties on the shared parameters: (1) resilience against vanishing/exploding gradients, and (2) representational expressiveness of individual layers sharing parameters.
We ﬁrst show theoretically that a shared ﬁlter basis can cause vanishing gradients and exploding gradients problems, and this problem can be controlled to a large extent by making ﬁlter bases orthogonal.
To enforce the orthogonality of ﬁlter bases, we propose an orthogonality regularization to train ConvNets having deeply shared ﬁlter bases.
Our experimental results show that the proposed orthogonality regularization reduces the redundancy not just in deeply shared ﬁlter bases, but also in none-shared parameters, resulting in better performance than over-parameterized counterpart networks.
Next, we make convolution layers with shared parameters more expressive using a hybrid approach to sharing ﬁlter bases, in which a small number of layer-speciﬁc non-shared ﬁlter basis components are combined with shared ﬁlter basis components.
With this hybrid scheme, the constructed ﬁlters can be positioned in different vector subspaces that reﬂect the peculiarity of individual convolution layers.
We argue that these layer-speciﬁc variations contribute to increasing the representation power of the networks when a large portion of parameters is shared.
Since our focus is not on pushing the state-of-the-art performance, we show the validity of our work using widely-used ResNets as a base model on image classiﬁcation tasks with CIFAR and ImageNet datasets.
Our experimental results demonstrate that when each ﬁlter basis is shared by up toconvolution layers, our method consistently outperforms counterpart ConvNet models while reducing a signiﬁcant amount of parameters and computational costs.
For example, our method can save up to 63.8% of parameters and 33.4% of FLOPs, respectively, while achieving lower test errors than much deeper counterpart models.
Our parameter sharing structure and training mechanism can be applied to modern compact networks, such as MobileNets (Howard et al., 2017) and ShufﬂeNets (Zhang et al., 2018) with minor adaptations.
Since these compact models already have decomposed convolution blocks, some parts of each block can be identiﬁed as a shareable ﬁlter basis and the rest of the layer-speciﬁc parts.
In Experiments, we demonstrate that compact MobileNetV2 can achieve further 8-21% parameter savings with our scheme while retaining, or improving, the performance of the original models.
R ELATED W ORK
Recursive networks and parameter sharing: Recurrent neural networks (RNNs) (Graves et al., 2013) have been well-studied for temporal and sequential data.
As a generalization of RNNs, recursive variants of ConvNets are used extensively for visual tasks (Socher et al., 2011; Liang & Hu, 2015; Xingjian et al., 2015; Kim et al., 2016; Zamir et al., 2017).
For instance, Eigen et al.
(2014) explore recursive convolutional architectures that share ﬁlters across multiple convolution layers.
They show that recurrence with deeper layers tends to increase performance.
However, their recursive architecture shows worse performance than independent convolution layers due to overﬁtting.
In most previous works, ﬁlters themselves are shared across layers.
In contrast, we propose to share ﬁlter bases that are more fundamental and reusable building blocks to construct layer-speciﬁc ﬁlters.
More recently, Jastrzebski et al.
(2018) show that iterative reﬁnement of features in ResNets suggests that deep networks can potentially leverage intensive parameter sharing.
Guo et al.
(2019) introduce a gate unit to determine whether to jump out of the recursive loop of convolution blocks to save computational resources.
These works show that training recursive networks with naively shared blocks leads to bad performance due to the problem of gradient explosion and vanish like RNN (Pascanu et al., 2013; Vorontsov et al., 2017).
In order to mitigate the problem of gradient explosion and vanish, they suggest unshared batch normalization strategy.
In our work, we propose an orthogonality regularization of shared ﬁlter bases to further address this problem.
Savarese & Maire (2019)’s work is also relevant to our work.
In their work, the parameters of recurrent layers of ConvNets are generated by a linear combination of 1-2 parameter tensors from a global bank of templates.
Though similar to our work, our work suggests more ﬁne-grained ﬁlter bases as more desirable building blocks for effective parameter sharing since ﬁlter bases can be easily combined with layer-speciﬁc non-shared components for better representation power.
Our result shows that these layer-speciﬁc non-shared components are critical to achieve high performance.
Although they achieve about 60% parameter savings, their approach does not outperform counterpart models and incurs slight increases in computational costs due to the overheads in reparameterizing tensors from the templates.
Model compression and efﬁcient convolution block design: Reducing storage and inference time of ConvNets has been an important research topic for both resource constrained mobile/embedded systems and energy-hungry data centers.
A number of research techniques have been developed such as ﬁlter pruning (LeCun et al., 1990; Polyak & Wolf, 2015;
Li et al., 2017; He et al., 2017), low-rank factorization (Denton et al., 2014;
Jaderberg et al., 2014), quantization (Han et al., 2016), and knowledge distillation (Hinton et al., 2015; Chen et al., 2017), to name a few.
These compression techniques have been suggested as post-processing steps that are applied after initial training.
Unfortunately, their accuracy is usually bounded by the approximated original models.
By contrast, our models are trained from scratch as in Ioannou et al. (2017)’s work and our result shows that
[Figure]
parameter-sharing approaches can outperform the counterpart models, if a proper training method is combined, while achieving signiﬁcant savings in parameters.
Some compact networks such as ShufﬂeNet (Zhang et al., 2018) and MobileNet (Howard et al., 2017; Sandler et al., 2018) show that delicately designed internal structure of convolution blocks acquire better ability with lower computational complexity.
Our work can be applied to these compact networks since they already exploit decomposed convolutions for their efﬁcient convolution blocks.
For instance, since MobileNet’s convolution blocks have pointwise-depthwise-pointwise convolution steps, the ﬁrst two convolution steps can be considered a reusable ﬁlter basis and the last step can be considered layer-speciﬁc coefﬁcients.
In Experiments, we show that MobileNetV2 combined with our parameter-sharing scheme outperforms the original models while saving about 8-21% parameters.
D EEP R ECURSIVE S HARING OF A F ILTER B ASIS
In this section, we discuss how to decompose typical convolution layers into more recursive units, or ﬁlter bases, and remaining layer-speciﬁc parts.
We also discuss how to train ConvNets effectively when ﬁlter bases are deeply shared by repetitive convolution layers.
3. F ILTER BASES OF CONVOLUTION LAYERS
We assume that a convolution layer with S input channels, T output channels, and a set of ﬁlters
W = { W t ∈ R k × k × S , t ∈ [1 ..T ] } .
Each ﬁlter W t can be decomposed using a lower rank ﬁlter basis
W basis and coefﬁcients α :
[Equation]
where W basis =
{ W rbasis ∈ R k × k × S , r ∈ [1 ..R ] } is a ﬁlter basis, and α = { α rt ∈ R , r ∈ [1 ..R ] , t ∈ [1 ..T ] } is scalar coefﬁcients.
In Equation 1, R is the rank of the basis.
In a typical convolution layer, output feature maps V t ∈ R w × h × T , t ∈ [1 ..T ] are obtained by the convolution between input feature maps U ∈ R w × h × S and ﬁlters W t , t ∈ [1 ..T ] .
With Equation 1, this convolution can be rewritten as follows:
[Equation]
In Equation 3, the order of the convolution operation and the linear combination of ﬁlter basis is reordered according to the linearity of convolution operators.
This result shows that a standard convolution layer can be replaced with two successive convolution layers as shown in Figure 1-(b).
The ﬁrst decomposed convolution layer performs R convolutions between W r basis , r ∈ [1 ..R ] and input feature maps U , and it generates an intermediate feature map basis V basis ∈ R w × h × R .
The second decomposed convolution layer performs point-wise 1 × 1 convolutions that linearly combine R intermediate feature maps V basis to generate output feature maps V .
The computational complexity of the original convolution is O ( whk 2 ST ) while the decomposed operations take O ( wh ( k 2 SR + RT )) .
As far as R < T , the decomposed convolution has lower computational complexity than the original convolution.
Due to this computational efﬁciency, many compact networks such as MobileNets and ShufﬂeNets also have similar block structures of decomposed convolution layers.
For instance,
MobileNets have repetitive convolution blocks of pointwise-depthwise-pointwise convolutions.
The ﬁlters in the ﬁrst two steps can be considered a reusable ﬁlter basis and the remaining 1x1 ﬁlters in the last step can be considered layer-speciﬁc coefﬁcients.
3. R ECURSIVE SHARING OF A FILTER BASIS
In typical ConvNets, convolution layers have different ﬁlters W s and, hence, each decomposed convolution layer has its own ﬁlter basis W basis and coefﬁcients α .
In contrast, our primary goal in decomposing convolution layers is to share a single ﬁlter basis (or a small number of ﬁlter bases) across many recursive convolution layers.
Unlike some previous works (Jastrzebski et al., 2018;
Köpüklü et al., 2019), in which convolution ﬁlters W themselves are shared recursively, we argue that a ﬁlter basis W basis is a more intrinsic and reusable building block that can be shared effectively since a ﬁlter basis constitutes a subspace, in which high dimensional ﬁlters across many convolution layers can be approximated.
Though components of a basis only need to be independent and span a vector subspace, some speciﬁc bases are more convenient and appropriate for speciﬁc purposes.
For the purpose of sharing a ﬁlter basis, we need to ﬁnd an optimal ﬁlter basis W basis that can expedite the training of ﬁlters of shared convolution layers.
Although this optimization can be done with a typical stochastic gradient descent (SGD), one problem is that exploding/vanishing gradients problems might prevent efﬁcient search of the optimization space.
More formally, we consider a series of N decomposed convolution layers, in which a ﬁlter basis W basis is shared N times.
Let x i be the input of the i -th convolution layer, and a i +1 be the output of the convolution of x i with the ﬁlter basis W basis
[Equation]
In Equation 4, W basis ∈ R k 2 S × R is a reshaped ﬁlter basis that has basis components at its columns.
We assume that input x is properly adapted (e.g., with im2col) to express convolutions using a matrix-matrix multiplication.
Since W basis is shared across N recusrive convolution layers, the gradient of W basis for some loss function L is:
[Equation]
, where
[Equation]
If we plug W basis ∂ x j ∂a j in Equation 6 into Equation 5, we can see that (cid:81) ∂a
j +1 ∂a
j is the term that makes gradients unstable since W basis is multiplied many times.
This exploding/vanishing gradients can be controlled to a large extent by keeping W basis close to orthogonal (Vorontsov et al., 2017).
For instance, if W basis admits eigendecomposition, [ W basis ] N can be rewritten as follows:
[Equation]
where Λ is a diagonal matrix with the eigenvalues placed on the diagonal and Q is a matrix composed of the corresponding eigenvectors.
If W basis is orthogonal, [ W basis ] N neither explodes nor vanishes, since all the eigenvalues of an orthogonal matrix have absolute value 1.
Similarly, an orthogonal shared basis ensures that forward signals neither explodes nor vanishes.
We also need to ensure that the norm of ∂ x j ∂a j in Equation 5 is bounded (Pascanu et al., 2013) for stability during forward and backward passes.
It is shown that batch normalization after non-linear activation at each convolution layer ensures healthy norms (Ioffe & Szegedy, 2015;
Guo et al., 2019; Jastrzebski et al., 2018).
For training networks, the orthogonality of shared bases can be enforced with an orthogonality regularizer .
For instance, when each residual block group of a ResNet shares a ﬁlter basis for its convolution layers, the objective function L R can be deﬁned to have an orthogonality regularizer in addition to the original loss L :
[Equation]
where W ( g ) basis is a shared ﬁlter basis for g -th residual block group and λ is a hyperparameter.
3. A HYBRID APPROACH TO SHARING A FILTER BASIS
In our ﬁlter basis sharing approach, ﬁlters of many convolution layers are constructed by the linear combination of a shared ﬁlter basis as in Equation 1.
This implies that those high-dimensional ﬁlters are all in the same low-dimensional subspace.
If the rank of a ﬁlter basis is too low, it is very challenging to ﬁnd such subspace that can express individual peculiarity of many layers’ ﬁlters.
Conversely, if the rank of a shared ﬁlter basis is too high (e.g., R ≥ T ), the gain in computational complexity from decomposing ﬁlters is mitigated.
One way to increase the representational power of each convolution layer, while still maintaining its computational complexity low, is adding a small number of layer-speciﬁc components to the ﬁlter basis.
For instance, we build a ﬁlter basis W basis not only using shared components, but also using non-shared components:
[Equation]
where W bs _ shared =
{ W rbs _ shared ∈ R k × k × S , r ∈ [1 ..n ] } are shared ﬁlter basis components, and W bs _ unique = { W rbs _ unique ∈ R k × k × S , r ∈ [ n +1 ..R ] } are per-layer non-shared ﬁlter basis components.
With this hybrid scheme, ﬁlters in different convolution layers are placed in different layer-speciﬁc subspace.
One disadvantage of this hybrid scheme is that non-shared ﬁlter basis components require more parameters.
The ratio of non-shared basis components can be varied to control the tradeoffs.
But, our results in Section 4 show that only a few per-layer non-shared basis components is enough to achieve high performance.
E XPERIMENTS
In this section, we perform a series of experiments on image classiﬁcation tasks.
Using ResNets (He et al., 2016) as base models, we train networks with the proposed method and compare them with the baseline networks.
We also analyze the effect of the orthogonality regularization and the hybrid scheme.
4. R ESULTS ON CIFAR 4.1. M ODEL CONFIGURATION AND TRAINING DETAILS
Throughout the experiments, we use ResNets (He et al., 2016) as base networks by replacing their 3 × 3 convolution layers to decomposed convolution layers sharing ﬁlter bases.
Since each residual block group of ResNets have different number of channels and kernel sizes, our networks share a ﬁlter basis only in the same group (Figure 5 in Appendix).
In each group with n residual blocks, the ﬁrst block has a different stride, and, hence, it does not share a ﬁlter basis.
Each residual block of the baseline ResNets has two 3 × 3 convolution layers, and, hence, our networks’ each group has 2( n − 1) decomposed convolution layers sharing a ﬁlter basis.
Throughout the experiments, we denote by ResNet L -S
s U u a ResNet with L layers that has a ﬁlter basis with s shared components and u layer-speciﬁc non-shared components in the ﬁrst residual block group.
This ratio between s and u is maintained for all residual block groups.
The CIFAR-10/100 datasets contains 50,000 and 10,000 three-channel 32 × 32 images for training and testing, respectively.
For training networks, we follow a similar training scheme in He et al.
Standardized data-augmentation and normalization are applied to input data.
Networks are trained for 300 epochs with SGD optimizer with a weight decay of 5e-4 and a momentum of 0.9.
The learning rate is initialized to 0.1 and is decayed by 0.1 at 50% and 75% of the epochs.
4.1. R ESULTS
Table 1 shows the results on CIFAR-100.
Networks trained with the proposed method consistently outperform their ResNet counterparts.
For instance, ResNet34-S32U1 requires only 36.2% parameters and 66.6% FLOPs of the counterpart ResNet34 while achieving even lower test error (21.79%) than much deeper ResNet50 (22.36%).
To show the generality of our work, we apply the proposed method to DenseNet (Huang et al., 2017), ResNeXt (Xie et al., 2017), and MobileNetV2 (Sandler et al., 2018).
Although the overall gain is not as great as ResNets’, we still observe reduction of resource usages in these networks.
For instance, ResNeXt50-S64U4 outperforms the counterpart ResNeXt while saving parameters and FLOPs by 16.7% and 12.1%, respectively.
In ResNeXt, the gain is limited since they mainly exploit group convolutions; each group convolution is decomposed for ﬁlter basis sharing in our network.
Similarly, for DenseNet, each 3 × 3 convolution layer has a relatively small number of output channels, and, hence the overall gain is not pronounced as much as ResNets’.
Unlike the other networks, MobileNetV2 already has a factorized block structure (pointwise-depthwise-pointwise), and, hence, we choose to share the pointwise-depthwise layers across successive convolution blocks with the same number of channels (Figure 7 in Appendix).
During the training, the proposed orthogonality regularization is applied to these shared ﬁlters separately.
Though MobileNetV2 does not have as many repeating blocks as ResNets, we observe that our MobileNetV2-Shared saves about 21.9% of parameters while outperforming the original MobileNetV2.
[Table] [Table]
The result on CIFAR-10 is presented in Table 2.
Unlike networks on CIFAR-100, networks on
CIFAR-10 has much fewer channels (e.g. 16 channels in the ﬁrst residual block group) and, hence, projecting ﬁlters to such low dimensional subspace might limit the performance of the networks.
For instance, in ResNet32-S8U1, ﬁlters are supposed to be projected onto 9 dimensional subspace consisting of 8 shared and 1 layer-speciﬁc ﬁlter basis components.
By increasing the rank of ﬁlter bases, the better accuracy can be achieved at the cost of increased FLOPs.
For deeper networks such as ResNet56, a ﬁlter basis is supposed to be shared by many residual blocks in the group, and it can damage the performance.
For example, every ﬁlter basis in ResNet56-S16U1 is shared by 8 residual blocks, or 16 convolution layers.
Due to this excessive sharing, though ResNet56-S16U1 saves 41.3% parameters, its testing error (7.46%) is higher than the counterpart ResNet56’s (6.97%).
To remedy this problem, we introduce a variant, in which each residual block group of the networks uses 2 shared bases; one basis is shared by the ﬁrst convolution layers of all residual blocks, and the other is shared by the second convolution layers of the same blocks.
In Table 2, networks with a ‘ (cid:63) ’ mark denote this variant.
Though this variant slightly increases the parameters of the networks, it can prevent excessive sharing of parameters.
For example, although ResNet56-S16U1 (cid:63) needs 0.04M more parameters for additional shared bases, it still saves 63% parameters of the counterpart
ResNet56 and achieves lower testing error, 6.30%.
In Table 2, we compare our results with similar state-of-the-art techniques. Our method achieves better performance and parameter-saving than other approaches such as ﬁlter pruning (Li et al., 2017), kernel clustering (Li et al., 2019), and recursive sharing (Guo et al., 2019).
4.1. A NALYSIS : E FFECTS OF O RTHOGONAL B ASES
[Figure]
To inverstigate the effect of orthogonality regularization during training, we track the ﬂows of gradients while training ResNet34-S16U1. Figure 2 shows the maximum and mean absolute gradients in the four shared bases during the 20th epoch.
Jastrzebski et al. (2018) and Guo et al.
(2019) showed that unshared batch normalization (BN) mitigates vanishing/exploding gradients problems, and our result in Figure 2-(b) shows that unshared BN following shared bases improves the ﬂow of gradients. When the proposed orthogonality regularization is applied to the shared bases, similar effect on gradient ﬂows is observed in Figure 2-(c).
When both unshared BN and orthogonality regularization are applied together, in Figure 2-(d), further stronger, but still bounded, ﬂow of gradients are observed.
This trend is consistently observed during the training. We conjecture that this healthy ﬂows of gradients improve the optimization process during training.
To further analyze the effect of the orthogonality regularization, in Figure 3, we illustrate absolute cosine similarities of all ﬁlter basis components and coefﬁcients of the 2nd and the 3rd residual block groups of ResNet34-S16U1.
In the upper low, the X and Y axes display the indexes to the shared basis components ﬁrst, and all non-shared basis components in respective groups next.
As expected, the shared ﬁlter basis components have almost zero cosine similarities when the orthogonality regularization in Equation 8 is applied.
The bottom low shows the absolute cosine similarities of coefﬁcients of the corresponding groups.
In Figure 3, we can clearly see that coefﬁcients manifest lower similarities when the orthogonality regularization is applied.
Without the orthogonality regularization, interesting grid patterns are observed in coefﬁcients.
This repetitive grid pattern might be related to ResNets’ nature of iterative reﬁnement (Jastrzebski et al., 2018).
However, since bases and coefﬁcients are used to build ﬁlters of recursive layers, such high cosine similarity is directly related to the higher redundancy in the networks.
When the orthogonality regularization is applied, such repetitive patterns are less evident, implying that recursive layers perform less repetitive tasks. [Figure] [Figure] =
= = = =
=
=
=
=
= =
[Figure]
4.1. A NALYSIS : E FFECTS OF R ANKS OF S HARED /U NSHARED B ASES
Figure 4 shows test errors as parameters and FLOPs are increased by varying the number of shared/non-shared basis components of networks.
In general, the higher performance is expected with the more parameters.
We observe that this presumption is true for shared basis components.
For instance, when the number of shared basis components s is varied from 8 to 32, the test error sharply decreases from 23.1% to 21.7%.
However, non-shared basis components manifest counter-intuitive results.
Although a small number of non-shared basis components (e.g., u = 1 ) are clearly beneﬁcial to the performance, the higher u ’s do not always lead to the higher performance.
For instance, when u = 4 , both ResNet34-S16U u and ResNet34-S32U u show the worst performance.
This result demonstrates the difﬁculty of training networks with larger parameters.
Further study is required for this problem.
4. R ESULTS ON I MAGE N ET
We evaluate our method on the ILSVRC2012 dataset (Russakovsky et al., 2015) that hasclasses.
The dataset consists of 1.28M training and 50K validation images.
We use ResNet34/ and MobileNetV2 as base models.
Since the block structure of ResNet50 and MobileNetV2 already resembles the decomposed convolution blocks, our models derived from ResNet50 and MobileNetV share the ﬁrst 1x1 and 3x3 convolution layers recursively without ﬁlter decomposition for repeating convolution blocks (Figure 6 and 7 in Appendix).
We train the ResNet-derived models for 150 epochs with SGD optimizer with a mini-batch size of 512, a weight decay of 1e-4, and a momentum of 0.9.
The learning rate starts with 0.1 and decays by 0.1 at 60-th, 100-th, and 140-th epochs.
MobileNetV and our MobileNetV2-Shared models are trained for 300 epochs with a weight decay of 1e-5.
Its learning rate starts with 0.1 and decays by 0.1 at 150-th, 225-th, and 285-th epochs.
[Table]
The results in Table 3 show that ResNet34-S48U1 (cid:63) outperforms the counterpart ResNet34 while using only 54.0% parameters of the counterpart.
Although ResNet34-S48U1 (cid:63) requires lower FLOPs than the counterpart ResNet34, it takes 14% longer latency on Jetson TX2. This overhead mostly comes from performing convolution operations separately for shared and non-shared basis components. Since our ResNet50- and MobileNetV2-derived models do not decompose convolution blocks to deﬁne shared bases, overall parameter-saving is not as pronounced as in ResNet34. However, they still save about 28.6% and 14.9% parameters, respectively, while achieving comparable performance.
For example [Table] MobileNetV2-Shared † even outperforms the counterpart model by 0.51%. This result shows that even compact models such as MobileNetV2 can beneﬁt from the proposed parameter-sharing mechanism using orthogonality regularization.
Since our ResNet50- and MobileNetV2-derived models do not perform separate convolution for shared and non-shared basis, their theoretical FLOPs are equal to the counterpart models’.
However, the latency of our parameter-efﬁcient models is slightly lower than the counterpart models on actual devices.
This is because our parameter-efﬁcient models better make use of the limited cache and memory of the embedded device.
C ONCLUSIONS AND F UTURE W ORKS
In this work, we propose to share ﬁlter bases of decomposed convolution layers for efﬁcient and effective sharing of parameters in ConvNets.
We both theoretically and empirically show that gradient explosion/vanishing problem of deeply shared ﬁlter bases can be effectively addressed by the proposed orthogonality regularization of the ﬁlter bases.
With our approach, while a signiﬁcant amount of parameters is shared, the representation power of each convolution layer is retained, or further enhanced, by reducing the redundancy effectively.
Experimental results show that our approach consistently outperforms the counterpart models while signiﬁcantly reducing parameters.
We believe that the proposed parameter-sharing and training method suggests important architectural possibilities for future neural architecture search (NAS) for resource-efﬁcient networks.