N ETWORK I TERATIVE L EARNING FOR D YNAMIC D EEP N EURAL N ETWORKS VIA M ORPHISM
Anonymous authors
Paper under double-blind review
A BSTRACT
In this research, we present a novel learning scheme called network iterative learning for deep neural networks.
Different from traditional optimization algorithms that usually optimize directly on a static objective function, we propose in this work to optimize a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized.
The optimization is implemented as a series of intermediate neural net functions that is able to dynamically grow into the targeted neural net objective function.
This is done via network morphism so that the network knowledge is fully preserved with each network growth.
Experimental results demonstrate that the proposed network iterative learning scheme is able to signiﬁcantly alleviate the degradation problem.
Its effectiveness is veriﬁed on diverse benchmark datasets.
I NTRODUCTION
Deep convolutional neural networks have recently demonstrated their continuous excellent performances on diverse computer vision problems, including image classiﬁcation (Lang et al., 1990;
Simonyan & Zisserman, 2014; Szegedy et al., 2014; He et al., 2015a), object detection (Girshick et al., 2014;
Ren et al., 2015), and semantic segmentation (Long et al., 2015).
Deep neural network algorithms involve many optimization problems.
One of the most important is network training.
It is quite common to invest days to months of time on hundreds of machines in order to solve even a single instance of the neural network learning problem (Goodfellow et al., 2016).
The optimization of deep neural networks is much more difﬁcult than traditional optimization problems.
One of the most obvious difﬁculties is that the problem of optimizing a deep neural network is ill-conditioned (non-convex) and contains numerous local minima (Goodfellow et al., 2016).
Using the fact that neural network parameters are symmetric, one can show that an m -layer neural network with n units at each layer can impose n !
m ways of hidden units arrangements.
This indicates that there can be an extremely large or even uncountably inﬁnite amount of local minima in a deep neural network optimization problem.
Local minima can be problematic if they have high cost in comparison to the global minimum.
It remains an open question whether there are many local minima of high cost for networks of practical interest (Goodfellow et al., 2016).
It is a common sense that deep neural network optimization algorithms will not converge to a global minimum but instead a local one.
However, many practitioners start to believe that, for sufﬁciently large neural networks, most local minima have a low cost function value (Saxe et al., 2013;
Dauphin et al., 2014; Goodfellow et al., 2014; Choromanska et al., 2015).
In this research, we shall show that in fact there is still a relatively large gap between the local minima and its global minimum for a large deep neural network optimized by traditional optimization algorithms.
In order to further reduce this gap, we shall propose a novel network iterative learning scheme.
The intuition for the proposed network iterative learning scheme is illustrated in Fig.
Let x G be the global minimum of the function f ( x ) , and x L i , i = 1 , 2 , 3 be its local minima.
Assuming that we use an uniform initialization, then a gradient-based optimization algorithm will have a probability of α (0 < α < 1) to converge to the global minimum.
Instead of directly optimizing the function f ( x ) , we ﬁrst use a simpler function f (1) ( x ) to approximate f ( x ) , and let f (2) ( x ) =
f
( x ) .
Using a gradient-based optimization algorithm, regardless of the initialization, the algorithm will converge to the global minimum of f (1) ( x ) at x (1) G .
Suppose that x (1) G falls into the convex subregion of f (2) ( x ) which contains x G , a second round of a gradient-based optimization algorithm will converge to the global minimum of f ( x ) at x G .
In this case, we will have a probability of 1 to converge to the global minimum of f ( x ) using the proposed network iterative learning scheme.
For complex functions such as deep neural networks, function f (1) ( x ) might not be convex and x (1) G might not fall into the convex subregion of f (2) ( x ) containing x G either.
We believe that the proposed network iterative learning scheme will have higher probability to fall into low cost local minima than directly optimizing the objective function f ( x ) .
This belief was veriﬁed by the proposed experiments.
It is worth noting that the problem of local minima with high cost can not be rectiﬁed by gradientbased optimization algorithms, including the backpropagation-based algorithms for training deep neural networks.
Theoretically, a local minimum is guaranteed to be a global minimum if the objective function is convex.
However, for a deep neural network, its objective function is obviously non-convex (Bishop, 2006).
Empirically, when we use different optimization algorithms to train the deep neural networks, they all converge to a similar performance, but can not converge to lower cost local minima comparing against the proposed network iterative learning scheme (Fig. 9).
The proposed network iterative learning scheme for deep neural networks is based on network morphism (Wei et al., 2016).
Network morphism is an effective scheme to morph a well-trained neural network into a new one with the network function preserved.
After morphing a parent network, the child network is expected to inherit the knowledge from its parent network and also has the potential to continue growing into a more powerful one.
Fig.
2(e) illustrates the idea of the proposed network iterative learning scheme for deep neural networks based on network morphism.
As shown, an 8-layer network is ﬁrst learned, and then iteratively grown into 14, 26, 50, and 98 layers to achieve continuously improved performance 1 .
The proposed learning scheme is not only a novel learning scheme, but also contains a novel deep dynamic neural network architecture.
To achieve this, we introduce a uniﬁed network architecture called DynamicNet to enable the dynamic growth of neural networks.
As illustrated in Fig.
3, this architecture follows a modularized architecture design.
For each level, a repeatable template block is used to enable the dynamic change of its depth.
DynamicNet can represent a large family of networks.
In this research, we verify the proposed scheme on two classic architectures, i.e., PlainNet and ResNet (He et al., 2015a).
Since these two networks are not initially proposed with architecture change, certain modiﬁcations need to be made to adapt them to the DynamicNet family.
Furthermore, we also elaborate how each of these two networks can be iteratively evolved by the proposed learning scheme.
The detailed network iteration processes are illustrated in Fig.
In this research, we also developed an algorithm based on optimal gradient descent to solve for the network morphism equation.
The proposed network iterative learning scheme is also capable of learning new knowledge based on the established knowledge base.
This is because it avoids re-learning from scratch when the network is going deeper, but inherits the full knowledge that has already learned.
It is known that when deeper neural networks are trained, a degradation problem may be encountered.
As shown by the blue curve in Fig. 2(c), the error rate becomes saturated and then rapidly increases as the network depth increases.
Such performance degradation is not caused by over-ﬁtting (He et al., 2015a), because the training error of the deeper network is also larger than that of the shallower network (Fig. 2(a)).
The degradation problem can be greatly alleviated by the proposed learning scheme.
Its effectiveness is shown in Fig. 2(b), where the deeper network could achieve lower errors in both training and testing than the shallower one.
Fig. 2(c) and (d) also compare these two learning schemes on PlainNet and
ResNet, in which the performance improvement brought by the proposed scheme is indicated by the purple-pink region.
In the proposed experiments, we shall also demonstrate that the models obtained
[Figure]
by the proposed learning scheme are able to consistently achieve better performances than those learned with a ﬁxed architecture on diverse network architectures
[Figure]
R ELATED W ORK Local Minima of Deep Neural Network
The object function of a deep neural network is non-convex.
Hence, a local minimum can not be guaranteed to be also a global minimum (Bishop, 2006).
There can be an extremely large or even uncountably inﬁnite amount of local minima of a deep neural network optimization problem.
Local minima can be problematic if they have high cost in comparison to the global minimum.
It remains an open question whether there are many local minima of high cost for networks of practical interest (Goodfellow et al., 2016).
Researchers have constructed small neural networks that have local minima with higher cost than the global minimum (Sontag & Sussmann, 1989; Brady et al., 1989; Gori & Tesi, 1992).
In this research, we shall show that the problem of local minima with high cost in optimization is common and arises naturally for deep neural networks.
Optimization Algorithms The main stream optimization algorithms for deep neural networks are gradient-based.
They are mathematically derived from the chain-rule of calculating the derivatives of the composition of two or more functions in Calculus.
Currently widely adopted gradient-based optimization algorithms include stochastic gradient descent (SGD), Nesterov’s accelerated gradient descent (NAG) (Nesterov, 1983; 2013), RMSprop (Tieleman & Hinton, 2012), Adam (Kingma &
Ba, 2014), AdaGrad (Duchi et al., 2011), AdaDelta (Zeiler, 2012), etc.
However, the problem of local minima with high cost can not be rectiﬁed by these traiditonal gradient-based optimization algorithms.
While the proposed network iterative learning scheme is able to further reduce the gap between local minima and the global minimum for deep neural networks.
Knowledge Transferring and Network Morphism A series of work uses a student network to mimic the output of a teacher network (Bucilu et al., 2006; Ba & Caruana, 2014; Romero et al., 2014).
Usually the student network is lighter and the teacher network is more complex.
In such a teacherstudent formula, the student network is usually relearned from scratch, and its network architecture is ﬁxed.
Another series of work (Chen et al., 2015a; Wei et al., 2016) tries to add new layers into the original network architecture, allowing the network architecture to dynamically change.
Since the child network usually directly inherits the full knowledge of the parent network, we refer to this as the parent-child formula.
The proposed network iterative learning is based on the latter series.
Network morphism (Wei et al., 2016) is an effective scheme to morph a well-trained neural network into a new one with the network function preserved.
We choose it as the basic operation of the proposed network iterative learning scheme.
However, this research is different from (Wei et al., 2016) signiﬁcantly.
One essential difference between the proposed research and (Wei et al., 2016) is that network morphism is used as a building block of the proposed network iterative learning scheme.
Another essential difference is that, although only in theory, if we can add a single layer into a neural network, we are able to add an arbitrary number of layers.
In practice, this might not be true.
First, the choice of the morphing target is a great issue.
Second, we need to guarantee that the performance will continuously improve when adding more layers.
Otherwise, it would make no sense to do this.
Surely, there will be a limit for the number of layers we are allowed to add due to the over-ﬁtting problem.
This paper presents a systematic study on how to pick the morphing target and how can we practically successful to morph a network by adding a large amount of layers, in which (Wei et al., 2016) did not address.
T HE N ETWORK I TERATIVE L EARNING A PPROACH 3. N ETWORK I TERATIVE L EARNING
Different from traditional optimization algorithms that usually optimize directly on a static objective function, we propose in this work to optimize a dynamic objective function in an iterative fashion capable of adapting its function form when being optimized.
In this research, we split such process into multiple stages, with the knowledge learned in the previous stage being transferred to the next stage for continual optimization .
Let f be the target function, and θ be the parameters.
The proposed goal is to ﬁnd the minimum
[Equation]
Instead of directly optimizing f , we propose to ﬁnd a series of objective functions { f 1 ( θ 1 ) , f 2 ( θ 2 ) , · · · , f n ( θ n ) } with f i − 1 being able to dynamically grow into f i and θ i − 1 being able to transfer to θ i , satisfying f ( θ ) =
f n ( θ n ) .
Typically, the functional forms of { f i } are of increasing complexity, and the parameter space is also non-decreasing: size ( θ 1 ) ≤ size ( θ 2 ) ≤ · · · ≤ size ( θ n ) .
Then, the proposed scheme can be formulated as
[Equation]
For deep neural networks, such network iterative learning process is achieved by network architecture iteration and weight knowledge transferring.
In this case, the objective function f is a very deep neural network, and the functional series { f i } is the set of intermediate neural networks.
In this research, we design a DynamicNet architecture to allow the network architecture to iteratively evolve and elaborate the application of the proposed learning scheme on two classic networks, i.e., PlainNet and ResNet (He et al., 2015a).
We also adopt a modiﬁed version of network morphism (Wei et al., 2016) to transfer the knowledge from ˆ θ i − 1 to θ i and to facilitate exponential order growth, followed by further optimization to obtain a better-performing ˆ θ i .
3. T HE D YNAMIC N ET A RCHITECTURE
x 𝑛 x 𝑘 𝐷 𝐶 𝑇 𝐿 𝐹
For the proposed learning scheme, the ﬁrst requirement is a network architecture that allows to dynamically change.
Inspired by the observation that modern networks are designed by splitting its body into multiple stages (Simonyan &
Zisserman, 2014; He et al., 2015a), we propose a uniﬁed architecture called DynamicNet to enable network iterative learning.
The proposed architecture follows a modularized architecture design.
In this architecture, we calculate the features at several levels from lower ones to higher ones, and each level is further constructed with a repeatable template block to facilitate the network growing.
As illustrated in Fig.
3, the main body of a DynamicNet is composed of k level blocks L , in which each level block L further consists of a channel switching block C and several template blocks
T .
The channel switching block C is introduced to simplify the morphing by requiring that layers within the same level block shall have the same channel sizes.
This block usually consists of a single convolutional layer 2 .
The template block T actually deﬁnes the basic module of DynamicNet.
Representative network architectures, including PlainNet and ResNet, can be adapted as template blocks in DynamicNet.
In addition to the main body, there is also an optional fast down-sampling block D , which is designed to reduce the computational cost.
This block typically consists of one or several convolutional layers and pooling layers.
There is also a tail F in DynamicNet to represent a classiﬁer connecting to the object categories.
It usually consists of a global pooling layer followed by a fully connected layer and a softmax layer.
The DynamicNet is able to represent a large family of networks with modularized architectures.
In this work, we verify the proposed learning scheme on two representative network architectures, i.e.,
PlainNet and ResNet (He et al., 2015a), as shown in Fig.
The template block T of a PlainNet is simply composed of a single convolutional layer (Fig. 4(a)).
Since the channel switching block has the same architecture as the template block, we shall consider both of them contributing to the number of blocks n .
An expanded version of the network architecture is illustrated in Fig.
For ResNet (He et al., 2015a), the template block T is a two-branch subnet, with the ﬁrst branch consisting of two sequential convolutional layers and the second branch a scaled identity mapping layer (residual ratio r ).
These two branches are joined by addition (Fig. 4(b)).
In order to be compatible with the ResNet proposed in (He et al., 2015a), we make the architecture of the channel switching block C the same as that of the template block T , with the identity mapping layer being replaced with a 1 × 1 convolutional layer.
The channel switching block shall also be considered for contributing to the number of blocks n .
We made certain modiﬁcations to the ResNet to make it ﬁt for the DynamicNet architecture.
In (He et al., 2015a), ResNet is proposed with identity connections, while in this research, we extend them as scaled identity connections controlled by a residual ratio r .
This extension is to avoid ﬁlling the decomposed convolutional ﬁlters with all zeros.
Networks in the DynamicNet family allow for exponentially growing their network depths.
In general, this can be achieved by making a multiplication on n , the number of blocks.
N EURAL N ETWORK
A RCHITECTURE
I TERATION
BN/ReLU
BN/PReLU
BN/ReLU 𝐺 𝐹 𝑙 𝐹 𝑙+ (a) PlainNet
BN ReLU BN ReLU
BNReLU
BN ReLU
BN PReLU
BN ReLU id (b) ResNet
We explain the detailed network iteration processes for PlainNet and ResNet in the DynamicNet family.
We adopt network morphism (Wei et al., 2016) operations as the basic operations in these iteration processes, to guarantee that the network function is unchanged after morphing.
Here we focus on the architecture iteration of networks, leaving detailed morphing algorithms to the next section.
The Iteration of PlainNet
For PlainNet, as shown in Fig. 4(a) and Fig.
5, a convolutional layer G is decomposed into two convolutional layers F l and F l +1 , that satisfy the following network morphism equation:
[Equation]
5) where G, F l , and F l +1 are the convolutional ﬁlters associated with the convolutional layers with shapes of ( C i , C j , K, K ) , ( C i , C l , K l , K l ) , and ( C l , C j , K l +1 , K l +1 ) .
˜ G is a zero-padded version of G whose kernel size is ˜ K
= K l + K l +1 − 1 .
In the proposed morphing operation for PlainNet, the original BatchNorm and ReLU layers are ﬁrst duplicated to the end of the second morphed convolutional layer, and the network function will remain the same.
However, we still have to insert another BatchNorm/ReLU pair after the ﬁrst convolutional layer.
Here, we assume that such operations are possible.
In Section 3.4, we will introduce how to solve the network morphism equation (5) and how to insert a BatchNorm/ReLU pair both with the network function unchanged.
The Iteration of ResNet
The iteration process for ResNet is illustrated in Fig. 4(b).
As shown, a copy of the original template block is ﬁrst made.
For the second template block, we maintain r Id for the shortcut connection, and decompose (1 − r ) Id into two convolutional layers as we did for PlainNet, where r is the residual ratio and Id is the identity mapping.
It is worth noting that we have extended the identity connection in (He et al., 2015a) to a scaled one controlled by a residual ratio r in this research.
The network function shall not change in the proposed iteration process.
As for the BatchNorm and ReLU layers, we simply insert them where they are necessary.
[Figure]
BN
[Figure]
BN/
[Figure]
BN/
[Figure]
BN/
[Figure]
BN/
[Figure]
BN
[Figure]
BN
[Figure]
Input: G of shape ( C i ,C j ,K,K ) ; C l , K l , K l +1 Output: F l of shape ( C i ,C l ,K l ,K l ) , F l +1 of shape ( C l ,C j ,K l
+1 ,K l +1 )
Initialize F l with unit random noise and initialize F l +1 with the initializer described in (He et al., 2015b).
Normalize G with unit standard variance, expand G to ˜ G with kernel size ˜ K
= K l + K l +1 − 1 by padding zeros.
Keep a record of the standard variance for ˜ G .
repeat
Fix F l +1 , and calculate the optimal learning rate for F l as η lopt .
Update F l ← F l − η lopt ∂l∂Fl
Fix F l , and calculate the optimal learning rate for F l +1 as η l +1 opt .
Update F l +1 ← F l +1 − η l +1 opt ∂l ∂Fl +
Calculate the loss l = (cid:107) ˜
G − conv ( F l ,F l +1 ) (cid:107) 2 until l = 0 or maxIter is reached
Multiply F l with the standard variance of ˜ G recorded, and normalize F l and F l +1 with equal standard variances.
procedure C ALC O PTIMAL
L EARNING R ATE ( G , F l , F l +1 , ∂l∂Fl ) for η = 0 ,α, 2 α do
Set ˆ F l ← F l − η ∂l∂Fl , and calculate the loss l = (cid:107) ˜
G − conv ( ˆ F l ,F l +1 ) (cid:107) 2 end for
Let l 0 ,l 1 ,l 2 be the losses calculated in the above loop.
return α 2 −
[Figure]
2 l 1+ l 0 end procedure
3. M ORPHING A LGORITHMS
[Figure]
(a)
[Figure]
(b)
In this section, we shall describe how to decompose one convolutional layer into two convolutional layers that satisfy the network morphism equation (5), and also how to safely insert BatchNorm and ReLU layers into the network so that its network function is unchanged.
The Morphing of Convolutional Layers
In the proposed experiment for PlainNet, the algorithm in (Wei et al., 2016) suffered from a dramatic performance drop for the ﬁrst few iterations.
This may be due to its lack of ability to evenly distribute the convolutional ﬁlter information into the morphed ﬁlters.
Consider the case of decomposing one 1 ×convolutional ﬁlter into two 1 × 1 convolutional ﬁlters, least square based algorithm (Wei et al., 2016) will transfer all information of the original ﬁlter into only one of the morphed ﬁlters.
To solve this problem, we present a novel and more robust algorithm.
Using this algorithm, there was no noticeable performance drop except for the occasions caused by a change of learning rate in the proposed experiments (Fig. 2(e)).
The proposed algorithm is based on modiﬁed gradient descent.
The major difference is that the optimal learning rate for each operand is solved for in the proposed algorithm.
Hence, this algorithm is called optimal Gradient Descent (optGD) algorithm in this work.
This can be achieved by leveraging the fact that the loss function l = l ( η ) is a quadratic function of the learning rate η :
[Equation]
[Equation]
where ˆ F l is the updated version of F l .
Therefore, we can analytically solve for the optimal learning rate η opt with any three sample points on the curve.
The algorithm is illustrated in Algorithm 1 and Fig.
As shown, we iteratively optimize F l or F l + with the other one ﬁxed.
The gradients ∂l∂F l and ∂l ∂F l +1 are calculated by the chain-rule, which is implemented in typical deep learning libraries (Jia et al., 2014; Chen et al., 2015b).
Its computational graph is also illustrated in Fig. 6(a).
For the optimal learning rate, it is calculated by the procedure
C ALC O PTIMIAL L EARNING R ATE in Algorithm 1.
We sample three points in the curve l ( η ) with a step size of α , and the optimal learning rate can be given by
[Equation]
where l 0 , l 1 , l 2 are the losses of l ( η ) when η takes values of 0 , α, 2 α .
Since the minimum of a quadratic function actually does not depend on the three points sampled, we simply set α = 1 .
Fig.
6(b) illustrates an example for the decomposition of a random convolutional ﬁlter with l 0 = 0 .
17836769 , l 1 = 0 .
15257044 , l 2 = 0 .
The optimal learning rate is calculated as η opt = 11 .
12 from Eqn.
Note that in the standard SGD algorithm with learning rate 0.1 and mini-batch size 256, its actual learning rate is 0 . 1 / 256 ≈ 4 .
e − 4 .
11.12 is much larger than 4 .
e −in scale, hence the proposed optGD algorithm converges signiﬁcantly faster than the standard SGD algorithm.
Eqn.
(7) is derived based on the fact that l ( η ) is a quadratic function of η (Eqn.
When this condition does not hold, i.e. l ( η ) is an arbitrary continuous function of η , Eqn.
(7) can still give a second-order approximation of the optimal learning rate.
In the future, we wish to apply the proposed algorithm to general optimization problems.
The Morphing of ReLU and BatchNorm
For ReLU and BatchNorm layers, when they are inserted into a network, the network function shall actually change.
The proposed solution is to reduce them to function as an identity mapping layer, so that they can be freely added to the neural network.
In this research, we adopt the solution proposed in (Wei et al., 2016) for the ReLU layer.
For any nonlinear activation function ϕ , its parametric form P - ϕ is deﬁned to be any continuous function family that is able to connect ϕ and ϕ id , where ϕ id is identity mapping.
Its canonical form is deﬁned as
[Equation]
When ϕ is the ReLU function, P - ϕ will be the PReLU function (He et al., 2015b).
With a = 1 as the initialization, the PReLU layer will become an identity layer.
It will function as a non-linear one once a has been learned.
For BatchNorm (Ioffe & Szegedy, 2015) deﬁned by
[Equation]
we simply initialize its gamma to ones, and beta to zeros.
Mathematically strictly speaking, the mean subtracted by the BatchNorm layer will incur the network function changed unless it is ﬁlled with zeros, yet we are only able to show that the expectation of the mean is zero.
However, this small perturbation will not result in a noticeable alteration to the network function, since the output is further normalized by the BatchNorm layer after the second convolution ( F l +1 in Fig. 4(a)) before passing to the next template block.
In the proposed experiments, no accuracy degradation was observed using this initialization strategy.
E XPERIMENTAL R ESULTS 4. E XPERIMENTAL R ESULTS ON THE CIFAR10 D ATASET
We ﬁrst conduct experiments on the CIFAR10 (Krizhevsky & Hinton, 2009) dataset, a benchmark dataset for image classiﬁcation.
It consists of 50,000 training images and 10,000 testing images inobject categories.
The network inputs are 32 × 32 color images with per-channel mean subtracted.
Since this dataset is composed of tiny images, the fast down-sampling block in DynamicNet is not necessary 3 .
We adopt
[Table] [Table]
a uniform number of levels k = 3 for both networks.
For PlainNet and ResNet, the depths are 3 n +and 6 n + 2 , respectively, where n represents the number of blocks in the level block L .
The main body of the network architecture is computed on channel sizes of { 16,32,64 } and feature map sizes of { 32,16,8 } .
This setup is the same as in (He et al., 2015a).
The neural networks were trained using SGD with a batch size of 256 for 320 epochs.
The learning rate starts from 0.1 and decreases by a factor of 10 at 160/240 epochs.
We use a decay of 0.
and a momentum of 0.9.
The images are padded with 4 pixels on each side for augmentation, and a random crop and a random horizontal ﬂip are applied.
The ﬁrst proposed experiment is on PlainNet.
In the iteration process, the number of blocks n takes values of 1 + 2 i − 1 , where 1 is for channel switching and i represents the i -th iteration from 1 to 6, resulting in the iteratively evolved network depths of 7, 10, 16, 28, 52, and 100.
The experimental results are shown in Table 1(a) and Fig. 2(c).
As the results shown, network iterative learning (named as IterLearn) is able to greatly alleviate the degradation problem.
The absolute and relative performance improvements can be up to 10.2% and 57.4%.
This performance gain is also illustrated by the shadowed purple-pink area in Fig. 2(c).
In Fig. 2(c), we also draw the performances of the best-performing models trained with different learning schemes (dashed lines).
The global performance improvement is indicated by the gap in between the dashed lines.
Besides, Fig. 2(a) and (b) also show the training and testing errors for traditional learning with a ﬁxed architecture (named as FixLearn) and for the proposed learning scheme, where the deeper network (52-layer) could achieve lower error rates in both training and testing than the shallower one (28-layer).
This trend in training error reduction is not quite obvious as the errors are actually less than 0.3% and may not be differentiable.
This trend can be more clearly observed on the CIFAR100 dataset as illustrated in Fig.
Then, we conduct experiments on ResNet.
In the iteration process, the number of blocks n takes values of 2 i , where i represents the i -th iteration from 1 to 5, resulting in the iteratively evolved network depths of 8, 14, 26, 50, and 98.
Fig.
2(e) illustrates this iteration in the proposed learning scheme, where the network performance is continually improved.
Table 1(b) shows the experimental results.
Due to space limitations, we did not show the absolute and relative performance improvements.
For the 98-layer ResNet with residual ratio 0.7, the absolute and relative performance improvements can be up to 0.83% and 12.6%.
ResNets with residual ratios < 1 are actually not good-performing network architectures.
As shown in Table 1(b), when the network depth is increased from 50 to 98, by learning with a ﬁxed architecture, the performance degradation is obvious with residual ratio 0.5 or 0.8.
This phenomenon was also observed in (He et al., 2016)
(Table 1 and Fig. 2(b)).
However, when these networks are learned by the proposed learning scheme, such performance degradation can be avoided.
The comparison results between learning with a ﬁxed architecture and the proposed learning scheme is also illustrated in Fig. 2(d) with residual ratio r = 0 .
5 , in which the signiﬁcant performance gain is illustrated as the purple-pink area.
Combining with the effectiveness on resolving for the degradation problem on
PlainNet, we can conclude that the proposed learning scheme is able to ﬁx the inherent inferiority in these poor-performing network architectures.
Therefore, the effectiveness of network iterative learning has been shown on both representative networks.
We observe consistent performance improvements of network iterative learning over learning with a ﬁxed architecture.
The degradation problem is also greatly alleviated by the proposed network iterative learning.
We believe the reason is that network iterative learning allows full knowledge of learning to be inherited as the initialization when the network is iteratively evolved to a deeper one.
Such an iteration of network is regularized, and shall have lower probability to be trapped into a high cost local minimum than conventional scheme of learning with a ﬁxed architecture.
4. E XPERIMENTAL R ESULTS ON THE CIFAR100 D ATASET
[Figure]
on CIFAR100.
We also verify the proposed learning scheme on the CIFAR100 dataset.
It consists of 50,000 training images and 10,000 testing images in 100 object categories.
We follow the same setup and network architectures for PlainNet and ResNet as in the previous experiments.
The results are shown in
Table 2, from which we can reach the same conclusions as in the CIFAR dataset.
For PlainNet and ResNet, the absolute performance improvements can be up to 18.28%, 1.53% respectively, and the relative performance can be up to 36.7%, 5.1%, respectively, for the networks around 100 layers.
Due to space limitation, the details of the performance improvements are not included.
In addition to the consistent performance improvement, the degradation problem is also signiﬁcantly alleviated by the proposed learning scheme for this dataset.
From Table 2 we can see that, when the networks are extended from around 50 layers to 100 layers, the proposed learning scheme can achieve a consistent performance improvement for all three networks.
Training with a ﬁxed architecture cannot achieve this.
Fig.
7(a) and (b) also compare the training and testing errors on PlainNet between these two learning schemes.
For the proposed learning scheme, the deeper network (52-layer) could achieve lower error rates in both training and testing than the shallower one (28-layer), while learning with a ﬁxed architecture fails to achieve lower error rates.
4. E XPERIMENTAL R ESULTS ON THE I MAGE N ET D ATASET
[Figure]
We further conduct experiments on ImageNet (Russakovsky et al., 2014).
This dataset is composed of 1,000 object categories, with 1. million training images and 50,000 validation images.
For PlainNet, we set the number of levels to 4 and hence its depth is 4 n + 2 .
n represents the number of blocks and takes values of 1 + 2 i − 1 with i represents the i -th iteration from 1 to 5, resulting in the iteratively evolved network depths of 10, 14, 22, 38, 70.
The inputs to the network are 224 × 224 color patches cropped from resized images with shorter sides randomly sampled in [256,480].
The networks were trained with a batch size of 256 for 100 epochs.
The learning
[Table]
[Table]
rate starts from 0.1 and decreases with a factor of 10 every 30 epochs.
We use a decay of 0.0001 and a momentum of 0.9.
The setup is the same as in (He et al., 2015a).
Fig.
8 illustrates the experimental results.
We observe the degradation problem as marked by the blue curve.
However, this problem can be signiﬁcantly alleviated by the proposed learning scheme.
The performance gain brought by network iterative learning is indicated by the purple-pink region.
Due to time and computational resource constraints, we have not yet carried out experiments for ResNet, which will be completed soon.
4. E FFICIENCY AND T RAINING A CCELERATION
The proposed learning scheme contains two parts in each iteration period: network iteration (including knowledge transferring) and continual learning.
Since the time cost of the ﬁrst part takes only from seconds to minutes and could be neglected, the time cost of network iterative learning mainly depends on the second part.
Due to historical reasons, in this research, we adopted a uniform training strategy for the continual learning and learning with a ﬁxed architecture for 320 epochs.
Assume that the training time cost is proportional to the network depth given ﬁxed training steps, one may conclude that the time cost of network iterative learning is about twice the time cost of training the deepest neural network alone.
However, The network iterative learning process is capable of learning new knowledge based on the established knowledge base.
Hence, the training process can be greatly accelerated because it avoids re-learning from scratch when the network is going deeper.
Table 3 illustrates the experimental results of the training acceleration strategy for the proposed network iterative learning scheme.
In this table, “full” means training for 320 epochs and “half” means training for 160 epochs.
As shown, the “half” strategy does not incur any performance drop comparing against the “full” training strategy.
4. C OMPARING A GAINST T RADITIONAL O PTIMIZATION A LGORITHMS
[Figure]
We compare the proposed network iterative learning scheme against traditional optimization algorithms.
Fig.illustrates the experimental results for the 52-layer and 100-layer PlainNets on the CIFAR10 dataset.
As shown, traditional optimization algorithms, including stochastic gradient descent (SGD), Nesterov’s accelerated gradient descent (NAG)
(Nesterov, 1983; 2013) and Adam (Kingma & Ba, 2014), usually converge to similar performances for a neural network.
While the proposed network iterative learning scheme is able to further reduce the gap between the local minima and the global minimum.
This indicates that the problem of local minima with high cost in optimization is common and arises naturally for deep neural networks.
The effectiveness of the proposed network iterative learning scheme is also demonstrated.
We also conduct experiments for traditional optimization algorithms including RMSprop (Tieleman & Hinton, 2012), AdaGrad (Duchi et al., 2011) and AdaDelta (Zeiler, 2012).
However, they converge to a much worse-performing local minima in the proposed experiment case and their curves are not shown.
4. T HE E FFICIENCY OF THE P ROPOSED OPT GD A LGORITHM
[Figure]
Finally, we compare the proposed optGD algorithm with the least square (LSQ) based algorithms proposed in (Wei et al., 2016) for solving the network morphism equation (5).
The advantages of the proposed optGD over LSQ when adopted for network morphism of the proposed network iterative learning scheme are mainly in three folds.
First, in the proposed experiments, the primary morphing operations are to decompose a 3 × 3 convolutional kernel into two 3 × 3 convolutional kernels.
The proposed optGD algorithm is able to do this directly.
While for LSQ in (Wei et al., 2016), one has to decompose a 3 × 3 convolutional kernel into a 3 × 3 convolutional kernel and a 1 × 1 convolutional kernel, and then expand the latter to a 3 ×convolutional kernel in order to achieve the desired morphing.
Second, the proposed optGD algorithm does not incur an additional sharp error increase at the start besides the learning rate change (Fig.
10 blue curve).
While in (Wei et al., 2016), there is an additional sharp error increase at the start besides the learning rate change (Fig.
10 orange curve).
Such sharp error increase happened when we were expanding the 1 × 1 convolutional kernel into a 3 × 3 one.
It should be caused by ﬁlling the convolutional ﬁlters with too many zeros, as the Net2Net approach in (Chen et al., 2015a) also encounters the same problem due to the large amount of zeros in identity layers added into the network (Fig.
5 in (Chen et al., 2015a)).
Third, the proposed optGD algorithm is much more robust than the LSQ-based algorithm in (Wei et al., 2016).
This is probably because the proposed optGD algorithm is able to evenly distribute the convolutional ﬁlter information into the morphed ﬁlters.
While LSQ-based algorithm will likely transfer all information of the original ﬁlter into only one of the morphed ﬁlters.
Table 4 compares the experimental results for PlainNet on the CIFAR10 dataset.
As can be seen, the proposed optGD algorithm is much more effective than the LSQ-based algorithm (7.57% vs 12.55%) solving the network morphism equation for the proposed network iterative learning scheme.
It is also worth noting that, even when combined with the LSQ algorithm, the proposed scheme still outperforms the learning with a ﬁxed architecture scheme by a large margin (12.55% vs 17.77%).
C ONCLUSIONS
In this research, we show that the problem of local minima with high cost in optimization is common and arises naturally for deep neural networks.
In an attempt to further reduce the gap between the local minima and the global minimum, we present a novel learning scheme called network iterative learning for deep neural networks.
Different from traditional optimization algorithms that usually optimize directly on a static objective function, the proposed network iterative learning scheme contains a novel deep dynamic neural network architecture.
Extensive experiments have been carried out to demonstrate the effectiveness of the proposed network iterative learning scheme.