A VOIDING DEGRADATION IN DEEP FEED - FORWARD NETWORKS BY PHASING OUT SKIP - CONNECTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
A widely observed phenomenon in deep learning is the degradation problem: increasing the depth of a network leads to a decrease in performance on both test and training data.
Novel architectures such as ResNets and Highway networks have addressed this issue by introducing various ﬂavors of skip-connections or gating mechanisms.
However, the degradation problem persists in the context of plain feed-forward networks.
In this work we propose a simple method to address this issue.
The proposed method poses the learning of weights in deep networks as a constrained optimization problem where the presence of skip-connections is penalized by Lagrange multipliers.
This allows for skip-connections to be introduced during the early stages of training and subsequently phased out in a principled manner.
We demonstrate the beneﬁts of such an approach with experiments on MNIST, fashion-MNIST, CIFAR-10 and CIFAR-100 where the proposed method is shown to greatly decrease the degradation effect and is often competitive with ResNets.
I NTRODUCTION
The representation view of deep learning suggests that neural networks learn an increasingly abstract representation of input data in a hierarchical fashion (Zeiler & Fergus, 2014; Goodfellow et al., 2016; Greff et al., 2016).
Such representations may then be exploited to perform various tasks such as image classiﬁcation, machine translation and speech recognition.
A natural conclusion of the representation view is that deeper networks will learn more detailed and abstract representations as a result of their increased capacity.
However, in the case of feed-forward networks it has been observed that performance deteriorates beyond a certain depth, even when the network is applied to training data.
Recently, Residual Networks (ResNets; He et al. 2016a) and Highway Networks (Srivastava et al., 2015) have demonstrated that introducing various ﬂavors of skip-connections or gating mechanisms makes it possible to train increasingly deep networks.
However, the aforementioned degradation problem persists in the case of plain deep networks (i.e., networks without skip-connections of some form).
A widely held hypothesis explaining the success of ResNets is that the introduction of skipconnections serves to improve the conditioning of the optimization manifold as well as the statistical properties of gradients employed during training.
Raiko et al.
(2012) and Schraudolph (2012) show that the introduction of specially designed skip-connections serves to diagonalize the Fisher information matrix, thereby bringing standard gradient steps closer to the natural gradient.
More recently,
Balduzzi et al.
(2017) demonstrated that the introduction of skip-connections helps retain the correlation structure across gradients.
This is contrary to the gradients of deep feed-forward networks, which resemble white noise.
More generally, the skip-connections are thought to reduce the effects of vanishing gradients by introducing a linear term (He et al., 2016b).
The goal of this work is to address the degradation issue in plain feed-forward networks by leveraging some of the desirable optimization properties of ResNets.
We approach the task of learning parameters for a deep network under the framework of constrained optimization.
This strategy allows us to introduce skip-connections penalized by Lagrange multipliers into the architecture of our network.
In our setting, skip-connections play an important role during the initial training of the network and are subsequently removed in a principled manner.
Throughout a series of experiments we demonstrate that such an approach leads to improvements in generalization error when compared to architectures without skip-connections and is competitive with ResNets in some cases.
The contributions of this work are as follows:
• We propose alternative training strategy for plain feed-forward networks which reduces the degradation in performance as the depth of the network increases.
The proposed method introduces skip-connections which are penalized by Lagrange multipliers.
This allows for the presence of skip-connections to be iteratively phased out during training in a principled manner.
The proposed method is thereby able to enjoy the optimization beneﬁts associated with skip-connections during the early stages of training.
• A number of benchmark datasets are used to demonstrate the empirical capabilities of the proposed method.
In particular, the proposed method greatly reduces the degradation effect compared to plain networks and is on several occasions competitive with ResNets.
R ELATED WORK
The hierarchical nature of many feed-forward networks is loosely inspired by the structure of the visual cortex where neurons in early layers capture simple features (e.g., edges) which are subsequently aggregated in deeper layers (Hubel & Wiesel, 1962).
This interpretation of neural networks suggests that the depth of a network should be maximized, thereby allowing the network to learn more abstract (and hopefully useful) representations (Bengio et al., 2013).
However, a widely reported phenomenon is that deeper networks are more difﬁcult to train.
This is often termed the degradation effect in deep networks (Srivastava et al., 2015; He et al., 2016a).
This effect has been partially attributed to optimization challenges such as vanishing and shattered gradients (Hochreiter et al., 2001; Balduzzi et al., 2017).
In the past these challenges have been partially addressed via the use of supervised and unsupervised pre-training (Bengio et al., 2009) and more recently through careful parameter initialization (Glorot & Bengio, 2010; He et al., 2015) and batch normalization (Ioffe & Szegedy, 2015).
In the past couple of years further improvements have been obtained via the introduction of skip-connections.
ResNets (He et al., 2016a;b) introduce residual blocks consisting of a residual function F together with a skip-connection.
Formally, the residual block is deﬁned as:
[Equation]
where F l : R n → R n (cid:48) represents some combination of afﬁne transformation, non-linearity and batch normalization parameterized by W l .
The matrix W (cid:48)
l parameterizes a linear projection to ensure the dimensions are aligned 1 .
More generally, ResNets are closely related to Highway Networks (Srivastava et al., 2015) where the output of each layer is deﬁned as:
[Equation]
where · denotes element-wise multiplication.
In Highway Networks the output of each layer is determined by a gating function
[Equation]
inspired from LSTMs.
We note that both ResNets and Highway Networks were introduced with the explicit goal of training deeper networks.
Inspired by the success of the these methods, many variations have been proposed.
Huang et al.
(2016a)
propose DenseNet, where skip-connections are passed from all previous activations.
Huang et al.
(2016b) propose to shorten networks during training by randomly dropping entire layers, leading to better gradient ﬂow and information propagation, while using the full network at test time.
Recently, the goal of learning deep networks without skip-connections has begun to receive more attention.
Zagoruyko & Komodakis (2017) propose a novel re-parameterization of weights in feedforward networks which they call the Dirac parameterization.
Instead of explicitly adding a skipconnection, they model the weights as a residual of the Dirac function, effectively moving the skipconnection inside the non-linearity.
In related work, Balduzzi et al.
(2017) propose to initialize weights in a CReLU activation function in order to preserve linearity during the initial phases of training.
This is achieved by initializing the weights in a mirrored block structure.
During training the weights are allowed to diverge, resulting in non-linear activations.
Finally, we note that while the aforementioned approaches have sought to train deeper networks via modiﬁcations to the network architecture (i.e., by adding skip-connections) success has also been obtained by modifying the non-linearities (Clevert et al., 2015; Klambauer et al., 2017).
V ARIABLE A CTIVATION N ETWORKS
The goal of this work is to train deep feed-forward networks without suffering from the degradation problem described in previous sections.
To set notation, we denote x 0 as the input and x L as the output of a feed-forward network with L layers.
Given training data { y , x 0 } it is possible to learn parameters { W l } Ll =1 by locally minimizing some objective function
[Equation]
First-order methods are typically employed due to the complexity of the objective function in equation (3).
However, directly minimizing the objective is not practical in the context of deep networks: beyond a certain depth performance quickly deteriorates on both test and training data.
Such a phenomenon does not occur in the presence of skip-connections.
Accordingly, we take inspiration from
ResNets and propose to modify equation (1) in the following manner 2 :
[Equation]
where α l ∈ [0 , 1] n determines the weighting given to the skip-connection.
More speciﬁcally, α l is a vector were the entry i dictates the presence and magnitude of a skip-connection for neuron i in layer l .
Due to the variable nature of parameters α l in equation (4), we refer to networks employing such residual blocks as Variable Activation Networks (VAN).
The objective of the proposed method is to train a feed-forward network under the constraint that α l = 1 for all layers, l .
When the constraint is satisﬁed all skip-connections are removed.
The advantage of such a strategy is that we only require α l = 1 at the end of training.
This allows us to initialize α l to some other value, thereby relaxing the optimization problem and obtaining the advantages associated with ResNets during the early stages of training.
In particular, whenever α l (cid:54) = 1 information is allowed to ﬂow through the skip-connections, alleviating issues associated with shattered and vanishing gradients.
As a result of the equality constraint on α l , the proposed activation function effectively does not introduce any additional parameters.
All remaining weights can be trained by solving the following constrained optimization problem:
[Equation]
The associated Lagrangian takes the following simple form (Boyd & Vandenberghe, 2004):
[Equation]
where each λ l ∈ R n are the Lagrange multipliers associated with the constraints on α l .
In practice, we iteratively update α l via stochastic gradients descent (SGD) steps of the form:
[Equation]
where η is the step-size parameter for SGD.
Throughout the experiments we will often take the non-linearity in F l to be ReLU.
Although not strictly required, we clip the values α l to ensure they remain in the interval [0 , 1] n .
From equation (6), we have that the gradients with respect to Lagrange multipliers are of the form:
[Equation]
We note that since we require α l ∈ [0 , 1] n , the values of λ l are monotonically decreasing.
As the value of Lagrange multiplier decreases, this in turn pushes α l towards 1 in equation (7).
We set the step-size for the Lagrange multipliers, η (cid:48) , to be a fraction of η .
The motivation behind such a choice is to allow the network to adjust as we enforce the constraint on α l .
E XPERIMENTS
The purpose of the experiments presented in this section is to demonstrate that the proposed method serves to effectively alleviate the degradation problem in deep networks.
We ﬁrst demonstrate the capabilities of the proposed method using a simple, non-convolutional architecture on the MNIST and Fashion-MNIST datasets (Xiao et al., 2017) in Section 4.1.
More extensive comparisons are then considered on the CIFAR datasets (Krizhevsky & Hinton, 2009) in Section 4.2.
4. MNIST AND F ASHION -MNIST
Networks of varying depths were trained on both MNIST and Fashion-MNIST datasets.
Following
Srivastava et al.
the networks employed in this section were thin , with each layer containing 50 hidden units.
In all networks the ﬁrst layer was a fully connected plain layer followed by l layers or residual blocks (depending on the architecture) and a ﬁnal softmax layer.
The proposed method is benchmarked against several popular architectures such as ResNets and Highway Networks as well as the recently proposed DiracNets (Zagoruyko & Komodakis, 2017).
Plain networks without skipconnections are also considered.
Finally, we also considered VAN network where the constraint α l = 1 was not enforced.
This corresponds to the case where λ l = 0 for all l .
This comparison is included in order to study the capacity and ﬂexibility of VAN networks without the need to satisfy the constraint to remove skip-connections.
For clarity, refer to such networks as VAN ( λ = 0 ) networks.
For all architectures the ReLU activation function was employed together with batch-normalization.
In the case of ResNets and VAN, the residual function consisted of batch-normalization followed by
ReLU and a linear projection.
The depth of the network varied from l = 1 to l = 30 hidden layers.
All networks were trained using
SGD with momentum.
The learning rate is ﬁxed at η = 0 .
001 and the momentum parameter at 0.9.
Training consisted of 50 epochs with a batch-size of 128.
In the case of VAN networks the α l values were initialized to 0 for all layers.
As such, during the initial stages of training VAN networks where equivalent to ResNets.
The step-size parameter for Lagrange multipliers, η (cid:48) , was set to be one half of the SGD step-size, η .
Finally, all Lagrange multipliers, λ l , are initialized to -1.
R ESULTS
The results are shown in Figure 1 where the test accuracy is shown as a function of the network depth for both the MNIST and Fashion-MNIST datasets.
In both cases we see clear evidence of the degradation effect: the performance of plain networks deteriorates signiﬁcantly once the network depth exceeds some critical value (approximately 10 layers).
As would be expected, this is not the case for ResNets, Highway Networks and DiracNets as such architectures have been explicitly designed to avoid this behavior.
We note that VAN networks do not suffer such a pronounced degradation as the depth increases.
This provides evidence that the gradual removal of skip-connections via Lagrange multipliers leads to improved generalization performance compared to plain networks.
Finally, we note that VAN networks obtain competitive results across all depths.
Crucially, we note that VAN networks outperform plain networks across all depths, suggesting that the introduction of variable skip-connections may lead to convergence at local optima with better generalization performance.
Finally, we note that VAN ( λ = 0 ) networks, where no constraint is placed on skip-connections, obtain competitive results across all depths.
4. CIFAR
As a more challenging benchmark we consider the CIFAR-10 and CIFAR-100 datasets.
These consist of 60000 32 × 32 pixel color images with 10 and 100 classes respectively.
The datasets are divided into 50000 training images and 10000 test images.
We follow He et al.
(2016a)
and train deep convolutional networks consisting of four blocks each consisting of n residual layers.
The residual function is of the form conv-BN-ReLU-conv-BN-ReLU .
This corresponds to the pre-activation function (He et al., 2016b).
The convolutional layers consist of 3 × 3 ﬁlters with downsampling at the beginning of blocks 2, 3 and 4.
The network ends with a fully connected softmax layer, resulting in a depth of 8 n + 2 .
The architecture is described in Table 1.
Networks were trained using SGD with momentum over 165 epochs.
The learning rate was set to η = 0 . 1 and divided by 10 at the 82nd and 125th epoch.
The momentum parameter was set to 0.9.
Networks were trained using mini-batches of size 128.
Data augmentation followed Lee et al.
(2015): this involved random cropping and horizontal ﬂips.
Weights were initialized following He et al.
As in Section 4.1, we initialize α l = 0 for all layers.
Furthermore, we set the step-size parameter for the Lagrange multipliers, η (cid:48) , to be one tenth of η and all Lagrange multipliers, λ l , are initialized to -1.
On CIFAR-10 we ran experiments with n ∈ { 1 , 2 , 3 , 4 , 5 , 6 , 8 , 10 } yielding networks with depths ranging from 10 to 82.
For CIFAR-100 experiments were run with
[Table]
Results for experiments on CIFAR-10 are shown in Figure 2.
The left panel shows the mean test accuracy over ﬁve independent training sessions for ResNets, VAN, VAN ( λ = 0 ) and plain networks.
While plain networks provide competitive results for networks with fewer than 30 layers, their performance quickly deteriorates thereafter.
We note that a similar phenomenon is observed in
VAN networks but the effect is not as dramatic.
In particular, the performance of VANs is similar to ResNets for networks with up to 40 layers.
Beyond this depth, ResNets outperform VAN by an increasing margin.
This holds true for both VAN and VAN ( λ = 0 ) networks, however, the difference is reduced in magnitude in the case of VAN ( λ = 0 ) networks.
These results are in line with
He et al.
(2016b), who argue that scalar modulated skip-connections (as is the case in VANs where the scalar is 1 − α l ) will either vanish or explode in very deep networks whenever the scalar is not the identity.
The right panel of Figure 2 shows the training and test error for a 26 layer network.
We note that throughout all iterations, both the test and train accuracy of the VAN network dominates that of the plain network.
The thick gold line indicates the mean residuals of the α l parameters across all layers.
This is deﬁned as 1 L (cid:80) Ll =1 ( 1 − α l ) 2 and is a measure of the extent to which skip-connections are present in the network.
Recall that if all α l values are set to one then all skip-connections are removed (see equation (4)).
From Figure 2, it follows that skip-connections are fully removed from the VAN network at approximately the 120 th iteration.
More detailed traces of Lagrange multipliers and α l are provided in Appendix B.
A comparison of the performance of VAN networks in provided in Table 2.
We note that while
VAN networks do not outperform ResNets, they do outperform other alternatives such as Highway networks and FitNets (Romero et al., 2014) when networks of similar depths considered.
However, it is important to note that both Highway networks and FitNets did not employ batch-normalization, which is a strong regularizer.
In the case of both VAN and VAN ( λ = 0 ) networks, the best performance is obtained with networks of 26 layers while ResNets continue to improve their performance as depth increases.
Finally, current state-of-the-art performance, obtained by Wide ResNets (Zagoruyko & Komodakis, 2016) and DenseNet Huang et al.
(2016a), are also provided in Table
Figure 3 provides results on the CIFAR-100 dataset.
This dataset is considerably more challenging as it consists of a larger number of classes as well as fewer examples per class.
As in the case of
CIFAR-10, we observe a fall in the performance of both VAN and plain networks beyond a certain depth; in this case approximately 20 layers for plain networks and 30 layers for VANs.
Despite this drop in performance, Table 2 indicates that the performance of VAN networks with both 18 and 26 layers are competitive with many alternatives proposed in the literature.
Furthermore, we note that the performance of VAN ( λ = 0 ) networks is competitive with ResNets in the context of the
CIFAR-100 dataset.
Training curves are shown on the right hand side of Figure 3.
As in the equivalent plot for CIFAR-10, the introduction and subsequent removal of skip-connections during training leads to improvements in generalization error.
[Table]
D ISCUSSION
This manuscript presents a simple method for training deep feed-forward networks which greatly reduces the degradation problem.
In the past, the degradation issue has been successfully addressed via the introduction of skip-connections.
As such, the goal of this work is to propose a new training regime which retains the optimization beneﬁts associated with ResNets while ultimately phasing out skip-connections.
This is achieved by posing network training as a constrained optimization problem where skip-connections are introduced during the early stages of training and subsequently phased out in a principled manner using Lagrange multipliers.
Throughout a series of experiments we demonstrate that the performance of VAN networks is stable, displaying a far smaller drop in performance as depth increases and thereby largely mitigating the degradation problem.
A VAN RE - FORMULATION
The original formulation for the VAN residual block was as follows:
[Equation]
We thank an anonymous reviewer for suggesting that such a formulation may be detrimental to the performance of very deep VAN networks.
The reason for this is that scaling constant within each block is always less than one, implying that the contributions of lower layers vanish exponentially as the depth increases.
This argument is also provided in He et al.
(2016b) who perform similar experiments with ResNets.
In order to validate this hypothesis, we compare the performance of VAN networks employing the residual block described in equation (4) and the residual block described in equation (9).
The results, shown in Figure 4, provide evidence in favor of the proposed hypothesis.
While both formulations for VAN networks obtain similar performances for shallow networks, as the depth of the network increases there is a more pronounced drop in the performance of VAN networks which employ residual blocks described in equation (9).
In a further experiment, we also studied the performance of ResNets with the following residual block:
[Equation]
The results in Figure 4 demonstrate that ResNets which employ the residual blocks deﬁned in equation (10) show a clear deterioration in performance as the depth of the network increases.
Such a degradation in performance is not present when standard ResNets are employed.
B L AGRANGE MULTIPLIER TRACES
In this section we provide addition ﬁgures demonstrating the evolution of Lagrange multipliers, λ l throughout training.
We note that the updates to Lagrange multipliers are directly modulated by the current value of each α l (see equation (8)).
As such, we also visualize the mean residuals of the α l parameters across all layers.
This is deﬁned as 1 L (cid:80) Ll =1 ( 1 − α l ) 2 and is a measure of the extent to which skip-connections are present in the network.
Once all skip-connections have been removed, this residual will be zero and the values of Lagrange multipliers will no longer change.
This is precisely what we ﬁnd in Figure 5.
The left panel plots the mean value of Lagrange multipliers across all layers, while the right panel shows the mean residual of α l .
We observe that for networks of different depths, once the constraint to remove skip-connections is satisﬁed, the value of Lagrange multipliers remains constant.
This occurs at different times; sooner for more shallow networks whilst later on for deeper networks.