Paper:
REGULARIZING CNNS WITH LOCALLY CONSTRAINED DECORRELATIONS
Regularization is key for deep learning since it allows training more complex models while keeping lower levels of overfitting. However, the most prevalent regularizations do not leverage all the capacity of the models since they rely on reducing the effective number of parameters. Feature decorrelation is an alternative for using the full capacity of the models but the overfitting reduction margins are too narrow given the overhead it introduces. In this paper, we show that regularizing negatively correlated features is an obstacle for effective decorrelation and present OrthoReg, a novel regularization technique that locally enforces feature orthogonality. As a result, imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing the regularizer to reach higher decorrelation bounds, and reducing the overfitting more effectively. In particular, we show that the models regularized with OrthoReg have higher accuracy bounds even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of state-of-the-art CNNs on CIFAR-10, CIFAR-100, and SVHN.
1 INTRODUCTION
Neural networks perform really well in numerous tasks even when initialized randomly and trained with Stochastic Gradient Descent (SGD) (see Krizhevsky et al. (2012)). Deeper models, like Googlenet (Szegedy et al. (2015)) and Deep Residual Networks (Szegedy et al. (2015); He et al. (2015a)) are released each year, providing impressive results and even surpassing human performances in well-known datasets such as the Imagenet (Russakovsky et al. (2015)). This would not have been possible without the help of regularization and initialization techniques which solve the overfitting and convergence problems that are usually caused by data scarcity and the growth of the architectures.
From the literature, two different regularization strategies can be defined. The first ones consist in reducing the complexity of the model by (i) reducing the effective number of parameters with weight decay (Nowlan & Hinton (1992)), and (ii) randomly dropping activations with Dropout (Srivastava et al. (2014)) or dropping weights with DropConnect (Wan et al. (2013)) so as to prevent feature co-adaptation. Due to their nature, although this set of strategies have proved to be very effective, they do not leverage all the capacity of the models they regularize.
The second group of regularizations is those which improve the effectiveness and generality of the trained model without reducing its capacity. In this second group, the most relevant approaches decorrelate the weights or feature maps, e.g. Bengio & Bergstra (2009) introduced a new criterion so as to learn slow decorrelated features while pre-training models. In the same line Bao et al. (2013) presented ”incoherent training”, a regularizer for reducing the decorrelation of the network activations or feature maps in the context of speech recognition. Although regularizations in the second group are promising and have already been used to reduce the overfitting in different tasks, even with the presence of Dropout (as shown by Cogswell et al. (2016)), they are seldom used in the large scale image recognition domain because of the small improvement margins they provide together with the computational overhead they introduce.
Although they are not directly presented as regularizers, there are other strategies to reduce the overfitting such as Batch Normalization (Ioffe & Szegedy (2015)), which decreases the overfitting by reducing the internal covariance shift. In the same line, initialization strategies such as ”Xavier” (Glorot & Bengio (2010)) or ”He” (He et al. (2015b)), also keep the same variance at both input and output of the layers in order to preserve propagated signals in deep neural networks. Orthogonal initialization techniques are another family which set the weights in a decorrelated initial state so as to condition the network training to converge into better representations. For instance, Mishkin & Matas (2016) propose to initialize the network with decorrelated features using orthonormal initialization (Saxe et al. (2013)) while normalizing the variance of the outputs as well.
In this work we hypothesize that regularizing negatively correlated features is an obstacle for achieving better results and we introduce OrhoReg, a novel regularization technique that addresses the performance margin issue by only regularizing positively correlated feature weights. Moreover, OrthoReg is computationally efficient since it only regularizes the feature weights, which makes it very suitable for the latest CNN models. We verify our hypothesis through a series of experiments: first using MNIST as a proof of concept, secondly we regularize wide residual networks on CIFAR-10, CIFAR-100, and SVHN (Netzer et al. (2011)) achieving the lowest error rates in the dataset to the best of our knowledge.
2 DEALING WITH WEIGHT REDUNDANCIES
Deep Neural Networks (DNN) are very expressive models which can usually have millions of parameters. However, with limited data, they tend to overfit. There is an abundant number of techniques in order to deal with this problem, from L1 and L2 regularizations (Nowlan & Hinton (1992)), early-stopping, Dropout or DropConnect. Models presenting high levels of overfitting usually have a lot of redundancy in their feature weights, capturing similar patterns with slight differences which usually correspond to noise in the training data. A particular case where this is evident is in AlexNet (Krizhevsky et al. (2012)), which presents very similar convolution filters and even ”dead” ones, as it was remarked by Zeiler & Fergus (2014).
In fact, given a set of parameters θI,j connecting a set of inputs I = {i1, i2, . . . , in} to a neuron hj , two neurons {hj , hk}, j 6= k will be positively correlated, and thus fire always together if θI,j = θI,k and negatively correlated if θI,j = −θI,k. In other words, two neurons with the same or slightly different weights will produce very similar outputs. In order to reduce the redundancy present in the network parameters, one should maximize the amount of information encoded by each neuron. From an information theory point of view, this means one should not be able to predict the output of a neuron given the output of the rest of the neurons of the layer. However, this measure requires batch statistics, huge joint probability tables, and it would have a high computational cost.
In this paper, we will focus on the weights correlation rather than activation independence since it still is an open problem in many neural network models and it can be addressed without introducing too much overhead, see Table 1. Then, we show that models generalize better when different feature detectors are enforced to be dissimilar. Although it might seem contradictory, CNNs can benefit from having repeated filter weights with different biases, as shown by Li et al. (2016). However, those repeated filters must be shared copies and adding too many unshared filter weights to CNNs increases overfitting and the need for stronger regularization (Zagoruyko & Komodakis (May 2016)). Thus, our proposed method and multi-bias neural networks are complementary since they jointly increase the representation power of the network with fewer parameters.
In order to find a good target to optimize so as to reduce the correlation between weights, it is first required to find a metric to measure it. In this paper, we propose to use the cosine similarity between feature detectors to express how strong is their relationship. Note that the cosine similarity is equivalent to the Pearson correlation for mean-centered normalized vectors, but we will use the term correlation for the sake of clarity.
2.1 ORTHOGONAL WEIGHT REGULARIZATION
This section introduces the orthogonal weight regularization, a regularization technique that aims to reduce feature detector correlation enforcing local orthogonality between all pairs of weight vectors. In order to keep the magnitudes of the detectors unaffected, we have chosen the cosine similarity between the vector pairs in order to solely focus on the vectors angle β ∈ [−π, π]. Then, given any pair of feature vectors of the same size θ1, θ2 the cosine of their relative angle is:
cos(θ1, θ2) = 〈θ1, θ2〉 ||θ1||||θ2||
(1)
Where 〈θ1, θ2〉 denotes the inner product between θ1 and θ2. We then square the cosine similarity in order to define a regularization cost function for steepest descent that has its local minima when vectors are orthogonal:
C(θ) = 1
2 n∑ i=1 n∑ j=1,j 6=i cos2(θi, θj) = 1 2 n∑ i=1 n∑ j=1,j 6=i ( 〈θi, θj〉 ||θi||||θj || )2 (2)
Where θi are the weights connecting the output of the layer l − 1 to the neuron i of the layer l, which has n hidden units. Interestingly, minimizing this cost function relates to the minimization of the Frobenius norm of the cross-covariance matrix without the diagonal. This cost will be added to the global cost of the model J(θ;X, y), where X are the inputs and y are the labels or targets, obtaining J̃(θ;X, y) = J(θ;X, y) + γC(θ). Note that γ is an hyperparameter that weights the relative contribution of the regularization term. We can now define the gradient with respect to the parameters:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 〈θi, θi〉〈θk, θk〉 − θ(i,j)〈θi, θk〉2 〈θi, θi〉2〈θk, θk〉 (3)
The second term is introduced by the magnitude normalization. As magnitudes are not relevant for the vector angle problem, this equation can be simplified just by assuming normalized feature detectors:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 (4)
We then add eq. 4 to the backpropagation gradient:
∆θ(i,j) = −α ( ∇Jθ(i,j) + γ n∑ k=1,k 6=i θ(k,j)〈θi, θk〉 )
(5)
Where α is the global learning rate coefficient, J any target loss function for the backpropagation algorithm.
Although this update can be done sequentially for each feature-detector pair, it can be vectorized to speedup computations. Let Θ be a matrix where each row is a feature detector θ(I,j) corresponding to the normalized weights connecting the whole input I of the layer to the neuron j. Then, ΘΘt contains the inner product of each pair of vectors i and j in each position i, j. Subsequently, we
Algorithm 1 Orthogonal Regularization Step.
Require: Layer parameter matrices Θl, regularization coefficient γ, global learning rate α. 1: for each layer l to regularize do 2: η1 = norm rows(Θl) . Keep norm of the rows of Θl. 3: Θl1 = div rows(Θ
l, η1) . Keep a Θl1 with normalized rows. 4: innerProdMat = Θl1transpose(Θ l 1) 5: ∇Θl1 = γ(innerProdMat− diag(innerProdMat))Θl1 . Second term in eq. 6 6: ∆Θl = −α(∇JΘl + γ∇Θl1) . Complete eq. 6. 7: end for
(a) Global loss plot (eq. 2) (b) Local loss plot (eq. 7)
(c) Direction of gradients for the loss in (a). (d) Direction of gradients for loss in (b).
Figure 1: Comparison between the two loss functions represented by eq.2 and 7. (a) is the original loss, (b) is the new loss that discards negative correlations given for different λ values. It can be seen λ = 10 reaches a plateau when approximating to π2 . (c) and (d) shows the directions of the gradients for the two loss functions above. For instance, a red arrow coming from a green ball represents the gradient of the loss between the red and green balls with respect to the green one. In (d) most of the arrows disappear since the loss in (b) only applies to angles smaller than π2 .
subtract the diagonal so as to ignore the angle from each feature with respect to itself and multiply by Θ to compute the final value corresponding to the sum in eq. 5:
∆Θ = −α ( ∇JΘ + γ(ΘΘt − diag(ΘΘt))Θ ) (6)
Where the second term is∇CΘ. Algorithm 1 summarizes the steps in order to apply OrthoReg.
2.2 NEGATIVE CORRELATIONS
Note that the presented algorithm, based on the cosine similarity, penalizes any kind of correlation between all pairs of feature detectors, i.e. the positive and the negative correlations, see Figure 1a. However, negative correlations are related to inhibitory connections, competitive learning, and self-organization. In fact, there is evidence that negative correlations can help a neural population to increase the signal-to-noise ratio (Chelaru & Dragoi (2016)) in the V1. In order to find out
the advantages of keeping negative correlations, we propose to use an exponential to squash the gradients for angles greater than π2 (orthogonal):
C(θ) = n∑ i=1 n∑ j=1,j 6=i log(1 + eλ(cos(θi,θj)−1)) = log(1 + eλ(〈θi,θj〉−1)), ||θi|| = ||θj || = 1 (7)
Where λ is a coefficient that controls the minimum angle-of-influence of the regularizer, i.e. the minimum angle between two feature weights so that there exists a gradient pushing them apart, see Figure 1b. We empirically found that the regularizer worked well for λ = 10, see Figure 2b. Note that when λ ' 10 the loss and the gradients approximate to zero when vectors are at more than π2 (orthogonal). As a result of incorporating the squashing function on the cosine similarity, negatively correlated feature weights will not be regularized. This is different from all previous approaches and the loss presented in eq. 2, where all pairs of weight vectors influence each other. Thus, from now on, the loss in eq. 2 is named as global loss and the loss in eq. 7 is named as local loss.
The derivative of eq. 7 is:
δ
δθ(i,j) C(θ) = n∑ k=1,k 6=i λ eλ〈θi,θk〉θ(k,j) eλ〈θi,θk〉 + eλ (8)
Then, given the element-wise exponential operator exp, we define the following expression in order to simplify the formulas:
Θ̂ = exp(λ(ΘΘt)) (9)
and thus, the ∆ in vectorial form can be formulated as:
∇CΘ = λ (Θ̂− diag(Θ̂))Θ
Θ̂− diag(Θ̂) + eλ (10)
In order to provide a visual example, we have created a 2D toy dataset and used the previous equations for positive and negative γ values, see Figure 2. As expected, it can be seen that the angle between all pairs of adjacent feature weights becomes more uniform after regularization. Note that Figure 2b shows that regularization with the global loss (eq. 2) results in less uniform angles than using the local loss as shown in 2c (which corresponds to the local loss presented in eq. 7) because vectors in opposite quadrants still influence each other. This is why in Figure 2d, it can be seen that the mean nearest neighbor angle using the global loss (b) is more unstable than the local loss (c). As a proof of concept, we also performed gradient ascent, which minimizes the angle between the vectors. Thus, in Figures 2e and 2f, it can be seen that the locality introduced by the local loss reaches a stable configuration where feature weights with angle π2 are too far to attract each other.
The effects of global and local regularizations on Alexnet, VGG-16 and a 50-layer ResNet are shown on Figure 3. As it can be seen, OrthoReg reaches higher decorrelation bounds. Lower decorrelation peaks are still observed when the input dimensionality of the layers is smaller than the output since all vectors cannot be orthogonal at the same time. In this case, local regularization largely outperforms global regularization since it removes interferences caused by negatively correlated feature weights. This suggests why increasing fully connected layers’ size has not improved networks performance.
3 EXPERIMENTS
In this section we provide a set of experiments that verify that (i) training with the proposed regularization increases the performance of naive unregularized models, (ii) negatively correlated feature weights are useful, and (iii) the proposed regularization improves the performance of state-of-the-art models.
3.1 VERIFICATION EXPERIMENTS
As a sanity check, we first train a three-hidden-layer Multi-Layer Perceptron (MLP) with ReLU non-liniarities on the MNIST dataset (LeCun et al. (1998)). Our code is based in the train-a-digit-classifier example included in torch/demos1, which uses an upsampled version of the dataset (32×32). The only pre-processing applied to the data is a global standardization. The model is trained with SGD and a batch size of 200 during 200 epochs. No momentum neither weight decay was applied. By default, the magnitude of the weights of this experiments is recovered after each regularization step in order to prove the regularization only affects their angle.
Sensitivity to hyperparameters. We train a three-hidden-layer MLP with 1024 hidden units, and different γ and λ values so as to verify how they affect the performance of the model. Figure 4a
1https://github.com/torch/demos
shows that the model effectively achieves the best error rate for the highest gamma value (γ = 1), thus proving the advantages of the regularization. On Figure 4b, we verify that higher regularization rates produce more general models. Figure 5a depicts the sensitivity of the model to λ. As expected, the best value is found when lambda corresponds to Orthogonality (λ ' 10). Negative Correlations. Figure 5b highlights the difference between regularizing with the global or the local regularizer. Although both regularizations reach better error rates than the unregularized counterpart, the local regularization is better than the global. This confirms the hypothesis that negative correlations are useful and thus, performance decreases when we reduce them.
Compatibility with initialization and dropout. To demonstrate the proposed regularization can help even when other regularizations are present, we trained a CNN with (i) dropout (c32-c64-l512-d0.5-l10)2 or (ii) LSUV initialization (Mishkin & Matas (2016)). In Table 2, we show that best results are obtained when orthogonal regularization is present. The results are consistent with the hypothesis that OrthoReg, as well as Dropout and LSUV, focuses on reducing the model redundancy. Thus, when one of them is present, the margin of improvement for the others is reduced.
2cxx = convolution with xx filters. lxx = fully-connected with xx units. dxx = dropout with prob xx.
3.2 REGULARIZATION ON CIFAR-10 AND CIFAR-100
We show that the proposed OrthoReg can help to improve the performance of state-of-the-art models such as deep residual networks (He et al. (2015a)). In order to show the regularization is suitable for deep CNNs, we successfuly regularize a 110-layer ResNet3 on CIFAR-10, decreasing its error from 6.55% to 6.29% without data augmentation.
In order to compare with the most recent state-of-the-art, we train a wide residual network (Zagoruyko & Komodakis (November 2016)) on CIFAR-10 and CIFAR-100. The experiment is based on a torch implementation of the 28-layer and 10th width factor wide deep residual model, for which the median error rate on CIFAR-10 is 3.89% and 18.85% on CIFAR-1004. As it can be seen in Figure 6, regularizing with OrthoReg yields the best test error rates compared to the baselines.
The regularization coefficient γ was chosen using grid search although similar values were found for all the experiments, specially if regularization gradients are normalized before adding them to the weights. The regularization was equally applied to all the convolution layers of the (wide) ResNet. We found that, although the regularized models were already using weight decay, dropout, and batch normalization, best error rates were always achieved with OrthoReg.
Table 3 compares the performance of the regularized models with other state-of-the-art results. As it can be seen the regularized model surpasses the state of the art, with a 5.1% relative error improvement on CIFAR-10, and a 1.5% relative error improvement on CIFAR-100.
3.3 REGULARIZATION ON SVHN
For SVHN we follow the procedure depicted in Zagoruyko & Komodakis (May 2016), training a wide residual network of depth=28, width=4, and dropout. Results are shown in Table 4. As it
3https://github.com/gcr/torch-residual-networks 4https://github.com/szagoruyko/wide-residual-networks
can be seen, we reduce the error rate from 1.64% to 1.54%, which is the lowest value reported on this dataset to the best of our knowledge.
4 DISCUSSION
Regularization by feature decorrelation can reduce Neural Networks overfitting even in the presence of other kinds of regularizations. However, especially when the number of feature detectors is higher than the input dimensionality, its decorrelation capacity is limited due to the effects of negatively correlated features. We showed that imposing locality constraints in feature decorrelation removes interferences between negatively correlated feature weights, allowing regularizers to reach higher decorrelation bounds, and reducing the overfitting more effectively.
In particular, we show that the models regularized with the constrained regularization present lower overfitting even when batch normalization and dropout are present. Moreover, since our regularization is directly performed on the weights, it is especially suitable for fully convolutional neural networks, where the weight space is constant compared to the feature map space. As a result, we are able to reduce the overfitting of 110-layer ResNets and wide ResNets on CIFAR-10, CIFAR-100, and SVHN improving their performance. Note that despite OrthoReg consistently improves state of the art ReLU networks, the choice of the activation function could affect regularizers like the one presented in this work. In this sense, the effect of asymmetrical activations on feature correlations and regularizers should be further investigated in the future.
ACKNOWLEDGEMENTS
Authors acknowledge the support of the Spanish project TIN2015-65464-R (MINECO/FEDER), the 2016FI B 01163 grant of Generalitat de Catalunya, and the COST Action IC1307 iV&L Net (European Network on Integrating Vision and Language) supported by COST (European Cooperation in Science and Technology). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of a Tesla K40 GPU and a GTX TITAN GPU, used for this research.

Review:
The paper proposes a new regulariser for CNNs that penalises positive correlations between feature weights, but does not affect negative correlations. An alternative version which penalises all correlations regardless of sign is also considered. The paper refers to these as "local" and "global" respectively, which I find a bit confusing as these are very general terms that can mean a plethora of things.
The experimental validation is quite rigorous. Several experiments are conducted on benchmark datasets (MNIST, CIFAR-10, CIFAR-100, SVHN) and improvements are demonstrated in most cases. While these improvements may seem modest, the baselines are already very competitive as the authors pointed out. In some cases it does raise some questions about statistical significance though. More results with the global regulariser (i.e. not just on MNIST) would have been interesting, as the main novelty in the paper seems to be leaving the negative correlations alone, so it would be interesting to see exactly how much of a difference this makes.
One of my main concerns is ambiguity stemming from the fact that the paper sometimes discusses activations and sometimes filter weights, but refers to both as "features". However, the authors have already said they will address this.
The paper somewhat ignores interactions with the choice of nonlinearity, which seems like it could be very important; especially because the goal is to obtain feature activations that are uncorrelated, and this is done only by applying a penalty to the weights (i.e. in a data-agnostic way and also ignoring any nonlinearity). I believe the authors already mentioned in their responses to reviewer questions that this would be addressed, but I think this important and it definitely needs to be discussed.
In response to the authors' answer to my question about the role of biases: as they point out, it is perfectly possible to combine their proposed technique with the "multi-bias" approach, but this was not really my point. Rather, the latter is an example that challenges the idea that features should not be positively correlated / redundant, which seems to be the assumption that this work is built upon. My current intuition is that it's okay to have correlated features, as long as you're not wasting model capacity on them. This is the case for "multi-bias", seeing as the weights are shared across sets of correlated features.
The dichotomy between regularisation methods that reduce capacity and those that don't which is described in the introduction seems a bit arbitrary to me, especially considering that weight decay is counted among the former and the proposed method is counted among the latter. I think this very much depends on ones definition of model capacity (clearly weight decay does not actually reduce the number of parameters in a model).
Overall, the work is perhaps a bit incremental, but it seems to be well-executed. The results are convincing, even if they aren't particularly ground-breaking.