SHADE: SHA NNON DE CAY I NFORMATION -B ASED R EGULARIZATION FOR D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Regularization is a big issue for training deep neural networks.
In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay.
The originality of the approach is to deﬁne a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data ﬁtting term.
We explain why this quantity makes our model able to achieve invariance with respect to input variations.
We empirically validate the efﬁciency of our approach to improve classiﬁcation performances compared to standard regularization schemes on several standard architectures.
I NTRODUCTION
Deep neural networks (DNNs) have shown impressive state-of-the-art results in the last years on numerous tasks and especially for image classiﬁcation (Krizhevsky et al., 2012;
He et al., 2016).
One key element is the use of very deep models with a huge number of parameters.
Accordingly, DNNs need to be trained on a lot of data ( e.g. ImageNet).
Regularization methods such as weight decay (Krogh & Hertz, 1992), dropout (Srivastava et al., 2014) or batch normalization (Ioffe & Szegedy, 2016) are common practice to mitigate the ratio between the numbers of training samples and model parameters.
Despite constant progress of DNNs performance, their generalization ability is still largely misunderstood and theoretical tools such as PAC-based analysis seem limited to explain it.
Meanwhile, the question of DNN regularization remains open as demonstrated by Zhang et al.
Information Bottleneck (IB) is an interesting framework to address regularization (Tishby et al., 1999).
It suggests that it is possible to compare the generalization ability of different models by studying the compression rate of their representations: a model that is able to remove more information from the input while providing enough information about the label is more likely to generalize well.
Considering an input variable X , label C and its deep representation Y = h
( X ) , IB regularizes the training by minimizing the mutual information
I ( X, Y ) at constant mutual information
I ( C, Y ) .
Along with IB also come some theoretical investigations, with the deﬁnition of generalization bounds in Shamir et al.
A challenge for applying IB principles to DNNs is the deﬁnition of information-based regularizers that are differentiable and easily computable in order to be efﬁciently trained using back-propagation (LeCun et al., 1998).
There are recent attempts in this direction.
For example, Alemi et al.
(2017) proposes a DNN that predicts the parameters of the distribution of the random representation variables conditionally to the input and use it to compute IB measures with variational bounds.
Achille &
Soatto (2016) model the information about the data in the learned weights and propose to minimize it.
In this paper, we propose a new regularization loss minimizing H ( Y | C ) , the entropy of the representation variable conditionally to the label variable, instead of the mutual information I ( X, Y ) .
For a deterministic model, meaning that the entropy H ( Y | X ) is null, we have I ( X, Y ) =
I ( C, Y ) + H ( Y | C ) 1 .
By minimizing directly H ( Y | C ) we explicitly decouple the data ﬁtting term
I ( C, Y ) , which attempts at ﬁnding correlations between inputs and labels, and a term which represents the compression rate of the representation H ( Y | C ) .
Our contributions are three-fold.
First, we derive in Section 2 a new regularization scheme, called
SHADE for SHAnnon DEcay, to tackle the difﬁculty of getting an accurate, differentiable and scalable estimate of H ( Y | C ) .
It is based on a latent code Z representing the activation state of a neuron Y , allowing us to express a complete tractable loss with few additional computation.
Second, we discuss the interest of our regularization framework based on conditional entropy in Section 3.
In particular, we argue that this measure H ( Y | C ) is well designed to compare representations in terms of invariance while still taking the target task into account.
Finally, we extensively experiment our
SHADE regularizer on different deep learning models and several challenging datasets (CIFAR-10,
ImageNet, MNIST-M) in Section 4.
SHADE: A NEW R EGULARIZATION M ETHOD B ASED ON H ( Y | C )
Notations and deﬁnitions.
X is the input from input space X with high dimensionality and C is the class label from ﬁnite class space C
= { 1 , 2 , .., |C|} .
A hypothesis space is designed with a family of parametric (continuous) functions H
= { h ( w, . ) :
X → Y ; w ∈ W = R m } .
A model h ∈ H outputs a representation of the input from which a class is predicted.
In many tasks like image classiﬁcation, the representation Y = h ( X ) of an input X usually deﬁnes a distribution, noted q Y | X , on the class space ( e.g. obtained by applying a softmax function).
The training in supervised learning consists in ﬁnding the parameters w of h minimizing the expectancy of a loss function L cls ( Y, C ) that quantiﬁes the difference between the prediction and the label.
It is usually the cross-entropy between the obtained distribution q Y | X and the one-hot distribution with unit probability for the class C noted p C :
(cid:96) cls ( Y, C )
= − (cid:80) c ∈C p C ( c ) log (cid:0) q Y
| X ( c ) (cid:1) .
A regularization term Ω( · ) with a coefﬁcient β is often added to the classiﬁcation loss, aiming at inﬂuencing the learning toward models with lower complexity 2 .
In image classiﬁcation, the most commonly used regularization criterion is the weight decay: Ω( w ) =
||
w || 22 .
In the IB framework, the regularization term would be Ω( w ) =
I (cid:0) h ( w, X ) , X (cid:1) .
The ﬁnal loss for supervised learning is:
[Equation]
In this section, we will further describe SHADE, a new regularization term based on the conditional entropy H ( Y | C ) designed to drive the optimization toward more invariant representation.
2. E NTROPY - BASED R EGULARIZATION FOR D EEP N EURAL N ETWORKS
Layer-wise regularization.
A DNN is composed of a number L of layers that transform sequentially the input.
Each one can be seen as an intermediate representation variable, noted Y (cid:96) for layer (cid:96) , that is determined by the output of the previous layer and a set of parameters w (cid:96) .
Each layer ﬁlters out a certain part of the information from the initial input.
Thus, from the data processing inequality in Cover & Thomas (1991) can be derived the following inequalities for any (cid:96) :
[Equation]
Similarly to the recommendation of Tishby & Zaslavsky (2015), we apply regularization to all the layers, using a layer-wise criterion H ( Y (cid:96) | C ) , and producing a global criterion:
[Equation]
Unit-wise regularization.
Examining one layer (cid:96) , its representation variable is a random vector of coordinates Y (cid:96),i and of dimension D (cid:96) :
Y (cid:96) =
( Y
[Table]
, 1 , ..., Y
[Table]
,D (cid:96) ) . The upper bound 3 H ( Y (cid:96) | C ) ≤
[Table] [Table]
i =1 H ( Y (cid:96),i | C ) enables to deﬁne a unit-wise criterion that SHADE seeks to minimize.
For each unit i of every layer (cid:96) we design a loss ω unit ( Y (cid:96),i | C )
=
H ( Y (cid:96),i | C ) that will be part of the global regularization loss:
[Equation]
Later in the paper, we use the notation Y instead of Y (cid:96),i for simplicity since the coordinates are all considered independently to deﬁne ω unit ( Y (cid:96),i | C ) .
2. E STIMATING E NTROPY
In this section, we describe how to deﬁne a loss based on the measure
H ( Y | C ) with Y being one coordinate variable of one layer.
Deﬁning this loss is not obvious as the gradient of H ( Y | C ) with respect to the layer’s parameters may be computationally intractable.
Y has an unknown distribution and without modeling it properly it is not possible to compute H ( Y | C ) precisely for the following reasons.
Since H ( Y | C ) = (cid:80) c ∈C p ( c ) H ( Y | c ) it is necessary to compute |C| different entropies H ( Y | c ) .
This means that, given a batch, the number of samples used to estimate one of these entropies is divided by |C| on average which becomes particularly problematic when dealing with a large number of classes such as the 1,000 classes of ImageNet.
Furthermore, entropy estimators are extremely inaccurate considering the number of samples in a batch.
For example, LME estimators of entropy in Paninski (2003) converge in O ( (log K ) 2 / K ) for K samples.
Finally, most estimators such as LME require discretizing the space in order to approximate the distribution via a histogram.
This raises issues on the bins deﬁnition considering that the variable distribution is unknown and varies during the training in addition to the fact that having a histogram for each neuron is computationally and memory consuming.
To tackle these drawbacks we investigate the two following tricks: the introduction of a latent variable that enables to use more examples to estimate the entropy; and a bound on the entropy of the variable by an increasing function of its variance to avoid the issue of entropy estimation with a histogram and make the computation tractable and scalable.
Latent code.
First, inspecting a neuron Y (before ReLU), the ReLU activation makes it act as a detector, returning a signal when a certain pattern is present on the input.
If the pattern is absent the signal is zero, otherwise, it quantiﬁes the resemblance with it.
We therefore propose to associate a binomial variable Z with each unit variable Y (before ReLU).
This variable Z indicates if a particular pattern is present on the input ( Z = 1 when Y (cid:29) 0 ) or not ( Z = 0 when Y (cid:28) 0 ).
It acts like a latent code in variational models ( e.g. Kingma & Welling, 2014) or in generative models ( e.g. Chen et al., 2016).
In our implementation, we chose a binomial distribution p ( Z = 1 | Y )
= sigmoid( Y ) that matches this intuition, as detailed in Sec.
Furtherfore, it is very likely that most intermediate features of a DNN can take similar values for inputs of different classes – this is especially true for low-level features.
The semantic information provided by a feature is thus more about a particular pattern than about the class itself.
Only the association of features allows determining the class.
So Z represents a semantically meaningful factor about the class C and from which the input X is generated.
The feature value Y is then a quantiﬁcation of the possibility for this semantic attribute Z to be present in the input or not.
Then, we assume the Markov chain C
→ Z → X → Y .
Indeed, during the training, the distribution of Y varies in order to get as close as possible to a sufﬁcient statistic of X for C (see deﬁnition in Cover & Thomas, 1991).
Therefore, we expect Z to be such that Y draws near a sufﬁcient statistic of Z for C as well.
By assuming the sufﬁcient statistic relation I ( Y, C )
=
I ( Y, Z ) we get the equivalent equality H ( Y | C )
=
H ( Y | Z ) , and ﬁnally obtain:
[Equation]
This modeling of Z as a binomial variable (one for each unit) has the advantage of enabling good estimators of conditional entropy since we only divide the batch into two sets for the estimation ( z = 0 and 1 ) regardless of the number of classes. ( Y 1: Initialize: µ 0 = − 1 , µ 1 = 1 , p 0 = p 1 = 0 . 5 , λ = 0 .2: for each mini-batch {
, k
} do 3: for z
} do 4: p [Equation] ← [Equation]
[Equation]
y ( k ) ) 5: µ
[Equation]
←
[Equation]
−
[Equation]
[Equation]
z
[Equation]
y (
[Equation]
p z y ( k ) 6: Variance bound. The previous trick allows computing fewer entropy estimates to obtain the global conditional entropy, thus increasing the sample size used for each entropy estimation.
Unfortunately, it does not solve the bin deﬁnition issue.
To address this, we propose to use the following bound on
H ( Y | Z ) , that does not require the deﬁnition of bins:
[Equation]
This bound holds for any continuous distributions Y
and there is equality if the distribution is
Gaussian.
For many other distributions such as the exponential one, the entropy is also equal to an increasing function of the variance.
In addition, one main advantage is that variance estimators are much more robust than entropy estimators, converging in O ( 1 / K ) for K samples instead of
O ( log( K ) 2 / K ) .
Finally, the ln function being one-to-one and increasing, we only keep the simpler term V ar
( Y | Z ) to design our ﬁnal loss:
[Equation]
In next section, we detail the deﬁnition of the differential loss using V ar( Y | Z ) as a criterion computed on a mini-batch.
2. I NSTANTIATING SHADE
For one unit of one layer, the previous criterion writes:
[Equation]
The quantity V ar( Y | Z ) can be estimated with Monte-Carlo sampling on a mini-batch of input-target pairs (cid:8) ( x ( k ) , c ( k ) )
(cid:9) 1 ≤ k ≤ K of intermediate representations (cid:8) y
( k ) (cid:9) 1 ≤ k ≤ K as in Eq.
p ( Z | y ) interpreted as the probability of presence of attribute Z on the input, it should clearly be modeled such that p ( Z = 1 | y ) increases with y .
The more similarities between the input and the pattern represented by y , the higher the probability of presence for Z .
We suggest using: p
[Equation]
Z
[Equation]
[Equation]
with sigmoid function σ ( y ) =
[Equation]
y . For the expected values µ z = E ( Y | z ) we use a classic moving average that is updated after each batch as described in Algorithm 1.
Note that the expected values are not changed by the optimization since they have no inﬂuence on the entropy H ( Y | Z ) .
The concrete behavior of SHADE can be interpreted by analyzing its gradient as described in
Appendix C.
For this proposed instantiation, our SHADE regularization penalty takes the form:
[Equation]
We have presented a regularizer that is applied layer-wise and that can be integrated into the usual optimization process of a DNN.
The additional computation and memory usage induced by SHADE is almost negligible (computation and storage of two moving averages per neuron).
Namely, SHADE adds half as many parameters as batch normalization does.
C ONDITIONAL E NTROPY AS I NVARIANCE M EASURE FOR C LASSIFICATION
Inspired by the IB framework, we have derived a criterion that SHADE loss aims at minimizing.
In this section, we further explore the relevance of our conditional entropy term for regularization.
We ﬁrst establish a connection between the entropy H ( Y ) of a representation and its invariance.
Then we explain why H ( Y | C ) seems to be a better ﬁtting criterion for a classiﬁcation task.
H ( Y ) to measure invariance.
For an input X and a deterministic mapping f
( X ) =
Y , we have
H ( Y | X )
=
and therefore:
[Equation]
H ( X ) being ﬁxed, H ( Y ) is inversely related to H ( X | Y ) .
Besides, we assume that H ( X | Y ) is a good measure to quantify how invariant a representation is.
Indeed, if a representation is invariant to many transformations, many inputs have the same representation.
Consequently, given a representation sample, it is difﬁcult to guess from which input it has been computed.
These properties are perfectly captured by H ( X | Y ) , representing the uncertainty in X knowing Y .
The bigger the uncertainty, the harder it is to predict X precisely.
Concretely, when trying to guess X knowing Y , we can lower bound the error made in the best case, with an increasing function of the conditional entropy as developed in Appendix A.
Therefore, it seems that H ( Y ) is a good measure of the invariance of the model.
In the particular case of deep learning, He et al.
(2016) explain that the stacking of multiple layers is responsible for improving the generalization of DNN.
This fact can be explained by the data processing inequality (Cover & Thomas, 1991).
In the case of ﬁnite input space, each layer is responsible for ﬁltering a certain amount of information.
As clearly illustrated in Tishby & Zaslavsky (2015), for each stage, the representation has a lower entropy than the representation of the preceding layer.
Increasing the depth increases the capacity of the network to reduce the overall entropy of the
DNN representation thus increasing their invariance.
H ( Y ) as a regularizer.
From this intuition, one can use a variance bound similar to Eq.
H ( Y ) ≤ 1 / 2 ln(2 πe V ar( Y )) to derive a loss in order to minimize the representations’ entropy H ( Y ) .
For instance, we can show that the weight decay reduces this variance under some approximations.
In fact when Y = w (cid:62) X + b , by estimating Λ = C ov( X ) , the variance takes the immediate form V ar( Y ) =
w (cid:62) Λ w .
If Λ = Id , meaning that the input coordinates are considered independent with unit variance, then V ar( Y )
=
||
w ||
It corresponds to the weight decay regularization or L 2 penalty.
Even if within a DNN layer, the batch normalization tends to enforce this unit variance hypothesis and the depth of DNN tends to ensure the independence hypothesis the weight decay remains poor at improving generalization as illustrated in Zhang et al.
Considering fewer assumptions, it is possible to reduce the variance of an output coordinate by penalizing its empirical variance on a batch during the training.
We introduce an alternative loss on the variance called VarEntropy, constructed the same way SHADE has been derived, but avoiding the introduction of a latent variable Z :
[Equation]
Even if experimentally, VarEntropy regularization seems to behave better than simple weight decay, it is not as efﬁcient as SHADE and we try to explain why below.
Importance of conditional entropy.
We have seen that lowering the entropy of the representation enables to make it more invariant.
However, another fact reported by He et al.
(2016) is that stacking layers increases the difﬁculty to train the network.
Indeed, when reducing too much the entropy, there is a risk that the information about the label is ﬁltered as well.
The representation is so invariant that it is no longer possible to distinguish between the classes.
In He et al.
(2016), they solve this issue by forcing the transmission of additional information through skip-connections while IB prescribes to maximize the compression rate at constant information about the label.
All this highlights the fact that having invariant representations is interesting if it is intra-class invariant.
Since we do not want two inputs from different classes to have the same representation, we prefer to focus on a criterion quantifying the intra-class compression rate in order to maximize intra-class invariance: H ( Y | C ) .
As mentioned in the introduction, H ( Y | C ) thus differs from standard IB frameworks based on I ( X, Y )
(Achille & Soatto, 2016; Alemi et al., 2017).
Contrarily to the mutual information
I ( X, Y ) =
H ( Y | C )
+ I ( C, Y ) , our regularizer applied to H ( Y | C ) ignores I ( C, Y ) .
When minimizing I ( X, Y ) , there is no control on how both terms H ( Y | C ) and I ( C, Y ) are modiﬁed.
Our regularizer is therefore beneﬁcial in the sense that minimizing H ( Y | C ) does not conﬂict with the mutual information
I ( C, Y ) between the representation and the label, information that is therefore for classiﬁcation and should not be penalized.
E XPERIMENTS 4. I MAGE C LASSIFICATION WITH V ARIOUS A RCHITECTURES ON CIFAR-
[Table]
We perform image classiﬁcation on the CIFAR-10 dataset, which contains 50k training images and 10k test images of 32 × 32 RGB pixels, fairly distributed within 10 classes (see Krizhevsky, 2009, for details).
Following the architectures used in Zhang et al.
(2017), we use a small Inception model, a three-layer MLP, and an AlexNet-like model with 3 convolutional and 2 fully connected layers.
We also use a ResNet architecture from Zagoruyko & Komodakis (2016).
Those architectures represent a large family of DNN and some have been well studied in Zhang et al.
(2017) within the generalization scope.
For training, we use randomly cropped images of size 28 × 28 with random horizontal ﬂips.
For testing, we simply center-crop 28 × 28 images.
We use momentum SGD for optimization (same protocol as Zhang et al., 2017).
We compare SHADE with three regularization methods, namely weight decay , dropout and VarEntropy presented in Eq.
For all architectures, the regularization parameters have been crossvalidated to ﬁnd the best ones for each method and the obtained accuracies on the test set are reported in Table 1.
We obtain the same trends as Zhang et al.
(2017), which get a small improvement of 0.31% with weight decay on AlexNet.
The improvement with weight decay is slightly more important with
ResNet and Inception (0.87% and 0.90%) probably thanks to the use of batch normalization.
In our experiments dropout improves generalization performances only for AlexNet and MLP.
It is known that the use of batch normalization lowers the beneﬁt of dropout, which is in fact not used in He et al.
We ﬁrst notice that for all kind of architectures the use of SHADE signiﬁcantly improves the generalization performance.
It demonstrates the ability of SHADE to regularize the training of deep architectures.
Moreover, SHADE systematically outperforms other regularizations of the same type [Figure] such as weight decay or VarEntropy, illustrating the advantage of minimizing the conditional entropy instead of the entropy directly.
Finally, SHADE shows better performances than dropout on all architecture except on AlexNet, for which they seem to be complementary, probably because of the very large number of parameters in the fully-connected layers, with best performances obtained with SHADE coupled with dropout (named SHADE+D).
This association is also beneﬁcial for MLP.
On Inception and ResNet, even if dropout and SHADE independently improve generalization performances, their association is not as good as SHADE alone, probably because it enforces too much regularization.
4. L ARGE S CALE C LASSIFICATION ON I MAGE N ET
In order to experiment SHADE regularization on very large scale dataset, we train on ImageNet (Russakovsky et al., 2015) a WELDON network from Durand et al.
(2016) adapted from ResNet-101.
This architecture changes the forward and pooling strategy by using the network in a fully-convolutional way and adding a max+min pooling, thus improving the performance of the baseline network.
We used the pre-trained weights of ResNet-101 (from the torchvision package of PyTorch) giving performances on the test set of 77.56% for top-1 accuracy and 93.89% for top-5 accuracy.
Provided with the pre-trained weights, the WELDON architecture obtains 78.51% for top-1 accuracy and 94.65% for top-5 accuracy.
After ﬁne tuning the network using SHADE for regularization we ﬁnally obtained 80.14% for top-1 accuracy and 95.35% for top-5 accuracy for a concrete improvement.
This demonstrates the ability to apply SHADE on very large scale image classiﬁcation successfully.
4. T RAINING WITH A L IMITED N UMBER OF S AMPLES
When datasets are small, DNNs tend to overﬁt quickly and regularization becomes essential.
Because it tends to ﬁlter out information and make the network more invariant, SHADE seems to be well ﬁtted for this task.
To investigate this, we propose to train DNNs with and without SHADE on CIFARand MNIST-M with different numbers of samples in the training set.
First, we tested this approach on the digits dataset MNIST-M (Ganin & Lempitsky, 2015).
This dataset consists of the MNIST digits where the background and digit have been replaced by colored and textured information (see Fig.
1c for examples).
The interest of this dataset is that it contains lots of unnecessary information that should be ﬁltered out, and is therefore well adapted to measure the effect of SHADE.
A simple convolutional network has been trained with different numbers of samples of MNIST-M and the optimal regularization weight for SHADE have been determined on [Figure] the validation set (see training details in Appendix D).
The results can be seen on Figure 1a.
We can see that especially for small numbers of training samples ( < 1000), SHADE provides an important gain of 10 to 15% points over the baseline.
This shows that SHADE helped the model in ﬁnding invariant and discriminative patterns using less data samples.
Additionally, Figure 1c shows samples that are misclassiﬁed by the baseline model but correctly classiﬁed when using SHADE.
These images contain a large amount of intra-class variance (color, texture, etc.)
that is not useful for the classiﬁcation tasks, explaining why adding SHADE, that encourages the model to discard information, allows important performance gains on this dataset and especially when only few training samples are given.
Finally, to conﬁrm this behavior, we also applied the same procedure in a more conventional setting by training an Inception model on CIFAR-10.
Figure 1b shows the results in that case.
We can see that once again SHADE helps the model gain in performance and that this behavior is more noticeable when the number of samples is limited, allowing a gain of 6% when using 4000 samples.
4. C LASS -I NFORMATION P RESERVATION BY C ONDITIONAL E NTROPY
The main difference between SHADE and VarEntropy is the introduction of the latent variable Z , supposed to contain the information about the label C .
The motivation of this extension is that minimizing H ( Y | C ) instead of H ( Y ) enables to save the class information during the optimization of the regularization loss, as explained in Section 3.
To illustrate this beneﬁt, we compare the impact of the two regularization losses on the classiﬁcation performances of a pre-trained model.
To do so, we ﬁne tune a pre-trained Inception model only with a regularization loss, either based on V ar( Y ) or V ar( Y | Z ) ; without any label data or classiﬁcation loss.
The network performance obviously declines for both regularizers as we can see on Figure 2. However, this decline is slower with
SHADE than VarEntropy. This conﬁrms the intuition that Z contains class-information and that SHADE produces less class-information ﬁltering.
This is explained by the optimization of the metric V ar( Y | Z ) that uses implicitly-learned information encoded in the network.
C ONCLUSION
In this paper, we introduced a new regularization method for DNNs training, SHADE, which focuses on minimizing the entropy of the representation conditionally to the labels.
This regularization aims at increasing the intra-class invariance of the model while keeping class information.
SHADE is tractable, adding only a small computational overhead when included into an efﬁcient SGD training.
We show that our SHADE regularization method signiﬁcantly outperforms standard approaches such as weight decay or dropout with various DNN architectures on CIFAR-10.
We also validate the scalability of SHADE by applying it on ImageNet.
The invariance potential brought out by
SHADE is further illustrated by its ability to ignore irrelevant visual information (texture, color) on
MNIST-M. Finally, we also highlight the increasing beneﬁt of our regularizer when the number of training examples becomes small.
A E NTROPY BOUNDING THE RECONSTRUCTION ERROR
In section 3 we highlight a link between the entropy H ( X | Y ) and the difﬁculty to recover the input X from its representation Y .
Here we exhibit a concrete relation between the reconstruction error, that quantiﬁes the error made by a strategy that predicts X from Y , and the conditional entropy.
This relation takes the form of an inequality, bounding the error measure in the best case (with the reconstruction strategy that minimizes the error) by an increasing function of the entropy.
We note ˆ x ( Y ) ∈
X the reconstruction model that tries to guess X from Y .
The discrete case In case the input space is discrete, we consider the zero-one reconstruction error for one representation point Y : ε ( Y ) =
E X
[ 1 (ˆ x ( Y ) (cid:54) = X )] .
This is the probability of error when predicting ˆ x ( Y ) from Y .
The function that minimizes the expected error E = E Y ( ε ( Y )) is ˆ x ( Y ) =
arg max x ∈X p ( x | Y ) as shown in Proof 1.
We derive the following inequality:
[Equation]
The left side of the inequality uses Fano’s inequality in Cover & Thomas (1991), the right one is developed in proof 2.
This ﬁrst bound shows how the reconstruction error and the entropy are related.
For very invariant representations, it is hard to recover the input from Y and the entropy of H ( X | Y ) is high.
Besides, there can be an underlying continuity in the input space and it could be unfair to penalize predictions close to the input as much as predictions far from it.
We expose another case below that takes this proximity into account.
The continuous case In the case of convex input space and input variable with continuous density, we consider the 2-norm distance as reconstruction error: ε ( Y ) =
E X | Y (cid:2)
||
X − ˆ X ( Y ) ||
22 (cid:3) .
This error penalizes the average distance of the input and its reconstruction from Y .
The function that minimizes the expected error E = E Y [ ε ( Y )] is the conditional expected value: ˆ x ( y ) =
E [ X | Y =
y ] .
Then
E = V ar( X | Y ) .
Helped by the well-known inequality H ( X | Y ) ≤ 12 ln(2 πe V ar( X | Y ))
we obtain:
[Equation]
Here again, notice that a high entropy H ( X | Y ) implies a high reconstruction error in the best case.
A. P ROOF
We have
[Equation] [Equation]
Since
[Equation] [Equation]
the reconstruction that minimizes the error is ˆ x ( y ) =
arg max x ∈ X p ( x | y ) .
However, this is theoretical because in most cases p ( x | Y ) is unknown.
A. P ROOF
We have:
[Equation]
The Eq.
(21) is obtained using Jensen inequality and Eq.
(23) is obtained using the result of Proof
[Equation]
Thus,
[Equation]
B D EVELOPMENT OF SHADE LOSS
Below is the detail of the development of Equation (8)
[Equation]
C SHADE G RADIENTS
Here is studied the inﬂuence of SHADE on a gradient descent step for a single neuron Y of a single layer and for one training sample X .
The considered case of a linear layer, we have: Y = w (cid:62) X + b .
The gradient of Ω SHADE with respect to w is:
[Equation]
with
[Equation]
and
[Equation]
We can interpret the direction of this gradient by analyzing the two terms δ 1 and δ 2 as follows:
• δ 1 : If ( y − µ 0 ) 2 is bigger than ( y − µ 1 ) 2 that means that y is closer to µ 1 than it is to µ 0 .
Then δ 1 is positive and it contributes to increasing y meaning that it increases the probability of Z being from mode 1.
In a way it increases the average margin between positive and negative detections.
Note that if there is no ambiguity about the mode of Z meaning that σ ( y ) or 1 − σ ( y ) is very small then this term has negligible effect.
• δ 2 : This term moves y toward the µ z of the mode that presents the bigger probability.
This has the effect of concentrating the outputs around their expectancy depending on their mode to get sparser activation.
D E XPERIMENTS DETAILS ON MNIST-M
Dataset splits and creation.
To create MNIST-M, we kept the provided splits of MNIST, so we have 55,000 training samples, 5,000 validation samples, and 10,000 test samples.
Each digit of
MNIST is processed to add color and texture by taking a crop in images from BST dataset.
This procedure is explained in Ganin & Lempitsky (2015).
Subsets of limited size.
To create the training sets of limited size N , we keep N/ 10 (since there are 10 classes) randomly picked samples from each class.
When increasing N we keep the previously picked samples so the training samples for N = 100 are a subset of the ones for N = 250 .
The samples chosen for a given value of N are the same across all models trained using this number of samples.
Image preprocessing.
The only preprocessing applied to the input images is that their values are rescaled from [0 , 1] to [ − 1 , 1] .
Optimization.
For the training, we use mini-batch of size 50 and use Adam optimizer with the recommended parameters, i.e. λ r = 0 .
001 , β 1 = 0 .
9 , β 2 = 0 .
999 , (cid:15) =
10 − 8 .
Hyperparameter tuning.
For weight decay and SHADE, the optimal regularization weight of each model (for each value of N ) has been chosen to maximize the accuracy on the validation sets.
We tried the values { 10 − i , i = 1 .. 7 } .
Model architecture.
The model have the following architecture:
• 2D convolution ( 64 × 5 × 5 kernel, padding 2, stride 1) + ReLU •
MaxPooling
×• 2D convolution ( 64 × 3 × 3 kernel, padding 1, stride 1) + ReLU • MaxPooling 2 ×• 2D convolution ( 64 × 3 × 3 kernel, padding 1, stride 1) + ReLU • MaxPooling 2 ×• Fully connected (1024 inputs, 10 outputs) + SoftMax E E XPERIMENTS DETAILS ON I MAGENET
The ﬁne tuning in the experiment section 4.2 has been done with momentum-SGD with a learning rate of 10 − 5 and a momentum of 0 .
9 and a batch size of 16 images.
It took 8 epochs to converge.