O N THE T RAJECTORY OF S TOCHASTIC G RADIENT D ESCENT L EARNING IN THE I NFORMATION P LANE
Anonymous authors
Paper under double-blind review
A BSTRACT
Studying the evolution of information theoretic quantities during Stochastic Gradient Descent (SGD) learning of Artiﬁcial Neural Networks (ANNs) has gained popularity in recent years.
Nevertheless, these type of experiments require estimating mutual information and entropy which becomes intractable for moderately large problems.
In this work we propose a framework for understanding SGD learning in the information plane which consists of observing entropy and conditional entropy of the output labels of ANNs.
Through experimental results and theoretical justiﬁcations it is shown that, under some assumptions, the SGD learning trajectories appear to be similar for different ANN architectures. First, the SGD learning is modeled as a Hidden Markov Process (HMP) whose entropy tends to increase to the maximum.
Then, it is shown that the SGD learning trajectory appears to move close to the shortest path between the initial and ﬁnal joint distributions in the space of probability measures equipped with the total variation metric. Furthermore, it is shown that the trajectory of learning in the information plane can provide an alternative for observing the learning process, with potentially richer information about the learning than the trajectories in training and test error.
I NTRODUCTION
How do information theoretic quantities behave during the training of ANNs?
This question was addressed by Shwartz-Ziv & Tishby (2017) in an attempt to explain the learning through the lens of the information bottleneck method (Tishby et al., 1999).
In that work, the layers of an ANNs are considered random variables forming a Markov chain.
The authors constructed a 2D information plane by estimating the mutual information values between hidden layers, inputs, and outputs of ANNs.
Using this approach it was observed that the information bottleneck method provides an approximate explanation for SGD learning.
In addition, their experiments showed the role of compression in learning.
That initial paper motivated further work on this line of research (Saxe et al., 2018; Gabri´e et al., 2018).
The main practical limitation of that type of experiments is that it requires estimating mutual information between high dimensional continuous random variables.
This becomes prohibitive as soon we move to moderately large problems, such as the CIFARdataset, where the large ANNs are employed.
Other works dealing with information theoretic quantities tend to have these experimental limitations.
For instance, Russo & Zou (2015); Xu & Raginsky (2017); Asadi et al.
(2018) used generic chaining techniques to show that generalization error can be upper bounded by the mutual information between the training dataset and output of the learning algorithm.
Nevertheless, estimating that mutual information to verify those results experimentally becomes intractable.
Furthermore, in our previous work (Anonymous, 2018) we deﬁned a novel 2D information plane that only requires to estimate information theoretic quantities between the correct and estimated labels.
Since these random variables are discrete and one-dimensional, this framework can be used to study learning in large recognition problems as well.
Moreover, that work provides a preliminary empirical study on the behavior of those information theoretic quantities during learning along with some connections between error and conditional entropy.
In this work, we extend the experiments from Anonymous (2018) to more general scenarios and aim to characterize the observed behavior of SGD.
Our main contributions are as follows:
• We deﬁne a 2D-information plane, inspired by the works of Shwartz-Ziv & Tishby (2017), and use it to study the behavior of ANNs during SGD learning.
The main quantities are entropy of the output labels and its conditional entropy given true labels.
• It is shown that if the learning is done perfectly and under some other mild assumptions, the entropy tends to increase to its maximum. • It is additionally shown that SGD learning trajectory follows approximately the shortest path in the space of probability measures equipped with the total variation metric.
The shortest path is characterized well by a Markov chain deﬁned on probabilities of estimate labels conditioned on true labels. To that end we provide theoretical and experimental justiﬁcations for constructing a simple Markovian model for learning, and compare it with SGD through experiments. These experiments are conducted using various datasets such as MNIST (LeCun et al., 1998), CIFAR-10/ CIFAR-100, spirals (Anonymous, 2018), as well as different ANN architectures like Fully Connected Neural Networks (FCNNs), LeNet- (LeCun et al., 1999), and DenseNet (Huang et al., 2017).
• The trajectory, however, is not universal. Through a set of experiments, it is shown that SGD learning trajectory differs signiﬁcantly for different learning strategies, noisy labels, overﬁtting, and underﬁtting.
We show examples where this type of trajectories provide a richer view of the learning process than conventional training and test error, which allows us to spot undesired effects such as overﬁtting and underﬁtting. The paper is organized as follows: Section 2 introduces the notation as well as elementary notions from information theory.
Section 3 formulates learning as a trajectory on the space of probability measures, deﬁnes the notion of shortest learning path, and provides a connection to Markov chains.
Section 4 constructs a simple Markov chain model for gradient based learning that moves along the shortest learning path.
Finally, Section 5 performs an empirical evaluation of the proposed model.
S YSTEM M ODEL
Let x ∈
X be a random vector belonging to some set X of possible inputs.
We assume that there exists a function, known as “oracle”, that maps x to one of K ∈ N classes.
Formally, there exists a deterministic mapping c : X → Y where Y = { 0 , . .
, K − 1 } is the set of possible classes.
Then, let
˜ y ∈ Y denote the random variable ˜ y
=
c ( x ) .
One common assumption, that is present in popular datasets such as MNIST, CIFAR-10, CIFAR-100, and Imagenet, is that ˜ y is uniformly distributed.
We assume this to be true throughout this paper.
Note that the designer of the dataset has control over the marginal distribution of ˜ y.
We model the effect of having error-prone labels, denoted by the random variable y, in the data by introducing discrete independent random noise z ∈ Y to ˜ y in the form of modulo addition 1 , that is y = ˜
y ⊕ z ∈ Y .
Let θ ∈ Θ be the vector, possibly random, containing of all tunable parameters in the hypothesis space Θ .
Then a classiﬁer is a deterministic function g : Θ ×
X
→ Y that aims to approximate c .
Further, ˆ y =
g ( θ, x ) is deﬁned to be the random variable of the label predicted by the classiﬁer g ( θ, · ) .
Using this notation we deﬁne the three types of error: the dataset error p
=
P ( y (cid:54) =
˜ y ) , the test error ε =
P (ˆ y (cid:54) =
y ) , the true error ˜ ε
=
P (ˆ y (cid:54) =
˜ y ) .
A summary of this system model is provided in Figure 1.
We shortly review some elementary concepts from information theory such as entropy and mutual information.
The entropy of a discrete random variable y ∈ Y is deﬁned as
[Equation]
The entropy is bounded by 0 ≤ H ( y ) ≤ log | Y | and it measures the amount uncertainty present in y. Similarly, the conditional entropy between two random variables y and ˆ y is
[Equation]
and it quantiﬁes the uncertainty about y given that ˆ y is known.
Finally, the mutual information
I ( y ; ˆ y ) between y and ˆ y measures how much information does one random variable carry about the [Figure] other.
It may be deﬁned in terms of entropies as
[Equation]
Moreover, the following proposition is a well-known result from information theory, known as
Fano’s inequality, that relates test error ε and conditional entropy.
Proposition 1.
Fano’s Inequality (Csisz`ar & K¨orner (2011), Lemma 3.8)
The value of H ( y | ˆ y ) and H (ˆ y | y ) is upper bounded by a function of the expected error as
[Equation]
, where the function Ψ : [0 , 1] → R is deﬁned as
[Equation]
and h b ( x ) = − x log( x ) − (1 − x ) log(1 − x ) is the binary entropy function.
This results provides an upper bound on conditional entropy in terms of ε , that is known to be sharp.
In the works of Feder & Merhav (1994) it has been shown that I ( y ; ˆ y ) gives an upper and lower bound on the minimal error 3 between y and ˆ y.
In addition, the minimal error is minimized when
I ( y ; ˆ y ) reaches its maximum.
Therefore, learning can be modeled as ﬁnding θ such that I ( y ; ˆ y ) is maximized.
This can be written in terms of entropies as
[Equation]
[Equation] As in our previous work (Anonymous, 2018) we are interested on characterizing the trajectory in the 2D information plane, composed by H ( ˆ y ) and H ( ˆ y | y ) , during the learning process of artiﬁcial neural networks (ANNs).
In Figure 2 we observe that learning trajectory for the DenseNet architecture of 100 layers as it learns to classify data from the CIFAR-100 dataset, for p = 0 .
Intuitively, when solving equation 1, maximizing H (ˆ y ) is more related with the unsupervised component of learning since it does not depend y.
On the other hand, keeping H (ˆ y | y ) low while H (ˆ y ) increases can be seen as the supervised component of equation 1.
From this point of view it would be interesting to characterize the inﬂection point from which H (ˆ y | y ) starts decreasing, since it allows us to observe at which point SGD starts paying more attention to assigning labels correctly than to learn about the distribution of the input.
One also may wonder if this increasing-decreasing trajectory is an accidental result for that occurs only on this particular experimental setup, or if it is a fundamental property of SGD.
In Anonymous (2018) we showed that this behavior seems to appear regardless of the activation function employed (see Appendix C) on the spirals and MNIST dataset.
Moreover, in further sections we provide a justiﬁcation for this type of trajectory and show that it remains in other datasets.
As z is independent, minimizing ε amounts to g ( θ, · ) learning c ( · ) , regardless of the value of p (Angluin & Laird, 1988).
For more information about error and entropy relations in the presence of noisy labels see Appendix A.
6 H (ˆy)
H ( ˆ y | y ) p=0.0Theoretical
T r a i n i n g S t e p s ( % ) 0 .0 .0 .0 .0 .1 .
ε
H ( ˆ y | y ) p=0.0Theoretical
T r a i n i n g S t e p s
[Figure]
[Figure]
C ONNECTION B ETWEEN SGD AND M ARKOV C HAINS
In gradient based training of ANNs, the tunable parameters of the networks θ are changed in time by the gradient updates of the loss function, in order to minimize the learning error for a particular problem at hand.
Let θ n ∈ Θ denote the tunable parameters of an ANN after n ≥ 1 training steps of SGD.
The parameters are initialized as θ 0 , which can be random or deterministic.
The set Θ can be seen as a high dimensional Euclidean space with the network parameter θ as its vector.
At the training step n , the outcome of the learning algorithm is captured by the random variable ˆ y n which is modulated by the network function g ( θ n , · ) applied to the random input data x , i.e., ˆ y n = g ( θ n , x ) .
Therefore the SGD learning gives rise the following sequence of random variables
[Equation]
As n grows large with a successful training, the sequence of random variables converges approximately to the true labels ˜
y which itself follows a joint distribution with x .
Note that the above random variables are coupled through the common random variable x
and the sequence of parameter updates θ n .
If the probability distribution of ˆ y n is denoted by p n , a ﬁrst question is to see how SGD methods modify p n on the space of probability measures deﬁned on Y .
As a consequence, one can determine the trajectory of H (ˆ y n ) , which will be plotted later.
However it is additionally important in learning that the random variable ˆ y n approximates the true labels.
Therefore, a second question would be how SGD methods change the joint distribution of (ˆ y n , ˜ y n ) .
The answer could determine instead the trajectory of the conditional entropy H (ˆ y n | ˜ y n ) .
We ﬁrst study the trajectory of H (ˆ y n ) .
The random variables ˆ y n are deﬁned as g ( θ n , x ) .
Consider the sequence of random variables { θ n } .
Let T denote the set of training samples ( x , y ) that are obtained prior to training and independently.
In addition, let T n be a subset of T that is used at the step n for SGD update.
T n is assumed to be independent from ( θ 0 , . .
, θ n − 1 )
and it is either deterministic and known all n or randomly chosen at each step.
These variations correspond to the variants of SGD.
In pure gradient based methods without momentum based techniques, the network parameters obey the following recursive relation
[Equation]
where f denotes the update rule of SGD.
The model assumes that the SGD updates only depend on the parameters in the last step and the training set used in the current iteration.
We can assume that T n are i.i.d.
random variables if we neglect the effect of reusing training data in different batches.
The ﬁrst conclusion is that the sequence of random variables { θ n } is a Markov chain.
Proposition 2.
The sequence of random variables { θ n } deﬁned as equation 2 with i.i.d.
random variables T n is a Markov Chain.
Proof.
If T n ’s are i.i.d.
random variables, the proof follows directly from Serfozo (2009, Proposition 11) on equation 2.
The transition probability of this Markov chain can be obtained only from f and T 1 .
In that sense, the random process { θ n } is a homogeneous Markov chain.
Throughout this work, it is assumed that the Markov chain { θ n } has a stationary distribution which corresponds to the learned ANN.
This proposition shows that SGD updates induce Markov property for weights of an ANN.
The sequence of random variables ˆ y n = g ( θ n , x ) however is in general not a Markov chain, particularly because they are coupled through a common random variable x .
Since we are interested in H (ˆ y n ) and the distribution p n , these random variables can be decoupled by considering the random variables g ( θ n , x n ) where x n are i.i.d.
random variables with the same distribution as x .
Note that the value of the entropy function remains unchanged after decoupling, namely H (ˆ y n )
=
H ( g ( θ n , x n )) .
The new sequence is a function of a Markov chain and i.i.d.
random variables.
The question whether the resulting sequence is a Markov chain has been addressed in Spreij (2001);
Gurvits & Ledoux (2005) showing that ˆ y n is not a Markov chain in general unless certain conditions are met by the function g ( · ) .
Unfortunately the function g ( · ) is not injective and a non-injective function of a
Markov chain is not Markov chain in general.
However the random variable g ( θ n , x n ) can be seen as the observation of the Markov process { θ n } through a noisy memoryless channel g ( · , x n ) .
Therefore the random process { g ( θ n , x n ) } is a HMP.
See Ephraim & Merhav (2002) for an information theoretic survey.
If the learning is done perfectly, the HMP { g ( θ n , x n ) } converges to the uniform distribution of correct labels.
Since the random variables are discrete, entropy is a continuous function of the distribution p n .
Therefore as the correct labels are uniformly distributed, the entropy H (ˆ y n ) approaches its maximum log K .
The instantaneous entropy H (ˆ y n ) would converge monotonically to the entropy of the stationary distribution log K if the sequence were to be a Markov chain.
This could explain the monotonicity of H (ˆ y n ) in the experiments.
Although the sequence not a Markov chain but it is indeed a HMP, the following proposition shows that the entropy is lower-bounded by an increasing function.
Proposition 3.
Suppose that the Markov process { θ n } with the probability distribution q n has a stationary distribution q .
We have
[Equation]
The proof follows from data processing inequality for Kullback-Leibler divergence and is found in
Appendix B. Note that since { θ n } is a Markov process, D ( q n (cid:107) q ) is non-increasing.
SGD ON J OINT P ROBABILITY M EASURES
In the previous section, the non-decreasing property of H (ˆ y n ) was investigated by modeling the network output as an HMP. In the same spirit let us deﬁne ˜ y n = c ( x n ) to be i.i.d.
realizations of ˜ y. In the ideal situation where the network manages to learn successfully the true labels, we can say ˆ y n converges to ˜ y n almost surely 4 .
The joint distribution of (ˆ y n , ˜ y n ) speciﬁes a point on the space of probability measures on Y × Y .
The task of learning consists tuning the parameters θ n in a way that the joint distribution approaches the distribution of (˜ y n , ˜ y n ) .
Therefore the gradient descent steps corresponds to a sequence of points, that is joint distributions of (ˆ y n , ˜ y n ) , on the space of probability measures on Y × Y with the end point ideally being the joint distribution of (˜ y n , ˜ y n ) .
In this section, we investigate the gradient descent algorithm by exploring the path it takes on the space of probability measures on Y × Y 5 .
The trajectory of conditional entropy H (ˆ y n | ˜ y n ) is determined for the trajectory of probability distributions on the space of joint measures.
However it is in general difﬁcult to precisely characterize this path.
Instead, one might ask how the gradient descent trajectory compares with a certain natural path on the space of distributions. A
[Figure] relevant question is to ask what the shortest path between these probability measures is on this space and how similar is the trajectory of SGD compared to this shortest path.
To be able to formally address this issue we require to deﬁne curves and lengths on the metric space of probability measures.
The space of probability distributions deﬁned on the discrete space Y × Y , denoted by P ( Y × Y ) , with total variation metric d TV ( · , · ) is a simplex in a ﬁnite dimensional Euclidean space.
The total variation metric is equivalent to the L 1 -distance between the points in the corresponding Euclidean space.
A curve in this space is deﬁned by a continuous function σ : [0 , 1] → P ( Y × Y ) .
The curve is called a shortest path if it has minimal length among all curves with endpoints σ (0) and σ (1) .
Note that the length is measured in this space using L 1 -norm.
The following theorem guarantees that there is a shortest path on this space between probability measures and it can be traveled using a Markov chain.
Theorem 1.
The shortest path between two probability measures µ and ν on the space of discrete probability measures P ( Y × Y ) is given by tµ + (1 − t ) ν for t ∈ [0 , 1] .
Furthermore if the probability measures are represented by row vectors there is a transition matrix Π with the stationary distribution ν such that µ n = µ Π n is on the line segment between µ and ν and lim n →∞ µ n = ν .
Proof.
Not that the space of probability distributions here is a bounded compact metric space with each two points connected by a rectiﬁable curve.
The existence of shortest path follows from (Burago et al., 2001, Corollary 2.5.20,Theorem 2.5.23).
The transition matrix in Theorem 1 is given simply by
[Equation]
+ α 1 T ν where 1 = (1 , 1 , . . .
To see the implication of previous theorem more precisely, consider conditional distributions
P (ˆ y n ∈ ·| ˜ y n = l ) and let ˜ p l ( n ) ∈ R K be the following vector for l ∈ Y , that is ˜ p l ( n ) =
( P (ˆ y n = 0 | ˜ y n = l ) , P (ˆ y n = 1 | ˜ y n = l ) , · · · , P (ˆ y n = K − 1 | ˜ y n = l )) .
We use the compact notation ˜ P ( n )
∈ R K × K for the matrix with ˜ p 0 ( n ) , . . .
˜ p K − 1 ( n ) as rows.
Note that, with the assumption that ˜ y n is uniformly distributed, the joint distribution of (ˆ y n , ˜ y n ) is fully determined by ˜ P ( n ) since P (ˆ y , ˜ y n ) = 1 K P (ˆ y | ˜ y n ) .
Suppose that the initialization θ 0 is such that g ( θ 0 , · ) initially maps all inputs to the same class (the ﬁrst class is assumed for simplicity).
Therefore the initial distribution matrix ˜ P (0) assumes no knowledge about the input and is given by ˜ P (
[Equation]
= [Equation] Ideally, in a learning algorithm, the matrix ˜ P (0) converges to an optimal distribution ˜
P ∗ as n → ∞ , that is ˜ P ∗ =
lim n →∞ ˜ P ( n ) .
If P (ˆ y n = ˜ y n ) = 1 , we have ˜ P ∗
= I .
Now that we set the initial distribution and stationary distributions, the following transition matrix provides a way to pass from the initial distribution to the stationary one.
Deﬁnition 1.
α - Simple Markov Learning Chain ( α -SMLC)
Given 0 < α < 1 , the sequence of random pairs { (ˆ y n , ˜ y n ) } is an α -SMLC if ˜ p l ( n ) =
˜
p l ( n − 1) Π l , for every n ≥ 0 , l ∈ Y with [Equation] =
[Equation]
The above construction provides a different transition matrix for each ˜ p l ( n ) depending on l .
The following theorem describes how an α -SMLC moves ˜ P ( n ) in the space of stochastic matrices.
It actually shows this construction leads to points on the shortest path between the measures.
Theorem 2.
If (ˆ y n , ˜ y n ) is the n -th random pair generated of an α -SMLC, then ˜ P ( n ) = (1 − t ) ˜ P (0) + t I , with
[Equation]
Proof.
c.f. Appendix B
This shows that ˜ P ( n ) belongs to the continuous curve from equation 3, regardless of the choice of α .
Moreover, let ˆ y ( t ) be a version of ˆ y n parametrized by t ∈ [0 , 1] such that its conditional distribution with ˜ y n corresponds to ˜ P ( n ) =
(1 − t )
˜ P (0)
+ t I .
Proposition 4.
If { (ˆ y n , ˜ y n ) } is an α -SMLC and z follows the distribution
[Equation]
then
[Equation]
Proof.
c.f. Appendix B
Corollary 1.
In the setting of Proposition 4, if p → 0 then H (ˆ y ( t ) | y ) has one maximum at t = 12 ,
H (ˆ y ( 12 ) | y )
=
K − 1 K log 2 .
This result allows us to characterize the shape of the 2D curves ( H (ˆ y ( t )) , H (ˆ y ( t ) | y )) for the above construction.
We now consider the implication of Markov assumption for error.
Deﬁne ˆ y ln ∈ Y for all l ∈ Y to be random variables distributed according to the conditional probabilities P (ˆ y ln = k )
=
P ( g ( θ n , x ) = k | ˜ y n = l ) for all k ∈ Y .
Note that, since ˜ y n is uniformly distributed, one can compute the joint distribution of (ˆ y n , ˜ y n ) from the marginal distributions of ˆ y 0 n , . . .
, ˆ y K − 1 n and vice-versa.
The following propositions shows that the Markov trajectory implies the reduction in error as well. [Figure]
2(b), with α = 0 . 85 .
The shape of the makers differentiate between different values of p .
Proposition 5.
If { ˆ y ln } is a stationary Markov Chain converging to the distribution ˜ p ∗ =
e l then
[Equation]
Corollary 2.
If { ˆ y ln } is a stationary Markov Chain converging to the distribution ˜ p ∗ =
e l for all l ∈ Y then
[Equation]
This last results shows that ˜ ε is non-increasing with n (i.e., no over-ﬁtting), which is a desirable property for any learning algorithm.
We will show through numerical simulations how a comparable behavior is observed for gradient descent methods.
E XPERIMENTS
In this section we compare gradient based learning to the α -SMLC model through empirical simulations.
First, we use the datasets where SGD is extremely successful at the classiﬁcation task ( ˜ ε ≈ 0 ), that is the MNIST dataset and the spirals dataset (Anonymous, 2018).
The spirals dataset constitutes a 2D-spiral classiﬁcation task constructed as
[Equation]
where a ∈ [0 , 1] , b ∈ [0 , 0 . 1] and ˜ y ∈ { 0 , 1 , . .
, K − 1 } are independent uniformly distributed random variables and K = 3 .
This dataset is divided into a training set of 50 000 samples and a test set of 2 000 .
Furthermore, we train the FCNN of Anonymous (2018) for the spirals dataset and use LetNet-5 (LeCun et al., 1999) for MNIST, achieving an average accuracy above 99% in both cases.
For the sake of completeness we perform more experiments on the CIFAR-10 and CIFAR100 datasets using the DenseNet architecture from Huang et al.
(2017) with 40 and 100 layers respectively.
FCNN and LetNet-5 are trained using Adam’s optimizer, while DenseNet is trained with SGD.
More detailed explanation about the experimental setup is provided in Appendix C. We estimate use a naive estimator of entropy which consists on computing the empirical distribution of {+S p i r a l s 0 .0 .0 .0 . 6 ε 0 .0 .0 .
 H ( ˆ+} y | y ) λ
=0 .
λ =0 . 00059 λ =0 .λ
=0 .
λ
=0 .
0009 λ
=0 .λ
=0 .
00119 λ =0 .
{+0013 Theoretical T r a i n i n g S t e p s ( % ) 0 .0 .0 .0 . 6 ε 0 .0 .0 .
 H ( ˆ y | y ) p=0.1p=0.2p=0.+}
Theoretical
T r a i n i n g S t e p s ( % )
[Figure] ( ˆ y , y ) and directly calculating entropy afterwards. This method is known to have an approximation error of K 2 /N (Miller, 1955), which is good enough for our experiments since N is much larger than K 2 in our datasets.
For cases with larger K one could use more sophisticated methods, such as
Sch ¨ urmann (2004); Archer et al.
For our experiments we introduce i.i.d.
noise to the dataset labels before training according to P ( z = 0) =
1 − p and P ( z (cid:54) = 0)
=
p/ ( K − 1) .
We keep ﬁxed α = 0 .
85 for all simulations.
In Figure 3 we show the similarity between the α -SMLC model and ANNs on the spirals and MNIST dataset.
This ﬁgure is obtained by averaging the entropy values over 100 realizations of training.
We observe ANNs move along the information plane in a similar way as the α -SMLC model, and converge to the optimal distribution.
In addition, we display the inﬂection point of H ( y | ˆ y ) of the αSMLC model for different values of p .
Interestingly, as labels get noisier the inﬂection point occurs at a larger H (ˆ y ) value.
This seems to be the case for SGD learning as well.
This phenomena suggests that, as labels get noisier, a learning algorithm needs to know more about the input distribution before it can start assigning labels efﬁciently.
Similar conclusions can be drawn form Figure 4, which is also obtained by averaging over 100 realizations of training.
An interesting result is that, regardless of the value of p , a good learning algorithm should converge to a point that lies on the upper bound provided by Fano’s inequality.
These Figures artiﬁcially mitigate the randomness induced by SGD on the trajectory by averaging over several realizations of training 6 .
How much SGD oscillates seems to depend on the experimental setup, such as the learning rate, dataset, and the structure of the classiﬁer.
See Appendix C for examples of highly oscillating trajectories. Our last experiment consists on investigating how underﬁtting and overﬁtting affects the trajectory of SGD. To that end in Figure 5(a) we include an (cid:96) 1 norm regularization term into the loss function (details in Appendix C) controlled by a parameter λ .
As expected for sufﬁciently small λ the minimum error is attained. Interestingly, as we induce underﬁtting by increasing this regularization coefﬁcient the obtained models move away from Fano’s bound. This naturally leads to increased error values.
On the other hand, in Figure 5(b) we increase the number of parameters of FCNN and reduce the dataset size in order to induce overﬁtting (details in Appendix C).
While overﬁtting leads to larger error as well as underﬁtting, it can be distinguished by its trajectory in the entropy-error plane. Underﬁtting seems to push models away from Fano’s bound while overﬁtting happens when an ANN is at the bound. This experiment shows that the information plane provides a richer view beyond train and test error that allow us to observe effects that were previously hidden. Further understanding about desired trajectories is interesting since it may allow practitioners to monitor models during training, spot undesired behaviors, and possibly tune hyperparameters accordingly.
A C OLLECTION OF R ESULTS FOR L EARNING WITH R ANDOM L ABELS
For ε < 1 − 1 K , proposition 1 can be written as a lower bound on ε through the following Proposition of Anonymous (2018).
Proposition 6.
(Anonymous, 2018)
Φ( H ( z )) ≤ ε , where Φ : [0 , log | K | ]
→ [0 , 1 − 1 K ] is the inverse function of Ψ in the interval [0 , 1 − 1 K ] .
Proof.
For independent noise z, we have
[Equation]
By Proposition 1, it follows that H ( z ) ≤ Ψ( ε ) .
Applying the inverse Φ( x ) , x ∈ [0 , 1 − 1 K ] , completes the proof.
In Anonymous (2018) it is shown that this bound is sharp when z is distributed such that Φ( H ( z ))
= p , thus p ≤ ε .
We generalize this result in the following theorem for an arbitrary distribution of z, under some mild conditions, and show that p ≤ ε is in fact a sharp lower bound for arbitrary z.
Lemma 1.
Let z ∈ Y be a random variable with P ( z = 0)
= 1 − p , then
H ( z ) ≤ Ψ( p ) .
Proof.
Let us deﬁne β k (cid:44) P ( z = k ) and the auxiliary random variable ˜ z ∈ { 1 , . .
, K − 1 } with
P (˜ z = k ) =
β k /p for all k = 1 , . . .
, K − 1 .
[Equation]
Ψ
[Equation]
Theorem 3.
If P ( z = 0) = 1 − p and P ( z = 0) > P ( z = k ) for all k = 1 , . .
, K − 1 , then p ≤ ε , and equality is attained if and only if P (ˆ y = ˜
y )
= 1 .
Proof. of Theorem
For the sake of notation let us deﬁne
[Equation]
From Proposition 1 we know that ε ≥ Φ( H ( z )) .
Since Ψ is an increasing function in the interval [0 , 1 − 1 K ] and Φ is its inverse in that interval, we have that Φ is an increasing function as well.
From
Lemma 1 (c.f. Appendix B) we know that H ( z ) ≤ Ψ( p ) , this leads to
[Equation]
Then, if the bound from Proposition 1 were to be sharp, ε could reach values strictly lower than p .
We will show that this is not possible.
[Equation]
If ε < p we obtain (1 − p ) < 1 − ε ≤ (1 − p ) which is a contradiction, hence it must hold that ε ≥ p .
Finally, if ε = p then equation 10 yields
[Equation]
Since β max < (1 − p ) , this inequality holds if and only if δ 0 = 1 , that is P (ˆ y = ˜
y )
= 1 .
This theorem shows that the minimum expected error ε can only be attained if g ( θ, · ) manages to denoise the labels, hence ˜ ε = 0 .
We extend this result by deriving bounds for ˜ ε , given ε and p , in the following theorem.
Theorem 4.
(Angluin & Laird, 1988)
Given ε < 1 − 1 K and p < 12 , if P ( z = 0) = 1 − p , and P ( z = k ) < 12 for all k = 1 , . . .
, K − 1 , then ˜ ε is bounded by
[Equation]
Proof. of Theorem
[Equation] [Equation]
Applying (cid:80) K − 1 k =
δ k β k ≥ 0 in equation 13 leads to 1 − ε ≥ (1 − p )(1 − ˜ ε ) thus [Equation] − p [Equation]
−
[Equation]
−
[Equation]
which completes the proof.
Corollary 3.
If p < 12 , ε < 1 − 1 K , and ε − p 1 − 2 p < 1 − 1 K then
[Equation]
Proof.
Since Ψ is an increasing function in the interval [0 , 1 − 1 K ] , the proof follows from applying
Theorem 4 on Proposition 1.
In information theory there is a result of this kind, known as Mrs. Gerber’s Lemma (MGL), that does not require knowledge about p and ε .
MGL provides an upper bound on H ( ˜ y | ˆ y ) given H ( y | ˆ y ) for the case of K = 2 .
This result also states that the minimum H ( y | ˆ y ) is attained when H (˜ y | ˆ y )
= 0 .
Since we assumed ˜ y to be uniformly distributed, this corresponds to ˜ ε
= 0 , up to permutation ambiguities.
Generalizing MGL for arbitrary K is still an open question in information theory.
Nevertheless,
Jog & Anantharam (2014) successfully proved MGL for the cases where K is a power of 2 .
We summarize that result in the following proposition.
Proposition 7.
Generalized Mrs. Gerber’s Lemma for K = 2 n (Jog & Anantharam, 2014)
[Equation]
with
[Equation]
otherwise where
[Equation]
and k is an arbitrary positive integer and a (cid:63) b (cid:44) a (1 − b )
+ b (1 − a ) .
We have derived inequalities that relate entropies and error values.
Then we showed that in the presence of corrupted labels, the best g ( θ, · ) can do for minimizing ε is to learn c , regardless of the value of p .
B D EFERRED P ROOFS
Proof of Proposition 3: The proof follows from the following general theorem.
Theorem 5.
Let { y n } be an HMP deﬁned as the observation of a Markov process { x n } through an arbitrary stationary memoryless channel with values in the state space Y .
Suppose that the probability distributions on the respective state spaces of { x n } and { y n } are given by { q n } and { p n } with the stationary distribution q and p .
Then
[Equation]
Proof.
Based on the assumption above, x n and y n are related according to the conditional distribution characterized by the conditional probabilities { r ( ·| x ) : x ∈ X } .
Denote the joint distribution of x n and y n by µ n deﬁned on X × Y and given by q n ( x ) × r ( ·| x ) .
The stationary joint distribution µ is deﬁned by q × r .
Using the chain rule (Cover & Thomas, 2006, Theorem 2.5.3), we have:
[Equation]
On the other hand, the chain rule and the non-negativity of Kullback-Leibler divergence shows that:
[Equation]
which implies the theorem.
[Figure]
p=0.3UBLB
T r a i n i n g S t e p s ( % )
Using the fact that the stationary distribution of ˆ y n is equal to the uniform distribution, we have:
[Equation]
This fact along with Theorem 5 proves Proposition 3.
Proof. of Theorem
Since 1 e T l 1 e T l = 1 e T l we get
[Equation]
l
[Equation]
By induction we get ( Π l ) n = 1 e T l + α n ( I − 1 e T l ) =
(1 − α n ) 1 e T l + α n I .
Replacing t =
(1 − α n ) ∈ [0 , 1] yields ( Π l ) n = t 1 e T l + (1 − t ) I . Finally, ˜ p
[Equation]
n [Equation]
=
[Equation]
Π l ) n = ˜
p [Equation] [Equation] [Equation]
=
[Equation]
[Equation]
t
[Equation]
p l
[Equation] + t e l , thus ˜ P [Equation]
n
[Equation]
=
[Equation]
˜ [Equation] n [Equation]
, ˜ p
[Equation]
= [Equation] 1 − t [Equation]
˜ P
[Equation]
(0) + ˜ t I . Proof. of Proposition Since y = ˜ y ⊕ z the distribution of y can be expressed as a circular convolution between the distributions of z and ˜ y. Informally, we express this as P ( y ) =
P (˜ y ⊕ z ) =
P (˜ y ) (cid:126) P ( z ) . The same holds true if we condition by ˆ y, that is P ( y | ˆ y = l ) =
P (˜ y ⊕ z | ˆ y = l ) = P (˜ y
| ˆ y = l ) (cid:126) P ( z ) for all l = 0 , . .
, K − 1 .
Then for uniformly distributed y, and z distributed according to equation 4, we can express this relation in matrix form as
[Equation]
Therefore, the ( i, j ) -th element of P ( t ) is denoted as P i,j ( t ) and given by
[Equation]
Differentiating P i,j ( t ) with respect to t yields
[Equation]
For particular choices of i, j these expressions boil down to P i,j ( t ) =
[Equation]
and ∂P
[Equation]
Finally, we make use of these to derive a closed form expression for H (ˆ y ( t ) | y ) ∂t , that is
H (ˆ y ( t ) | y ) ∂t = ∂ ∂t   −
K (cid:88)
j =
K
K (cid:88) i =
P i,j ( t ) log P
i,j ( t ) 
 = − 1 K
K (cid:88)
j =
K (cid:88) i = (cid:20) ∂P
i,j ( t ) ∂t (1 + log P
i,j ( t )) (cid:21) = − 1 K 1 (cid:88)
j = (cid:88) i
= j (cid:20) ∂P i,j ( t ) ∂t (1 + log P
i,j ( t )) (cid:21) −
K 1 (cid:88)
j = (cid:88) i (cid:54) =
j (cid:20) ∂P i,j ( t ) ∂t (1 + log P
i,j ( t )) (cid:21) −
K
K (cid:88)
j = (cid:88) i
= j (cid:20) ∂P i,j ( t ) ∂t (1 + log P
i,j ( t )) (cid:21) −
K
K (cid:88)
j = (cid:88) i (cid:54) =
j (cid:20) ∂P i,j ( t ) ∂t (1 + log P
i,j ( t )) (cid:21) =
K p log(1 − tp )
+ 1 K
( K − 1)( p K − 1 − 1)(1 + log(1 − t + t p
K − 1 )) −
K ( K − 1)(1 − p )(1 + log( t (1 − p ))) − 1 K
( K − 1) p (1 + log( t p K − 1)) K (cid:18) p − ( K − 1)(
[Equation]
Remark :
For p → 0 we now that the maximum the curve is at t = 12 since [Table] [Equation] [Table]
[Equation]
[Table]
[Table]
C E XPERIMENTAL D ETAILS AND C OMPLEMENTARY E XPERIMENTS
A fully connected ANN with four hidden layers of ﬁve neurons each, as FCNN, is trained on the spirals dataset.
For the MNIST dataset, the popular convolutional network called LeNet- 5 LeCun et al.
(1999) is used.
To train these networks we let the learning rate γ ∈ R start at a given γ max ∈ R and then decay by 40% per epoch until reaching some given minimum learning rate γ min < γ max , that is γ = max { γ max 0 .
6 (cid:98) epoch (cid:99) , γ min } .
For the CIFAR-10 dataset we train a 100 layer DenseNet architecture as done in Huang et al.
but we stop the training after 10 epochs instead of the original 300 used by the authors.
The different conﬁgurations used for these experiments are summarized in Table 1.
For instance, Figure 2 shows 1 realization of SGD training for DenseNet on CIFAR-100. In that ﬁgure we observe a rather stable trajectory, with not much oscillation.
However, in Figure 8 we average over 2 realizations of SGD learning for DenseNet on CIFAR-10 and obtain highly oscillating trajectories.
As expected, both trajectories follow a similar behavior as the αSMLC model. In Figure 5(a) the same model as in Figure 3 is used but an additional regularization term is added to the loss function, that is λ (cid:107) w (cid:107) 1 where w is a vector containing all the weights in the network. In Figure 5(b) a FCNN with a single hidden layer of size 100 , and t anh activations, is used and the dataset sizes are reduced according to 3. Other parameters are γ max = 10 − 1 , γ max = 10 − 2 , and the number of epochs is 10 000 .
[Table]
[Figure] [Figure]