I NFORMATION R EGULARIZED N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
We formulate an information-based optimization problem for supervised classiﬁcation.
For invertible neural networks, the control of these information terms is passed down to the latent features and parameter matrix in the last fully connected layer, given that mutual information is invariant under invertible map.
We propose an objective function and prove that it solves the optimization problem.
Our framework allows us to learn latent features in a more interpretable form while improving the classiﬁcation performance.
We perform extensive quantitative and qualitative experiments in comparison with the existing state-of-the-art classiﬁcation models.
I NTRODUCTION
Quantities of information are nonlinear measures capable of describing complex relationship between unstructured data and they form the basis of the probabilistic algorithms in the literature of machine learning.
Information theoretic methods are also reported to be effective on improving deep generative models (Chen et al.
Kim & Mnih (2018)) and deep learning models for classiﬁcation (Grandvalet & Bengio (2004); Pereyra et al.
Information Bottleneck (IB) problem (Tishby et al., 1999) is formulated as:
[Equation]
where the solution random variable T is interpreted as a minimal sufﬁcient representation of signal
X for label Y and the mutual information is deﬁned as
[Equation]
The term I( X ; T ) has its origins in Lossy Compression and Rate-Distortion Theory (Cover &
Thomas, 2006), conveying an simple idea of ”keep only what is relevant”.
However, Saxe et al.
(2018) argued that the mutual information I( X ; T ) between signal X and feature T in intermediate layer is inﬁnite, as the transformation from X to continuous random variable
T is deterministic.
In addition they showed experimentally that layers equipped with ReLU actually do not compress too much information, which is supported by many recent work on the invertibility of the neural network (Dosovitskiy & Brox (2015); Jacobsen et al.
This motivates us to consider a different problem with similar principle idea: we would like to establish a theoretically valid objective that allows the neural network to extract only the relevant information for classiﬁcation from the data.
We focus on the discrete prediction
random variable (cid:98) Y inferred by the probabilistic model P ( (cid:98) Y | X ) and introduce the following information optimization problem for supervised classiﬁcation:
[Equation]
The intuition behind this objective lies in twofold:
Information perspective : A good classiﬁcation model should be robust against irrelevant features of X , and prevent over-ﬁtting in the learning process.
In optimization problem (3) we maximize the relevant information
I
( Y ; (cid:98) Y ) , while constraining the irrelevant information
I( X ; (cid:98) Y ) − I
( Y ; (cid:98) Y ) that X has about (cid:98) Y .
Although I( X ; (cid:98) Y ) − I( Y ; (cid:98) Y ) converges to zero as I( Y ; (cid:98) Y ) approaching its maximum (see Figure 1L), in practice it’s never attained due to the limited capacity of the models or over-ﬁtting.
Our proposed constrain addresses the problem of over-ﬁtting: if two models achieve the similar classiﬁcation accuracy, this constraint prefers the one that does not overﬁt to spurious factors of variation in X (e.g., pixel-level artifact/noise in the image that accidentally correlates to the labels in the training data).
Prediction conﬁdence perspective : A good classiﬁcation model should not be certain about its decision which is in fact wrong.
However, modern neural networks are too conﬁdent in their predictions (Guo et al., 2017; Szegedy et al., 2015; Pereyra et al., 2017).
To be more precise, high capacity neural networks mostly assign labels of data with prediction conﬁdence near 0 or 1.
In particular, they assign 0 probability to some correct labels and therefore do not have enough ﬂexibility to correct themselves from making the wrong prediction.
We propose to compress the irrelevant information I( X ; (cid:98) Y ) − I
( Y ; (cid:98) Y ) , where minimizing I( X ; (cid:98) Y ) decreases the conﬁdence on all predictions but maximizing I( Y ; (cid:98) Y ) increases the conﬁdence on the correct predictions.
Therefore the overall effect reduces the certainty on the false prediction of (cid:98) Y (see Figure 1L).
To solve this optimization problem, we ﬁrst present some insights on the dynamics of deep neural network, which can be decomposed into two stages: (i) Transformation stage: samples { X k } k =1: n of the high dimensional unstructured signal X are transformed under the deep invertible (information preserving) feature map F to become (almost) linearly separable; (ii) Classiﬁcation stage: the weight matrix w in the last fully connected layer together with the Softmax function, takes structured features { F ( X k ) } k =1: n as input and gives predictions.
Invertibility of F allows neural networks to pass the control of I( X ; (cid:98) Y )
=
I( F ( X ); (cid:98) Y ) towards
F ( X ) and w in the last layer, where F ( X ) can be interpreted as transformed signal that preserves information about the original signal X and the inference model becomes conceptually linear with classiﬁer w (see Figure 1R).
In Section 2 we derive objective function (7) and prove that it solves the optimization problem (3).
We show the classiﬁcation performance is improved in Section 4.
and the features F ( X ) are sculpted into a form with more interpretability entry-wise in Section 4.2.
Our optimization problem explicitly prohibits the growth of grey area throughout the training.
(R)
Logic chart of our formulation: our proposed optimization problem only involves F ( X ) and w , allowing us to have control over the latent feature F ( X ) directly.
The invertibility property has been empirically demonstrated for complex non-linear deep neural networks that are widely used in practice.
We will discuss the literature in Section 3.
In addition, we prove in Proposition C.1 that a lower bound of classiﬁcation error is minimized if neural network is invertible.
Our contribution: Our contribution lies in the following: (i) we formulated a novel information optimization problem for supervised classiﬁcation; (ii) we propose a simple objective function that improves supervised deep learning with better performance and interpretability; (iii) we formally justify the use of (cid:96) 1 , (cid:96) 2 regularization from an information perspective.
Different from the naive regularization on w , our regularization on w T F ( X ) is novel and effective.
M AIN R ESULTS
Consider a classiﬁcation problem where the training data D = { ( x k , y k ) } k =1: n are sampled from random variable pair ( X, Y ) with unknown joint distribution.
Each x k is fed into a deep probabilistic model, which outputs probability densities and predicts (cid:98) y k , a realization of the prediction random variable (cid:98) Y .
Let C denote the label class and X denote the signal space, then the mutual information between random variables, e.g., continuous X and discrete (cid:98) Y , is deﬁned as
I( X ; (cid:98) Y ) =
(cid:80) (cid:98) y ∈C (cid:82) X p ( x, (cid:98) y ) log (cid:16) p ( x, (cid:98) y ) p ( x ) p ( (cid:98) y ) (cid:17) dx , and the entropy of Y is deﬁned as H ( Y ) =
− (cid:80) y ∈C p ( y ) log p ( y ) .
We ﬁrst call out our assumptions used throughout our analysis.
(I) : we assume the marginal densities of Y, (cid:98) Y are uniform over C ; (II) : there exists a unique true label for every sample of X .
Mutual information is bounded and its gradient with respect to logits is approximately zero over a large domain.
In particular if the logits are initially small for true labels, gradient updates cannot effectively correct them.
Therefore it’s not practical to train mutual information terms directly.
In this section we introduce alternative terms and prove that they are feasible for our purpose.
We show in Proposition 2.2 that I( Y ; (cid:98) Y ) is maximized if the classical cross entropy objective is minimized.
On the other hand, we show in Proposition 2.1 that for invertible F , I( X ; (cid:98) Y ) =
I( F ( X ); (cid:98) Y ) is minimized if (cid:107) w T F ( X ) (cid:107) is minimized.
We derive our objective function (7) in Section 2.2.
Our experimental result in Section 4.1 veriﬁes that the proposed objective function does compress the irrelevant information I( X ; (cid:98) Y ) − I
( Y ; (cid:98) Y ) .
2. S OLVING O PTIMIZATION P ROBLEM WITH F EASIBLE T ERMS
Without loss of generality, we consider the binary classiﬁcation problem, i.e. the label class C = {± 1 } .
To tract the population quantities I( F ( X ); Y ) and I( Y ; (cid:98) Y ) , we decompose each of them into an empirical part and a probabilistic bound, which is negligible if sample size n is large.
In
Proposition 2.1, we show that in order to compress I( F ( X ); Y ) , we need to compress the norm of classiﬁer w and feature F ( X ) .
In particular, smaller | w T F ( X ) | represents lower conﬁdence of the model on its predictions (cid:98) Y , indicating a smaller amount of mutual information
I( F ( X ); (cid:98) Y ) .
The proof is provided in Appendix A.
Proposition 2.1.
I( X ; (cid:98) Y ) =
I( F ( X ); Y ) is well estimated by its empirical version (cid:80) nk =1 (cid:80) (cid:98) y ∈C p ( (cid:98) y
| x k ) log(2 p ( (cid:98) y
| x k ))
/n with high probability, which shares the same unique (global) minimum with (cid:80)
nk =1 | w T F ( x k ) | at w T F ( x k )
= 0 , for all k ∈ { 1 , ..., n } .
Denote the sigmoid function with σ ( a ) = 1 / (1 + e − a ) .
Proposition 2.2 establishes the relationship between maximization over mutual information
I( Y ; (cid:98) Y ) and minimization over cross entropy − (cid:80) nk =1 log σ ( y k w T F ( x k )) ; higher conﬁdence of the model on its correct predictions over the samples indicates a larger value of I( Y ; (cid:98) Y ) .
The proof is provided in Appendix B.
Proposition 2.2.
I( Y ; (cid:98) Y ) is well estimated by its empirical version (cid:80) y (cid:98)
y (cid:98) π y (cid:98)
y log (cid:16) (cid:98) π y (cid:98) y
(cid:98) π y + (cid:98)
π + (cid:98) y (cid:17) with high probability, which shares the same unique (global) maximum with (cid:80) nk =1 log σ ( y k w T F ( x k ))
given that y k w T F ( x k ) > 12 , for all k ∈ { 1 , ..., n } .
Here (cid:98) π y (cid:98) y
=
12 n y (cid:80) n y i =1 σ ( (cid:98) yw T F ( x i )) is an unbiased estimate of p ( y, (cid:98) y ) and (cid:98) π y
+ = (cid:80) (cid:98) y ∈C (cid:98) π y (cid:98) y , (cid:98)
π
+ (cid:98) y
= (cid:80) y ∈C (cid:98) π y (cid:98) y .
2. D ERIVATION OF O BJECTIVE F UNCTION
In Lagrangian form of optimization problem (3), the constant τ can be dropped and the objective becomes
[Equation]
Consider a single signal x k and its true label y k , we propose the following objective function for binary supervised classiﬁcation problem:
[Equation]
where R is some regularizer.
According to results in Section 2.1, minimizing (5) allows us to maximize I( Y ; (cid:98) Y ) while constraining I( X ; (cid:98) Y ) .
We typically choose the hyper-parameter α > 0 to be a reasonably small number.
The intuition comes from the observation that λ/ (1 + λ ) is upper bounded by one.
If we compress I( X ; (cid:98) Y ) harshly, then neural networks may choose to minimize
I( F ( X ); (cid:98) Y ) at a cost of minimizing I( Y ; (cid:98) Y ) .
Recall from the information theoretic perspective of our proposed optimization problem, our regularizer should prefer a model that does not overﬁt among all the ones with high training accuracy.
In this case neural networks assign only large logits w Ty k F ( x k ) to true label y k for each signal x k , and generalization of (5) to multi-class case for I( F ( X ); (cid:98) Y ) can be simpliﬁed to constraining w Ty k F ( x k ) − w Tj F ( x k ) , where w j is the j th column of w , assigning feature F ( x k ) a probability to label j .
We propose to simply constrain w Ty k F ( x k ) and does not encourage increasing w Tj F ( x k ) :
[Equation]
In our experiment, we take the Elastic Net approach by Zou & Hastie (2005) using a combination of (cid:96) 2 and (cid:96) 1 regularizers: we use Holder’s inequality to bound | w Ty k F ( x k ) | with both sup F ( X ) · (cid:107) w y k
(cid:107) 1 and (cid:107)
w y k
(cid:107) 2 · (cid:107) F ( x k ) (cid:107) 2 .
In practice we assume sup F ( X ) to be a constant and is absorbed into the hyper-parameter.
Our proposed objective function is of the form:
[Equation]
Notice that classical training methods maximize cross entropy and therefore I( Y ; (cid:98) Y ) , but do not compress I( X ; (cid:98) Y ) explicitly.
In Equation (7), we explicitly encourage the compression of irrelevant information
I( X ; (cid:98) Y ) − I
( Y ; (cid:98) Y ) .
As we demonstrate in Section 4.3, the proposed objective function does encourage a smaller amount of I( X ; (cid:98) Y )
− I( Y ; (cid:98) Y ) throughout the training process.
R ELATED WORK
Recent experimental work reported that neural networks with invertible structure have better performance.
Dosovitskiy & Brox (2015) showed that images can be resconstructed from the latent features in AlexNet through an inverting process; this reconstruction is further improved by Zhang et al.
(2016), where they built an encoder-decoder structure to encourage invertibility and showed reconstructive objective is beneﬁcial to the performance of the neural network (e.g., VGGNet).
Shang et al.
(2016) proposed an invertible activation scheme named CReLU to preserve information; Gilbert et al.
(2017) analyzed theoretically the invertibility of CNN; Jacobsen et al.
(2018) built a theoretic invertible structure whose performance is comparable to ResNet He et al.
Invertibility seems to be an intriguing property or design principle that often emerges in the recent state-of-the-art deep architectures.
Information theoretic methods are reported to be effective attacking machine learning problems.
In deep learning, IB was ﬁrst introduced in Tishby & Zaslavsky (2015) and the follow-up experimental work Shwartz-Ziv & Tishby (2017).
They argued that DNN structure forms a markov chain and information is compressed layer by layer.
The theoretical breakthrough by Achille & Soatto (2017) established a connection between the IB objective and the generalization in deep learning, carrying along with the notion of sufﬁciency and invariance of representations.
Strouse & Schwab (2016),Slonim & Tishby (1999) used IB objectives for clustering problems.
Alemi et al.
Gao et al.
(2018),Kim & Mnih (2018) established information theoretic approaches based on VAE to encourage disentangled and informative latent representations; Grandvalet & Bengio (2004) introduced entropy regularizer in semi-supervised learning; Krause et al.
(2010) took Regularized
[Table]
are
The ﬁrst table gives the results of naive regularization on w for ResNet-32, with α 1 = 0 , α 2 = [0 . 0005 , 0 . 001 , 0 . 002 , 0 . 003 , 0 . 004] , note that the choices of hyper-parameters are small because it regularize the whole matrix w but not to speciﬁc columns as proposed; the second table is for
ResNet-32, comparing original ResNet and RegResNets with α 1 = 0 , α 2 = [0 . 01 , 0 .
; the third table is for ResNet-28-10-Wide, comparing original ResNet and RegResNets with α 1 = 5 e − 6 , α 2 = [0 . 01 , 0 .
All results are calculated from 10 samples.
The constrain decreases as α 2 increases; the performance can be effectively improved under proper choice of α 2 .
Information Maximization(RIM) approach for classiﬁcation problems; Chen et al.
(2016) proposed to add information ingredients to the objective of GANs, encourage to learn disentangled representations.
Our framework decomposes deep neural network into a composition of nonlinear transformation map and a linear probablistic model; this idea was originated in Bell & Sejnowski (1995) where they considered the blind separation problem and decompose the prediction (cid:98) Y to be the sum of an invertible deterministic part and a stochastic part.
Amjad & Geiger (2018) and Kolchinsky et al.
(2017) also studied the IB problem in a stochastic setting.
Our idea of explicit regularization on w and F ( X ) is related to the margin based and stability based interpretations of generalization in deep learning respectively, studied by Arora et al.
(2018), Bartlett & Mendelson (2003), Neyshabur et al.
(2017), Sun et al.
E XPERIMENTS
In our experiments we build the feature map F with ResNet or InvNet (introduced in Section 4.2).
In Appendix D we prove that ResNet by He et al.
(2015) is invertible under mild assumptions.
We preﬁx the name of the model trained under our objective with ”Reg”, i.e. RegResNet/RegInvNet.
4. P ERFORMANCE
We report the accuracy of ResNet on test data of CIFAR100 in Table 1.
We compare the performance between our proposed regularization on w Ty k F ( x k ) and the naive regularization on w .
In both form of regularization we take α 1 = 0 and test over different choices of α 2 .
We pick smaller α 2 for naive regularization because it’s applied to the full matrix.
We observe the performance of ResNet-32 under naive regularization drops monotonically as α 2 increases.
Under a suitable choice of hyperparameters, RegResNet outperforms ResNet by a noticeable margin.
It implies that our proposed constrain on the irrelevant information
I( X ; (cid:98) Y ) − I
( Y ; (cid:98) Y ) is beneﬁcial to the classiﬁcation performance.
However, if the hyperparameters are too large, the performance drops, i.e. α 2 = 0 .
for ResNet-32 and α 2 = 0 .
15 for
ResNet-Wide; this matches the discussion in Section 2.2 that the model may try to reduce the relevant information
I( Y ; (cid:98) Y ) .
Our approach addresses the problem of over-ﬁtting; ResNet-Wide is improved by a larger margin compared to
Our
ResNet-32 because ResNet-Wide has higher capacity and is therefore over-ﬁts more to the training data.
4. A NALYSIS ON F EATURES
We introduce another invertible structured neural network on MNIST dataset and analyze the feature
F ( X ) learned in the last layer qualitatively.
The feature map F is built to be LeNet-300-100 and the decoder D has the opposite structure.
At each step during the training process, we update the autoencoder F + D and the InvNet F + w alternatively; our regularization is applied to w as usual.
In this section we report our result with α
= α 2 = 0 .
4.2. L EARNING TO O VERLOOK I RRELEVANT I NFORMATION
We feed 1k testing samples of digit 9 into the neural network to get 1k samples of features F ( X ) .
Recall that the features F ( X ) are vectors of dimension 100 and w is a matrix of size 100 × 10 .
We calculate the mean and standard deviation of each entry of the features from these 1k samples.
From
Figure 2 we see that under our regularization, the entry-wise products w 10 Ti F ( X ) i becomes sparse as only a few entries have large values for both feature and weight. This implies that the information needed to compute the logits for classiﬁcation is encoded into only a few entries of the feature, which we regard as relevant entries.
Our regularization forces small w 10 on irrelevant entries, so the logits it outputs are not sensitive to variations in these entries; on the other hand, we do encourage large w 10 on the relevant entries.
This matches our motivation that a neural network should be robust against irrelevant information I( F ( X ); (cid:98) Y ) − I ( Y ; (cid:98) Y ) and focus on the relevant information
I( Y ; (cid:98) Y ) .
[Figure]
4.2. L EARNING TO E XTRACT R ELEVANT I NFORMATION
It had been argued by Szegedy et al.
(2013) that it is the space but not individual units in high level features that encodes interpretable information.
Under our regularization, a meaningful basis of the informative space is found; in particular, the features of 9 are encoded into 10 entries (see Figure 2).
On the other hand, features that have high values in these entries are expected to be the features of digit 9.
To validate this conjecture, for each digit, we ﬁnd the entry of its feature mean with the highest value.
We use the micro and macro average ROC metric on these feature entries and compare the results from InvNet and RegInvNet in Figure 3.
The curve with larger area underneath indicates higher representative power of individual entries learned in the features.
We conclude that under our regularization, relevant information for classiﬁcation is encoded into only a few key entries of the features, and these entries are highly indicative and interpretable.
[Figure]
4. R ELATIONSHIP TO L INEAR R EGRESSION
We trained ResNet-32 on CIFAR10, the feature F ( X ) is a vector of size 64, the classiﬁer w is a matrix of size 64 × 10 .
Note that the product w T F ( X ) is a vector of size 10 representing the probability assigned by the model for each class.
Under our framework, deep learning can be conceptually simpliﬁed to regularized linear regression if we regard F ( X ) as input data.
However, F ( X ) depends on the model parameters in the previous layers so it’s not ﬁxed like real data. Moreover we observe in our experiments that naive regularization on w alone will upscale the norm of F ( X ) , which neutralizes the effect of regularization.
In Figure 4 we show that as under our regularization, the (cid:96) 2 -norm of w is suppressed while the (cid:96) 2norms of feature F ( X ) remain similar.
In addition, several rows of w are trained to be zero, which implies that many entries of the feature F ( X ) are regarded by the network as irrelevant information for classiﬁcation, since any variance in the entries of F ( X ) where the corresponding rows of w are zero has no inﬂuence on the probability assigned to each label class by the model.
T HE R OLE OF I NVERTIBILITY
Invertibility allows us to treat F ( X ) as transformed data that preserves all the information from X , and therefore work on the information regularization problem under a linear scheme.
In Appendix D we prove that ResNet is fairly invertible due to the intrinsic invertibility of the operator I + L given (cid:107)L(cid:107) < 1 .
In this section we build a PlainNet by using only L as the operator for each building block, so the theoretical guarantee for invertibility is not present for PlainNet.
In Table 2, we see that PlainNet-32 can still beneﬁt from our regularization, however, it’s performance is less stable compared to ResNet-32 if the hyper-parameters are too large.
The reason is for PlainNet, the feature in the last layer F ( X ) does not preserve information about X very well, so it has a higher demand on the capacity of the classifer w and is therefore more sensitive to our regularization.
[Table]
The means and standard deviations reported are based on 5 samples.
It can be observed that although
PlainNet-32 obtains marginal improvement for small α 2 , the performance drops dramatically and
C ONCLUSION
We give an interpretation of the deep learning dynamics by decomposing it into an signal transformation stage and feature classiﬁcation stage, where we emphasis importance of the classiﬁer w in the last fully connected layer given that the feature map F is invertible.
Then we take the advantage of the fact that mutual information quantities are invariance under invertible mapping to attack our proposed information optimization problem for supervised classiﬁcation in deep learning.
Our theory justiﬁes the use of direct regularization terms on w, F ( X ) for neural networks with invertibility property.
Our regularization improves the performance of neural networks by a noticeable margin and is capable of encouraging the interpretability of the entries of features learned in the last layer.
A P ROOF OF P ROPOSITION 2.
Proposition 2.1 establishes a connection between I( X, (cid:98) Y ) and the absolute value of the logits | w T F ( X ) | for the binary case.
Intuitively, decreasing the conﬁdence of the model on its predictions will decrease the mutual information
I( X ; (cid:98) Y ) .
Proposition 2.1.
I( X ; (cid:98) Y ) =
I( F ( X ); Y ) is well estimated by its empirical version (Montecarlo approximation) with high probability, which shares the same unique (global) minimum with (cid:80)
nk =1 | w T F ( x k ) | at w T F ( x k )
= 0 , for all k ∈ { 1 , ..., n } .
The mutual information I( X ; (cid:98) Y ) is given as
[Equation]
Apply the assumption (II) , the marginal distribution of (cid:98) Y is uniformly distributed:
[Equation]
Substituting (9) into (8) yields
[Equation]
According to the Hoeffding’s inequality for bounded random variables [Proposition 2.2.6, Vershynin (2018)], let M, m denote upper and lower bounds of the integrand of (10) correspondingly, we have
[Equation]
Equivalently, with probability at least 1 − δ ,
[Equation]
Here (cid:80) nk =1 (cid:80) (cid:98) y ∈C p ( (cid:98) y
| x k ) log(2 p ( (cid:98) y
| x k ))
/n is a Monte carlo estimation of RHS of I( X ; (cid:98) Y ) .
Recall that, for the binary case p (
[Equation]
y | x ) = p ( (cid:98) y | F ( x )) can expressed as
[Equation]
Then we have (cid:88)
[Equation]
y ∈C
[Equation]
which is bounded by [0 , log(2)] .
Take M = log(2) , m = 0 , we have
[Equation]
hold with probability at least 1 − δ .
The conclusion follows from the fact that (cid:80) nk =1 (cid:80) (cid:98) y ∈C p ( (cid:98) y | x k ) log(2 p ( (cid:98) y
| x k ))
/n has a unique global minimum at w T F ( x k ) = 0 for each x k .
B P ROOF OF P ROPOSITION 2.
Consider the training samples { ( x k , y k ) } k =1: n , each x k is fed into a deep probabilistic model which outputs probability densities and predicts (cid:98) y k , a realization of the prediction (cid:98) Y .
Let C = {± 1 } be the binary class and n y be the counts of observed occurrences of k satisfying y k = y ∈ C , then n = (cid:80) y ∈C n y
=
(cid:80) y n y , where we omit the range over C for convenience.
We denote the true joint probability with π y (cid:98) y
= p ( y, (cid:98) y ) , the marginal probabilities with π y + = (cid:80) (cid:98)
y π y (cid:98) y and π + (cid:98)
y
= (cid:80) y π y (cid:98) y .
The mutual information I( Y ; (cid:98) Y ) can be expressed as
[Equation]
Our empirical mutual information I( (cid:98) π ) is deﬁned as
[Equation]
where (cid:98) π y (cid:98) y
= 12 n y (cid:80) n y i =1 σ ( (cid:98) yw T F ( x i )) .
Proposition 2.2 establishes a connection between I( Y ; (cid:98) Y ) and the cross-entropy objective for the binary case.
Intuitively, increasing the conﬁdence of the model on its correct predictions will establish a more deterministic relationship between Y and (cid:98) Y and thus increase the mutual information
I( Y ; (cid:98) Y ) .
Proposition 2.2.
I( Y ; (cid:98) Y ) is well estimated by I( (cid:98) π ) with high probability, which shares the same unique (global) maximum with (cid:80) nk =1 log σ ( y k w T F ( x k )) at y k w T F ( x k ) → ∞ given that y k w T F ( x k ) > 12 , for all k ∈ { 1 , ..., n } .
Proposition 2.2 follows from Proposition B.1 and Proposition B.2, where Proposition B.1 shows that I( Y ; (cid:98) Y ) is well approximated by I( (cid:98) π ) with high probability and Proposition B.2 shows the remaining claims in Proposition 2.2.
As shown in Lemma B.1, (cid:98) π y (cid:98) y
= 12 n y (cid:80) n y i =1 σ ( (cid:98) yw T F ( x i )) is an unbiased estimate of π y (cid:98) y .
Here σ to represent the sigmoid function deﬁned by σ ( x ) =
e x / ( e x + 1) .
By leveraging the concentration property of bounded variables, i.e., σ ( (cid:98) yw T F ( x i )) , the estimation error can be bounded with high probability (Lemma B.2).
Lemma B.1.
The empirical joint probability, deﬁned as
[Equation]
, (18) is an unbiased estimate of the true joint distribution π y (cid:98) y .
Lemma B.2.
With probability at least 1 − δ , we have
[Equation]
Proposition B.1.
With probability at least 1 − δ ,
[Equation]
Proof.
To estimate the empirical mutual information given ﬁxed samples
[Equation]
we use the approach by
Hutter & Zaffalon (2005).
In particular, taylor expansion gives
[Equation]
where ∆ y (cid:98) y
= (cid:98) π y (cid:98)
y − π y (cid:98) y .
Hence, Eq (21) together with Lemma B.2 yield, with probability exceeding 1 − |C| 2 δ ,
[Equation]
Notice that, in the binary case the cardinality |C| = 2 .
Next we prove the intermediate results, Lemmas B.1 and B.2.
Proof of Lemma B.1.
Direct derivation on the true joint distribution π y (cid:98) y gives
[Equation]
Given assumption (I) which states that the marginal density of Y is uniform over C , for every true label y ∈ C we have p ( y ) =
We can therefore rewrite (23) as
[Equation] [Equation]
According to assumption (II) , p ( x | y ) is a probability density over space of signal x with true label y .
The Monte Carlo estimation of (24) gives the empirical joint probability which is unbiased:
[Equation]
Proof of Lemma B.2.
Again, by leveraging the Hoeffding’s inequality for bounded random variables [Proposition 2.2.6 of Vershynin (2018)]
[Equation]
we have
[Equation]
where X y is the data random variable whose true label is y
[Equation]
Equivalently, with probability at least 1 − δ ,
[Equation] [Equation]
where M, m are upper and lower bounds of random variable σ ( (cid:98) yw T F ( X y )) , respectively.
Substitute (24) and (25) into (27), with probability at least 1 − δ ,
[Equation]
To estimate the upper and lower bounds M, m for σ ( (cid:98) yw T F ( X )) , we use the Taylor’s theorem:
[Equation]
29) and
[Equation]
given that the derivative of sigmoid function is bounded by 14 .
It follows that
[Equation]
Also notice that M, m are the bounds for sigmoid function, so their difference is at most 1.
From derivations above, we can rewrite (28) as
[Equation]
and the lemma follows.
Proposition B.2.
The empirical mutual information I( (cid:98) π ) shares the same unique (global) maximum with (cid:80) nk =1 log σ ( y k w T F ( x k ))
as y k w T F ( x k )
→ ∞ given that y k w T F ( x k ) > 12 , for all k ∈ { 1 , ..., n } .
Proof.
The empirical information I( (cid:98) π ) is deﬁned by
I
[Equation]
where the empirical joint probability is given by
[Equation] [Equation]
It then follows that for any y ∈ C ,
[Equation]
In binary case it means that
[Equation]
and the empirical mutual information can decomposed as
[Equation]
We differentiate (37) with respect to (cid:98)
π 11 and (cid:98) π ( − 1)( − 1)
[Equation]
and calculate the critical points over the domain [0 , 12 ] for both variables
[Equation]
which gives
[Equation]
Observe that (38) is a global minimum over [0 , 12 ] × [0 , 12 ] .
Since this is the unique critical point where the derivative vanishes, the global maximums can only be obtained on the boundaries.
In particular, if we restrict ( (cid:98) π 11 , (cid:98) π ( − 1)( − 1) ) on [ 14 , 12 ] × [ 14 , 12 ] , (37) is a strictly increasing function over (cid:98) π 11 , (cid:98) π ( − 1)( − 1) and the unique global maximum is obtained at
[Equation]
The proposition follows from the deﬁnition (34) of (cid:98) π y (cid:98)
y that (39) is only approached when y k w T F ( x k ) → ∞ , for all k ∈ { 1 , ..., n } .
C I NVERTIBILITY IS B ENEFICIAL
We show in Proposition C.1 that the lower bound for the classiﬁcation error is itself lower bounded by a constant, which is attained if F is invertible.
Although the performance of the model also depends on the classiﬁer w , our bound claims that an invertible feature map F could provide a better environment for the classiﬁer w to perform well.
Intuitively, invertibility preserves the information of the signal X as it ﬂows through the neural network and reaches the classiﬁer w ; on the other hand, w potentially performs better on the input that preserves all information of the data compared to the one that doesn’t.
Proposition C.1.
(Fano’s Inequality)
The classiﬁcation error is lower bounded as follows:
[Equation]
The lower bound satisﬁes
[Equation]
for all F , and the equality is attained if F is invertible.
Let Z = F ( X ) and the machinery of deep learning can be decribed by the following Markov Chain:
[Equation]
Lemma C.1 is a technical result that helps to prove Proposition C.1.
The information Z = F ( X ) has about the true labels Y is maximized when F is invertible, which is beneﬁcial in the sense that the key information inﬂuential for classiﬁcation can be well preserved.
Lemma C.1 (Chain Rule) .
Given the Markov Chain assumption equation 42, we have
[Equation]
and the second equality is attained if F is invertible.
Proof.
We will only prove the second inequality and the ﬁrst inequality follows by a similar argument.
Consider the decomposition
[Equation]
Similarly we obtain
[Equation]
equation 44 together with equation 45 yields
[Equation]
According to the Markov Chain setting
[Equation]
Y and Z are conditionally independent given X , hence
I( Y ; Z | X ) = 0 ; in addition, the mutual information I( X ; Y | Z ) is nonnegative.
It follows from (46) that
[Equation]
Next we present a lower bound for the classiﬁcation error.
This lower bound is negatively related to the mutual information I ( Y ; F ( X )) , and it attains its minimum if F is invertible.
Although the performance also depends on the classiﬁer w , Proposition C.1 implies that an invertible feature map
F allows more chances for the classiﬁer w to perform well.
Proof of Proposition C.1.
Consider the random variable E deﬁned as:
[Equation]
By the Chain Rule following from similar arguments presented in Lemma C.1, we have
[Equation]
Note that H ( E | Y, (cid:98) Y ) = 0 , since the value of E is determined given the knowledge of Y, (cid:98) Y .
It then follows that
[Equation] [Equation]
On the other hand, Lemma C.1 shows that
[Equation] [Equation]
which gives
[Equation]
Substitute it into (50) yields the result
[Equation]
As for the second statement, Lemma C.1 shows that
[Equation]
It then follows that,
[Equation]
where the equality is attained if F is invertible.
D I NVERTIBILITY OF R ES N ET
ResNet is designed to allow the model to ”learn” the identity map easily.
Speciﬁcally, the input vector x and output vector y of a building block are related by
[Equation]
where the operator L could be a composition of activation functions, convolutions, dropout(Srivastava et al.
and batch normalization(Ioffe & Szegedy (2015)).
It’s shown in
Lemma D.1 below that if the operator norm |L| < 1 , then L + I is theoretically guaranteed to have an inverse, which enables information preservation among intermediate layers.
In Figure 5 we experimentally verify that |L| < 1 for all building blocks during the training process.
In general, operations such as ReLU, pooling, drop-out are not invertible(Dosovitskiy & Brox, 2015); it’s challenging to build a strictly invertible network (Jacobsen et al., 2018).
From this point of view, the beauty of ResNet lies in the fact that it’s guaranteed to be invertible regardless how L evolves during the training process, as long as |L| < 1 .
Although the usual design of ResNet does involve non-invertible components such as pooling, we argue that ResNet still has descent invertible property compared to the majority of other neural network designs.
We also experimentally verify that our regularization does not improve a very deep ResNet on its performance by a clear margin; we speculate that information will lose more as it goes deeper.
Lemma D.1.
Consider Holder Space C 0 , 1 ( U ) , where U is the closure of some bounded open set U , with equiped Holder norm |L| =
[Equation]
, here α is some positive scalar.
If L ∈ C 0 , 1 ( U ) and |L| < 1 , then there exists B such that B ( I + L ) =
( I + L ) B
= I .
Proof.
It’s well known that C 0 , 1 ( U ) is a Banach space (Lax).
Deﬁne
[Equation]
Since |L| < 1 , (57) is a Cauchy sequence and coverges in Banach space.
Convergence sequences can be multiplied termwise, it follows that
[Equation]
So B ( I + L ) =
I .
The other equality ( I + L ) B
= I can be shown to hold in the same fashion.
E I MPLEMENTATION D ETAILS
We implement all models using Tensorﬂow.
We modify the ResNet based on the code provided at https://github.com/tensorflow/models/tree/master/research/resnet .
We follow the same learning rate scheme proposed in the original code.
For the InvNet on MNIST, we train the network with initial learning rate 0.1, and decay it by 0.7 every 10 epochs.
For both
InvNet and ResNet, we apply the (cid:96) 1 norm regularization every 30 iterations.
F R ESULTS FOR A LL D IGITS ON I NV N ET
We observe that each digits have their speciﬁc entries with high value assigned to both weights and feature means.
An additional observation is that typically the feature entry with low mean also has low standard deviation, such entry rarely contributes to the logits for classiﬁcation if the corresponding value of weight is also small.
G R ESULTS ON I -R EV N ET
We reproduce the results in Section 4.2 on i-RevNet (Jacobsen et al.
The statistics and
ROC curves of the features produced by i-RevNet and Reg-i-RevNet on CIFAR10 is similar to those presented in Section 4.2.
H H ISTOGRAM ON F EATURES OF D IGIT
The histogram for values of feature entries of Digit 9 has a decaying shape but with a heavier tail compared to that of Gaussian with small variance.
The spasity of w depends on our choice of hyperparameters.
For example in Figure 4 we measure the sparsity of the learned w of RegResNet for CIFAR10: about 60% entries of w are precisely zero.
I T HE U SE OF S URROGATE F UNCTIONS TO M INIMIZE M UTUAL I NFORMATION
Our objective is
[Equation]
Assume (cid:98) Y is uniformly distributed on C , H ( (cid:98) Y ) becomes a constant and we have
[Equation]
Note that (60) is composed of functions in the form a log a where a is the output of a softmax function on logits.
For simplicity we consider the binary case where a = (cid:16) 1 + e − w T F ( x ) (cid:17) − 1 .
The derivative of it with respect to w takes the form
[Equation]
[Equation]
Assume | w T F ( x ) | is large, if w T F ( x ) > 0 then the numerator decays exponentially with respect to | w T F ( x ) | and the denominator converges to 1; if wF ( x ) <
then the denonimator grows exponentially with respect to | w T F ( x ) | and dominates the numerator.
To conclude, for large | w T F ( x ) | the gradient is decaying to zero exponentially.
So the effect of punishing large logits | w T F ( x ) | by this objective is not clear as the gradient vanishes for large | w T F ( x ) | .
We also analyzed the general softmax functions for multi-labels and found they exhibit similar properties.
We propose to use a surrogate function w T F ( X ) to minimize this objective to make the regularization effect more clear.
We prove that our regularizer achieves the same goal compared to the mutual information objective but provides a better gradient for training.