P ARAMETRIC I NFORMATION B OTTLENECK TO O PTIMIZE S TOCHASTIC N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we present a layer-wise learning of stochastic neural networks (SNNs) in an information-theoretic perspective.
In each layer of an SNN, the compression and the relevance are deﬁned to quantify the amount of information that the layer contains about the input space and the target space, respectively.
We jointly optimize the compression and the relevance of all parameters in an SNN to better exploit the neural network’s representation.
Previously, the Information
Bottleneck (IB) framework (Tishby et al.
(1999)) extracts relevant information for a target variable.
Here, we propose Parametric Information Bottleneck (PIB) for a neural network by utilizing (only) its model parameters explicitly to approximate the compression and the relevance.
We show that, as compared to the maximum likelihood estimate (MLE) principle, PIBs : (i) improve the generalization of neural networks in classiﬁcation tasks, (ii) push the representation of neural networks closer to the optimal information-theoretical representation in a faster manner.
I NTRODUCTION
Deep neural networks (DNNs) have demonstrated competitive performance in several learning tasks including image recognition (e.g., Krizhevsky et al.
(2012), Szegedy et al.
(2015)), natural language translation (e.g., Cho et al.
, Bahdanau et al.
and game playing (e.g., Silver et al.
Specially in supervised learning contexts, a common practice to achieve good performance is to train DNNs with the maximum likelihood estimate (MLE) principle along with various techniques such as data-speciﬁc design of network architecture (e.g., convolutional neural network architecture), regularizations (e.g., early stopping, weight decay, dropout (Srivastava et al.
(2014)), and batch normalization (Ioffe & Szegedy (2015))),
and optimizations (e.g., Kingma & Ba (2014)).
The learning principle in DNNs has therefore attributed to the MLE principle as a standard one for guiding the learning toward a beneﬁcial direction.
The question however is that does the MLE principle effectively and sufﬁciently exploit a neural network’s representative power and is there any better alternative?
As an attempt to address this important question, this work investigates the learning of DNNs from the information-theoretic perspective.
The Information Bottleneck (IB) framework (Tishby et al.
(1999)) extracts relevant information in an input variable X about a target variable Y .
More speciﬁcally, the IB framework constructs a bottleneck variable Z
= Z ( X ) that is an compressed version of X but preserves as much relevant information in X about Y as possible.
In this information-theoretic perspective, I ( Z, X ) , the mutual information of Z and X , captures the compression of Z about X and I ( Z, Y ) represents the relevance of Z about Y .
The optimal representation Z is determined via the minimization of the following Lagrangian:
[Equation]
where β is the positive Lagrangian multiplier that controls the trade-off between the complexity of the representation, I ( Z, X ) , and the amount of relevant information in Z , I ( Z, Y ) .
The exact solution to the minimization problem above is found (Tishby et al.
with the implicit self- consistent equations:
[Equation]
where Z ( x ; β ) is the normalization function, and D KL [ . (cid:107) .
] is the Kullback - Leibler (KL) divergence (Kullback & Leibler (1951)).
Unfortunately, the self-consistent equations are highly non-linear and still too abstract to be used for many practical applications.
Furthermore, the general IB framework assumes that the joint distribution p ( X, Y ) is known and does not specify concrete models.
On the other hand, the goal of the MLE principle is to match the model distribution p model as close to the empirical data distribution ˆ p D as possible (e.g., see Appendix I.B).
The MLE principle treats the neural network model p ( xxx ; θθθ ) as a whole without explicitly considering the contribution of its internal structures (e.g., hidden layers and hidden neurons).
As a result, a neural network with redundant information in hidden layers may have a good distribution match in a training set but show a poor generalization in a test set.
In the MLE principle, we only need empirical samples of the joint distribution to maximize the likelihood function of the model given the data.
The MLE principle are proved to be mathematically equivalent to the IB principle for multinomial mixture model for clustering problem when the input distribution X is uniform or has a large sample size (Slonim & Weiss (2002)).
However in general they are not obviously related.
In this work, we leverage neural networks and the IB principle by viewing neural networks as a set of encoders that sequentially modify the original data space.
We then propose a new generalized
IB-based objective that takes into account the compression and relevance of all layers in the network as an explicit goal for guiding the encodings in a beneﬁcial manner.
Since the objective is designed to optimize all parameters of neural networks and is mainly motivated by the IB principle for deep learning (Tishby & Zaslavsky (2015)), we name this method as Parametric Information Bottleneck (PIB).
Because the generalized IB objective is intractable, we approximate it using variational methods and Monte Carlo estimation.
We propose to re-use the existing neural network architecture as variational decoders for each hidden layers.
The approximate generalized IB objective in turn presents itself interesting connections with the MLE principle.
In practice, we empirically show that our PIBs have a better generalization and push the neural network’s representation closer to the information-theoretical optimal representation as compared to the MLE principle.
R ELATED W ORK
Originally, the general IB framework is proposed in Tishby et al.
The framework provides a principled way of extracting the relevant information in one variable X about another variable Y .
The authors represent the exact solution to the IB problem in highly-nonlinear self-consistent equations and propose the iterative Blahut Arimoto algorithm to optimize the objective.
However, the algorithm is not applicable to neural networks.
In practice, the IB problem can be solved efﬁciently in the following two cases only: (1) X, Y and Z are all discrete (Tishby et al.
(1999)); or (2) X, Y and Z are mutually joint Gaussian (Chechik et al.
where Z is a bottleneck variable.
Recently, the IB principle is applied to DNNs (Tishby & Zaslavsky (2015)).
The work proposes to use mutual information of a hidden layer with the input layer and the output layer to quantify the performance of DNNs.
By analyzing these measures with the IB principle, the authors establish an information-theoretic learning principle for DNNs.
In theory, one can optimize the neural network by pushing up the network and all its hidden layers to the IB optimal limit in a layer-wise manner.
Although the analysis offers a new perspective about optimality in neural networks, it rather proposes an general analysis of optimality than a practical optimization criteria.
Furthermore, estimating mutual information between the variables transformed by network layers and the data variables poses several computational challenges in practice that the author did not address in the work.
A small change in a multi-layered neural network could greatly modify the entropy of the input variables.
Thus, it is hard to analytically capture such modiﬁcations.
The recent work Alemi et al.
(2016) also uses variational methods to approximate the mutual information as an attempt to apply the IB principle to neural networks.
Their approach however considers one single bottleneck and parameterizes the encoder p ( zzz | xxx ; θθθ ) by an entire neural network.
The encoder maps the input variable xxx to a single bottleneck variable zzz that is not a part of the considered neural network architecture.
Therefore, their approach still learns a neural network as a whole rather than actually optimizing it in parts.
Furthermore, the work imposes a prior distribution in the code space.
In our work, the distribution of a code space is simply transformed by an encoder from another space.
Our work, on the other hand, focuses on better exploiting intermediate representations of a neural network architecture using the IB principle.
More speciﬁcally, our work proposes an optimization
IB criteria for an existing neural network architecture in an effort of learning better the layers’ representation to their IB optimality.
In estimating mutual information, we adopt the variational method as in Alemi et al.
for I ( Z, Y )
but use empirical estimation for I ( Z, X ) .
Furthermore, we exploit the existing network architecture as variational decoders rather than resort to variational decoders that are not part of the neural network architecture.
P ARAMETRIC I NFORMATION B OTTLENECK
This section presents an information-theoretic perspective of neural networks and then deﬁnes our
PIB framework.
This perspective paves a way for the soundness of constraining the compressionrelevance trade-off into a neural network.
We denote X, Y as the input and the target (label) variables of the data, respectively; Z l as a stochastic variable represented by the l th hidden layer of a neural network where 1 ≤ l ≤ L , L is the number of hidden layers.
We extend the notations of Z l by using the convention Z 0 :
=
X and Z − 1 := ∅ .
The space of X, Y and Z l are denoted as X , Y and Z l , respectively.
Each respective space is associated with the corresponding probability measures p u ( xxx ) , p u ( yyy ) and p ( zzz l ) where p u ( . ) indicates the underlying probability distribution of the data and p ( . )
denotes model distributions.
Each Z l is stochastically mapped from the previous stochastic variable Z l − 1 via an encoder p ( zzz l | zzz l − 1 ) .
We name Z l , 1 ≤ l ≤ L as a (information) bottleneck or code variable of the network.
In this work, we focus on binary bottlenecks where Z l ∈ { 0 , 1 } n l and n i is the dimensionality of the bottleneck space.
3. N EURAL N ETWORKS AS S EQUENTIAL Q UANTIZATION
An encoder p ( zzz | xxx ) introduces a soft partitioning of the space X into a new space Z whose probability measure is determined as p ( zzz ) =
(cid:82) p ( zzz | xxx ) p u ( xxx ) dxxx .
The encoding can modify the information content of the original space possibly including its dimensionality and topological structure.
On average, 2 H ( X | Z ) elements of X are mapped to the same code in Z .
Thus, the average volume of a partitioning of X is 2 H ( X ) / 2 H ( X | Z )
=
I
( X,Z ) .
The mutual information I ( Z, X ) which measures the amount of information that Z contains about X can therefore quantify the quality of the encoding p ( zzz | xxx ) .
A smaller mutual information I ( Z, X ) implies a more compressed representation Z in terms of X .
Since the original data space is continuous, it requires inﬁnite precision to represent it precisely.
However, only some set of underlying explanatory factors in the the data space would be beneﬁcial for a certain task.
Therefore, lossy representation is often more helpful (and of course more efﬁcient) than a precise representation.
In this aspect, we view the hidden layers of a multi-layered neural network as a lossy representation of the data space.
The neural network in this perspective consists of a series of stochastic encoders that sequentially encode the original data space X into the intermediate code spaces Z l .
These code spaces are lossy representations of the data space as it follows from the data-processing inequality (DPI) (Cover & Thomas (2006))
that
[Equation]
where we assume that Y, X, Z l and Z l +1 form a Markov chain in that order, i.e.,
[Equation]
A learning principle should compress irrelevant information and preserve relevant information in the lossy intermediate code spaces.
In the next subsection, we describe in details how a sequential series of encoders, compression and relevance are deﬁned in a neural network.
PIB M ODEL
Our PIB framework is an extension of the IB framework to optimize all paramters of neural networks.
In neural networks, intermediate representations represent a hierarchy of information bottlenecks that sequentially extract relevant information for a target from the input data space.
Existing
IB framework for DNNs speciﬁes a single bottleneck while our PIB preserves hierarchical representations which a neural network’s expressiveness comes from.
Our PIB also gives neural networks an information-theoretic interpretation both in network structure and model learning.
In PIBs, we utilize only neural network parameters θθθ for deﬁning encoders and variational relevance decoders at every level, therefore the name Parametric Information Bottleneck.
Our PIB is also a standard step towards better exploiting representational power of more expressive neural network models such as
Convolutional Neural Networks (LeCun et al.
and ResNet (He et al.
3.2. S TOCHASTICITY
In this paper, we focus on binary bottlenecks in which the encoder p ( zzz l | zzz l − 1 ) is deﬁned as p ( zzz l | zzz l − 1 ) =
n l (cid:89) i = p ( zzz l,
i | zzz i − 1 ) (5) where
[Equation]
σ ( . ) is the sigmoid function, and W ( l ) is the weights connecting the l th layer to the ( l + 1) th layer.
Depending on the structure of the target space Y , we can use an appropriate model for output distributions as follows: (1) For classiﬁcation, we model the output distribution with softmax function, p ( Y = i | zzz L ) =
softmax ( W ( L +1)
i : zzz L + bbb ( L +1) i ) ; (2) For binary output vectors Y , we use a product of Bernoulli distributions, p ( yyy | zzz L ) =
(cid:81)
i p ( yyy i | zzz L ) where p ( Y i = 1 | zzz L ) = σ ( W ( L +1) i : zzz L + bbb ( L +1) i ) ; (3) For real-valued output vectors Y , we use Gaussian distribution, p
[Equation]
Y
[Equation] [Equation]
The conditional distribution p ( yyy | xxx ) from the model is computed using the Bayes’ rule and the Markov assumption (Equation 4) in PIBs 1 :
[Equation]
where zzz = ( zzz 1 ,zzz 2 , ...,zzz L ) is the entire sequence of hidden layers in the neural network.
Note that for a given joint distribution p u ( xxx,yyy ) when a bottleneck Z l is deﬁned, so the encoding function p ( zzz l | xxx ) , the corresponding relevance decoder p true ( yyy | zzz l ) is uniquely determined by
[Equation]
It is also important to note that many stochastic neural networks have been proposed (e.g., Neal (1990), Neal (1992), Tang & Salakhutdinov (2013), Raiko et al.
(2014), Dauphin & Grangier (2016)).
However, our motivation for this stochasticity is that it enables bottleneck sampling given the data variables ( X, Y ) .
The generated bottleneck samples are then used to estimate mutual information.
Thus, our framework does not depend on a speciﬁc stochastic model.
For deterministic neural networks, we only have one sample of hidden variables given one data point.
Thus, estimating mutual information for hidden variables in this case is as hard as estimating mutual information for the data variables themselves.
3.2. L EARNING PRINCIPLE
Since the neural network is a lossy representation of the original data space, a learning principle should make this loss in a beneﬁcial manner.
Speciﬁcally in PIBs, we propose to jointly compress the network’s intermediate spaces and preserve relevant information simultaneously at all layers of the network.
For the l th -level bottleneck Z l , the compression is deﬁned as the mutual information between Z l and the previous-level bottleneck Z l − 1 while the relevance is speciﬁed as its mutual information with the target variable Y .
We explicitly deﬁne the learning objective for PIB as:
[Equation]
where the layer-speciﬁc Lagrangian multiplier β − 1 l controls the tradeoff between relevance and compression in each bottleneck, and the concept of compression and relevance is taken to the extreme when l = 0 (with convention that I ( Z 0 , Z − 1 )
=
I ( X, ∅ ) =
H ( X ) = constant ).
Optimizing PIBs now becomes the minimization of L PIB ( Z ) which attempts to decrease
I ( Z l , Z l − 1 ) and increase I ( Z l , Y ) simultaneously.
The decrease of I ( Z l , Z l − 1 ) makes the representation at the l th -level more compressed while the increase of I ( Z l , Y ) promotes the preservation of relevant information in Z l about Y .
In optimization’s aspect, the minimization of L PIB is much harder than the minimization of L IB since L PIB involves dependent terms that even the self-consistent equations of the IB framework are not applicable to this case.
Furthermore, L PIB is intractable since the bottleneck spaces are usually high-dimensional and the relevance encoders p true
( yyy | zzz l ) (computed by Equation 8) are intractable.
In the following section, we present our approximation to L PIB which fully utilizes the existing architecture without resorting to any model that is not part of the considered neural network.
The approximation then leads to effective gradientbased training of PIBs.
3. A PPROXIMATE LEARNING
Here, we present our approximations to the relevance and the compression terns in the PIB objective
L PIB .
3.3. A PPROXIMATE RELEVANCE
Since the relevance decoder p true ( yyy | zzz l ) (Equation 8) is intractable, we use a variational relevance decoder p v ( yyy | zzz l ) to approximate it.
Firstly, we decompose the mutual information into a difference
of two entropies:
[Equation]
where H ( Y ) =
constant can be ignored in the minimization of L ( Z ) , and
[Equation]
where the equality in Equation 12 holds due to the Markov assumption (Equation 4).
In PIBs, we propose to use the higher-level part of the existing network architecture at each layer to deﬁne the variational relevance encoder for that layer, i.e., p v ( yyy | zzz l ) =
p ( yyy | zzz l ) where p ( yyy | zzz l ) is determined by the network architecture.
In this case, we have:
[Equation]
We will refer to ˜ H
( Y | Z l ) as the variational conditional relevance for the l th -level bottleneck variable Z l for the rest of this work.
In the following, we present two important results which indicate that the relevance terms in our objective is closely and mutually related to the concept of the MLE principle.
Proposition 1 : The variational conditional relevance for the zero-level bottleneck Z
=
X is the negative log-likelihood of the PIB.
Proposition 2 :
The variational conditional relevance for the highest-level bottleneck variable Z L equals the variational relevance for the whole compositional bottleneck variable Z which is an upper bound on the negative log-likelihood.
That is,
[Equation]
While the Proposition 1 is a direct result of Equation 15, the Proposition 2 holds due to Jensen’s inequality (its detail derivation in Appendix I.A).
In PIB’s terms, the MLE principle can be interpreted as increasing the variational relevance of the network as a whole while the PIB objective takes into account the relevance at every level of the network.
The variational relevance can also be interpreted in terms of the MLE principle as follows.
Maximizing the variational relevance at level l is the maximum of the log-likelihood of p ( yyy | zzz l ) .
Furthermore, the variational conditional relevance for a multivariate yyy can be decomposed into the sum of that for each component of yyy (see Appendix I.C).
3.3. A PPROXIMATE C OMPRESSION
The compression terms in L PIB involve computing mutual information between two consecutive bottlenecks.
For simplicity, we present the derivation of I ( Z 1 , Z 0 ) only 2 .
For the compression, we decompose the mutual information as follows:
[Equation]
which consists of the entropy and conditional entropy term.
The conditional entropy can be further rewritten as:
[Equation]
approximate where Z 1 =
( Z 1 ,i ) N 1 i =1 and H ( Z 1 ,i | Z
=
z
z z 0 )
= − q log q − (1 − q ) log(1 − q ) where q = p ( Z 1 ,i = 1 | Z 0 = zzz 0 ) .
For the entropy term H ( Z 1 ) , we resort to empirical samples of zzz 1 generated by Monte
Carlo sampling to estimate the entropy:
[Equation]
where zzz ( k )∼ p ( zzz 1 )
=
E p ( zzz 0 )
[ p ( zzz 1 | zzz 0 )] .
This estimator is also known as the maximum likelihood estimator or ‘plug-in’ estimator (Antos & Kontoyiannis (2001)).
The larger number of samples M guarantees the better plug-in entropy by the following bias bound (Paninski (2003))
[Equation]
[Equation]
where | Z 1 | denotes the cardinality of the space of Z 1 .
In practice, log p ( zzz 1 ) may be numerically unstable for large cardinality | Z 1 | .
In the large space of Z 1 , the probability of a single point p ( zzz 1 ) may becomes very small that log p ( zzz 1 ) become numerically unstable.
To overcome this problem, we propose an upper bound on the entropy using Jensen’s inequality:
[Equation]
Thus,
[Equation]
The upper bound ˜ H ( Z 1 ) is numerically stable because the conditional distribution p ( zzz 1 | zzz 0 ) is factorized into (cid:81)
i p ( zzz 1 ,i | zzz 0 )
, therefore, log p ( zzz 1 | zzz 0 )
= (cid:80) i log p ( zzz 1 ,i | zzz 0 ) which is more stable.
The upper bound ˜ H ( Z 1 ) can then be estimated using Monte Carlo sampling for zzz 0 and zzz 1 .
3.3. A PPROXIMATE G RADIENTS VIA B INARY B OTTLENECKS
Discrete-valued variables in PIBs make standard back-propagation not straightforward.
Fortunately, one can estimate the gradient in this case.
Tang & Salakhutdinov (2013) used a Generalized EM algorithm while Bengio et al.
(2013) proposed to resort to reinforcement learning.
However, these estimators have high variance.
In this work, we use the gradient estimator inspired by Raiko et al.
(2014) for binary bottlenecks because it has low variance despite of being biased.
Speciﬁcally, a bottleneck zzz =
( z 1 , z 2 , ..., z n l ) can be rewritten as being continuous by z i =
σ ( a i ) + (cid:15) i where
[Equation]
probability
[Equation]
The bottleneck component z i deﬁned as above still gets value of either 0 or 1 but it is decomposed into the sum of a deterministic term and a noise term.
The gradient is then propagated only through the deterministic term and ignored in the noise term.
A detail of gradient-based training of PIB is presented in Algorithm 1.
E XPERIMENTS
We used the same architectures for PIBs and Stochastic Feed-forward Neural Networks (SFNNs)
(e.g., Tang & Salakhutdinov (2013))
and trained them on the MNIST dataset (LeCun et al.
for image classiﬁcation, odd-even decision problem and multi-modal learning.
Here, a SFNN simply prefers to feed-forward neural network models following the MLE principle for learning model parameters.
Each hidden layer in SFNNs is also considered as a stochastic variable.
The aforementioned tasks are to evaluate PIBs, as compared to SFNNs, in terms of generalization, learning dynamics, and capability of modeling complicated output structures, respectively.
All models are implemented using Theano framework (Al-Rfou et al.
4. MNIST C LASSIFICATION
In this experiment
[Table]
we compare PIBs with SFNNs and deterministic neural networks in the classiﬁcation task.
For comparisons
[Table]
we trained PIBs and ﬁve additional models.
The ﬁrst model (Model
A) is a deterministic neural network.
In Model D, we used the weight trained in Model A to perform stochastic prediction at test time.
Model E is SFNN and Model B is Model C with deterministic prediction during test phase.
Model C uses the weighted trained in PIB but we report deterministic prediction instead of stochastic prediction for test performance
[Table]
The MNIST dataset (LeCun (1998)) contains a standard split of 60000 , and 10000 examples of handwritten digit images for training and test, respectively in which each image is grayscale of size
28 × 28 pixels.
We used the last 10000 images of the training set as a holdout set for tuning hyperparameters.
The best conﬁguration chosen from the holdout set is used to retrain the models from scratch in the full training set.
The result in the test set is then reported (for stochastic prediction,
Because of the Markov assumption (Equation 4), we have H ( X ) ≥ I ( Z i , X ) ≥
I
( Z i +1 , X ) and
I ( X, Y ) ≥ I ( Z l , Y ) ≥ I ( Z l +1 , Y ) .
we report mean and standard deviation).
We scaled the images to [0 , 1] and do not perform any other data augmentation.
These base conﬁgurations are applied to all six models we use in this experiment.
The base architecture is a fully-connected, sigmoid activation neural network with two hidden layers and 512 units per layer.
Weights are initialized using Xavier initialization (Glorot & Bengio (2010)).
Models were optimized with stochastic gradient descent with a constant learning rate of 0 . 1 and a batch size of 8 .
For stochastic sampling, we generate M = 16 samples per point during training and
M = 32 samples per point during testing.
For stochastic prediction, we run the prediction 10 times and report its mean and deviation standard.
For PIBs, we set β l = β, ∀ 1 ≤ l ≤ L .
We tuned β from { 0 } ∪ { 10 − i : 1 ≤ i ≤ 7 } , and found β − 1 = 10 − 4 works best.
Table 1 provides the results in the MNIST classiﬁcation error in the test set for PIB and the comparative models (A), (B), (C), (D), and (E).
As can be seen from the table, PIB and Model C gives nearly the same performance which outperform deterministic neural networks and SFNNs, and their stochastic and deterministic version.
It is interesting to empirically see that the deterministic version of PIB at test time (Model C) gives a slightly better result than PIB.
This also empirically holds for the case of SFNN.
To investigate more in this, we compute the test error for various values of the number of samples used for Monte-Carlo averaging, M (Figure 2).
As we can see from the ﬁgure, the Monte-Carlo averaging of PIB obtains its good approximation around M = 30 and the deterministic prediction roughly places a lower bound on the Monte-Carlo averaging at test time.
For visualization of learned ﬁlters of PIB, see
Appendix II.A.
4. L EARNING DYNAMICS
One way to visualize the learning dynamic of each layer of a neural network is to plot the layers in the information plane (Tishby et al.
(1999), Slonim (2003)).
The information plane is an informationtheoretic plane that characterizes any representation Z
= Z ( X ) in terms of ( I ( Z, Y ) , I ( Z, X )) given the joint distribution
I ( X, Y ) .
The plane has I ( Z, X ) and I ( Z, Y ) as its horizontal axis and its vertical axis, respectively.
In the general IB framework, each value of β speciﬁes a unique point of Z in the information plane.
As β varies from 0 to ∞ , Z traces a concave curve, known as information curve for representation Z , with a slope of β − 1 .
The information-theoretic goal of learning a representation Z =
Z ( X ) is therefore to push Z as closer to its corresponding optimal point in the information curve as possible.
For multi-layered neural networks, each hidden layer Z l is a representation that can also be quantiﬁed in the information plane.
In this experiment, we considered an odd-even decision problem in the MNIST dataset in which the task is to determine if the digit in an image is odd or even.
We used the same neural network architecture of 784-10-10-10-1 for PIB and SFNN and trained them with SGD with constant learning rate of 0 .
01 in the ﬁrst 50000 training samples.
For PIB, we use β − 1 l = β − 1 = 10 − 4 .
Since the network architecture is small, we can compute mutual information I x := I ( Z i , X ) and I y :=
I ( Z i , Y ) precisely and plot them over training epochs.
As indicated by Figure 3, both PIB and SFNN enable the network to gradually encode more information into their hidden layers at the beginning as I ( Z i , X ) increases.
The encoded information at the beginning also contains some relevant information for the target variable as I ( Z i , Y ) increases as well.
However, information encoding in the PIB is more selective as it quickly encodes more relevant information (it reaches higher I ( Z, Y ) but in lesser number of epochs) while keeps the layers concise at higher epochs.
The SFNN, on the other hand, encodes information in a way that matches the model distribution to the empirical data distribution.
As a result, it may encode irrelevant information that hurts the generalization.
For additional visualization, an empirical architecture analysis of PIB and SFNN is presented in
Appendix II.B.
4. M ULTI - MODAL LEARNING
As PIB and SFNN are stochastic neural networks, they can model structured output space in which a one-to-many mapping is required.
A binary stochastic variable zzz l of dimensionality n l can take on 2 n l different states each of which would give a different yyy .
This is the reason why the conditional distribution p ( yyy | xxx ) in stochastic neural networks is multi-modal.
In this experiment, we followed Raiko et al.
and predicted the lower half of the MNIST digits using the upper half as inputs.
We used the same neural network architecture of 392-512-512for PIB and SFNN and trained them with SGD with constant learning rate of 0 .
We trained the models in the full training set of 60000 images and tested in the test set.
For PIB, we also used β − 1 l = β − 1 = 10 − 4 .
The visualization in Figure 4 indicates that PIB models the structured output space better and faster (using lesser number of epochs) than SFNN.
The samples generated by PIB is totally recognizable while the samples generated by SFNN shows some discontinuity (e.g., digit 2 , 4 , 5 , 7 ) and confusion (e.g., digit 3 confuses with number 8 , digit 5 is unrecognizable or confuses with number 6 , digit 8 and 9 are unrecognizable).
C ONCLUSION
In this paper we introduced an information-theoretic learning framework to better exploit a neural network’s representation.
We have also proposed an approximation that fully utilizes all parameters in a neural network and does not resort to any extra models.
Our learning framework offers a principled way of interpreting and learning all layers of neural networks and encourages a more informative yet compressed representation, which is supported by qualitative empirical results.
One limitation is that we consider here fully-connected feed-forward architecture with binary hidden layers.
Since we used generated samples to estimate mutual information, we can potentially extend the learning framework to larger and more complicated neural network architectures.
This work is our ﬁrst step toward exploiting expressive power of large neural networks using informationtheoretic perspective that is not yet fully utilized.
A PPENDIX I A. P ROOF OF THE P REPOSITIONS
Proof of the Preposition 2 : It follows from the Markov chain assumption (4) that p ( yyy | zzz ) =
p ( yyy | zzz L ,zzz L − 1 , ...,zzz 1 ) = p ( yyy | zzz L ) and from Jensen’s inequality
that
[Equation]
Hence, the variational compositional relevance ˜ H
( Y | Z ) equals the variational relevance for the last bottleneck and is an upper bound on the negative log-likelihood as well (Q.E.D).
B. MLE AS DISTRIBUTION MATCHING
The purpose of the MLE principle can be interpreted as matching the model distribution to the empirical data distribution using the KL divergence as a measure of their discrepancy.
Rigorously, given a set of samples X = { xxx 1 ,xxx 2 , ...,xxx N } i.i.d.
drawn from some underlying data distribution p D ( x x x ) , a parametric model p model ( x x x ; θ θ θ ) attempts to map any data sample x x x to a real number that estimates the true probability p D ( xxx ) .
The MLE principle maximizes the likelihood function under the empirical data distribution. This in turn can be interpreted as matching the model distribution p model with the data distribution p D by minimizing their KL divergence to ﬁnd the maximum likelihood (point) estimator for θθθ :
[Equation]
[Equation]
where expression (27) is an empirical estimation of expression (24) for N datapoints
[Equation]
C. V ARIATIONAL RELEVANCE FOR MULTIVARIATE TARGET VARIABLE
The variational conditional relevance at level l (deﬁned by (14), (15)) for a multivariate variable yyy can be decomposed into the variational conditional relevances for each of its components.
Indeed, without loss of generality, we assume bivariate target variable yyy =
( y 1 , y 2 ) .
It follows from the fact that the neurons within a layer are independent given some previous layer that we have
[Equation]
A PPENDIX II A. L EARNED FILTERS IN PIB
In a standard deterministic neural network and SFNN, all layers in the network are modiﬁed in a collaborative manner to reduce the likelihood function as a whole at each iteration.
In PIB, on the other hand, each layer contributes to the relevance level of the entire network and the layer itself.
Therefore, it is expected that a layer in an PIB captures more relevant information about the target variable.
To observe this effect, we look at the ﬁrst-level features learned by deterministic neural network, SFNN and PIB in Figure 5.
The ﬁgure shows that PIB shows sharper features at many units that deterministic neural network and SFNN cannot learn.
B. A RCHITECTURE A NALYSIS
It is also interesting to analyze how expanding 3 network architecture affects its learning and performance.
The expansion can either in vertical direction (Figure 6, 7 and 8), i.e., changing the number of units within a layer, or horizontal direction, i.e., changing the number of hidden layers.
For vertical expansion, it can be interpreted as changing the cardinality of the code space but in an effective way: exponentially.
Increasing a code space’s cardinality means allow more room for encoding more information.
To extract relevant information from the original data space, the code space needs to have enough room for such relevant information.
For this experiment, we use the same network architecture of 784-10-10-10-1 as a base one and decrease the number of units within a layer by 2.
As we shrink the second layer,it gets closer to the third layer in the information plane.
This can be explained as follows.
Since the code space of the second layer gets smaller, it is easier for the third layer to encode almost all of information from the second layer’s code space.
In case of PIB, they even get closer as we explicitly reduce the mutual information between the second and the third layer.